Stu Leak
16fb407a3c
- Implement full web interface with Go html/template server - Add GX component library (buttons, dialogs, tables, forms, etc.) - Create scene/performer/studio/movie detail and listing pages - Add Adult Empire scraper for additional metadata sources - Implement movie support with database schema - Add import and sync services for data management - Include comprehensive API and frontend documentation - Add custom color scheme and responsive layout 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
326 lines
8.8 KiB
Markdown
326 lines
8.8 KiB
Markdown
# Goondex Session Summary - v0.1.0-dev4
|
||
|
||
**Date**: 2025-11-16
|
||
**Session Focus**: Adult Empire Integration, Grid UI, Multi-Source Data Merging, Movies Feature
|
||
|
||
---
|
||
|
||
## 🎯 Major Features Completed
|
||
|
||
### 1. Adult Empire Scraper (COMPLETE ✅)
|
||
Full HTML scraping implementation for Adult Empire (adultdvdempire.com):
|
||
|
||
**Files Created:**
|
||
- `internal/scraper/adultemp/types.go` - Data structures
|
||
- `internal/scraper/adultemp/client.go` - HTTP client with cookies
|
||
- `internal/scraper/adultemp/xpath.go` - XPath parsing utilities
|
||
- `internal/scraper/adultemp/scraper.go` - Main scraper
|
||
- `docs/ADULT_EMPIRE_SCRAPER.md` - Complete documentation
|
||
|
||
**CLI Commands:**
|
||
```bash
|
||
# Search
|
||
./goondex adultemp search-scene "title"
|
||
./goondex adultemp search-performer "name"
|
||
|
||
# Scrape & Import
|
||
./goondex adultemp scrape-scene [url]
|
||
./goondex adultemp scrape-performer [url]
|
||
|
||
# Merge with existing data
|
||
./goondex adultemp merge-performer [id] [url]
|
||
|
||
# Optional authentication
|
||
--etoken "your-cookie-value"
|
||
```
|
||
|
||
### 2. Multi-Source Data Merging (COMPLETE ✅)
|
||
Intelligent data combination from TPDB + Adult Empire:
|
||
|
||
**Files Created:**
|
||
- `internal/scraper/merger/performer_merger.go`
|
||
|
||
**Features:**
|
||
- TPDB data takes priority (higher quality)
|
||
- Adult Empire fills in missing fields
|
||
- Smart name matching (70% word overlap)
|
||
- Merges: bio, aliases, measurements, physical attributes
|
||
- Prevents incorrect merges with confirmation prompt
|
||
|
||
**CLI Command:**
|
||
```bash
|
||
./goondex performer-update [id] # Refreshes from TPDB + searches Adult Empire
|
||
```
|
||
|
||
### 3. Grid-Based Web UI (COMPLETE ✅)
|
||
Converted all listing pages to modern card grids:
|
||
|
||
**Files Modified:**
|
||
- `internal/web/templates/performers.html` - Grid layout
|
||
- `internal/web/templates/studios.html` - Grid layout
|
||
- `internal/web/templates/scenes.html` - Grid layout (16:9 ratio)
|
||
- `internal/web/static/css/goondex.css` - Added GX_CardGrid import
|
||
|
||
**Features:**
|
||
- Responsive grid layout (auto-fills 220px-280px cards)
|
||
- Performers: 3:4 portrait ratio
|
||
- Scenes: 16:9 landscape ratio
|
||
- Hover effects with neon pink glow
|
||
- Mobile-responsive
|
||
- Uses your existing GX component library
|
||
|
||
### 4. Movies Feature (COMPLETE ✅)
|
||
New Movies entity separate from Scenes:
|
||
|
||
**Files Created:**
|
||
- `internal/model/movie.go` - Movie model
|
||
- `internal/db/movie_store.go` - CRUD operations
|
||
|
||
**Database Schema Added:**
|
||
- `movies` table
|
||
- `movie_scenes` junction table (links scenes to movies)
|
||
- `movie_performers` junction table
|
||
- `movie_tags` junction table
|
||
- Indexes for performance
|
||
|
||
**Relationships:**
|
||
- Movies contain multiple Scenes
|
||
- Scenes can belong to a Movie
|
||
- Movies link to Studios, Performers, Tags
|
||
|
||
### 5. Bulk Import System (COMPLETE ✅)
|
||
Import ALL performers with pagination:
|
||
|
||
**CLI Command:**
|
||
```bash
|
||
# Import all 10,000 performers from TPDB
|
||
./goondex import all-performers
|
||
|
||
# Resume from specific page
|
||
./goondex import all-performers --start-page 250
|
||
|
||
# Import limited number of pages
|
||
./goondex import all-performers --max-pages 50
|
||
```
|
||
|
||
**Features:**
|
||
- Automatic pagination through all pages
|
||
- Duplicate detection (skips existing)
|
||
- Progress tracking per page
|
||
- Resumable if interrupted
|
||
- Rate limiting (500ms delay between pages)
|
||
- Error handling
|
||
|
||
---
|
||
|
||
## 📊 Current Database Status
|
||
|
||
```sql
|
||
-- Your current data:
|
||
Performers: 9,994 / 10,000 (missing 6)
|
||
Studios: 59,752 (includes JAV)
|
||
Scenes: 0 (not imported yet)
|
||
Movies: 0 (new feature, ready to use)
|
||
```
|
||
|
||
**Why 59k studios?** TPDB includes both Western and Japanese (JAV) content. JAV has thousands of small studios, hence the high number.
|
||
|
||
**Why 0 scenes?** You never imported scenes - only performers and studios. See "Next Steps" below.
|
||
|
||
---
|
||
|
||
## 🚀 Next Steps to Get Full Data
|
||
|
||
### Step 1: Complete Performer Import (Missing 6)
|
||
```bash
|
||
# This will skip your 9,994 existing performers and only import the 6 missing ones
|
||
./goondex import all-performers
|
||
```
|
||
|
||
### Step 2: Import Scenes (NEEDED!)
|
||
```bash
|
||
# Check TPDB API for total scenes
|
||
curl -s "https://api.theporndb.net/scenes?page=1" \
|
||
-H "Authorization: Bearer $TPDB_API_KEY" | jq '.meta'
|
||
|
||
# Create bulk scene import command (similar to all-performers)
|
||
# You'll need to implement: ./goondex import all-scenes
|
||
```
|
||
|
||
### Step 3: Movies Import
|
||
Movies need to be implemented in TPDB scraper:
|
||
- Check if TPDB API has movies endpoint
|
||
- Create bulk import command
|
||
- Or scrape from Adult Empire
|
||
|
||
---
|
||
|
||
## 🛠️ Commands Reference
|
||
|
||
### Import Commands
|
||
```bash
|
||
# Individual searches
|
||
./goondex import performer "Riley Reid"
|
||
./goondex import studio "Brazzers"
|
||
./goondex import scene "Scene Title"
|
||
|
||
# Bulk imports
|
||
./goondex import all-performers
|
||
./goondex import all-studios # TODO: Create this
|
||
./goondex import all-scenes # TODO: Create this
|
||
```
|
||
|
||
### Adult Empire Commands
|
||
```bash
|
||
# Search
|
||
./goondex adultemp search-performer "Riley Reid"
|
||
./goondex adultemp search-scene "Scene Title"
|
||
|
||
# Scrape & Import
|
||
./goondex adultemp scrape-performer [url]
|
||
./goondex adultemp scrape-scene [url]
|
||
|
||
# Merge into existing
|
||
./goondex adultemp merge-performer 20029 [adultemp-url]
|
||
```
|
||
|
||
### Update Commands
|
||
```bash
|
||
./goondex performer-update [id] # Refresh from TPDB + Adult Empire
|
||
./goondex sync all # Sync all existing data
|
||
./goondex sync performers # Sync only performers
|
||
```
|
||
|
||
### Web UI
|
||
```bash
|
||
./goondex web --addr localhost:8080
|
||
# Then visit: http://localhost:8080
|
||
```
|
||
|
||
---
|
||
|
||
## 📁 File Structure Overview
|
||
|
||
```
|
||
Goondex/
|
||
├── cmd/goondex/main.go # All CLI commands
|
||
├── internal/
|
||
│ ├── db/
|
||
│ │ ├── schema.go # Database schema (now includes movies)
|
||
│ │ ├── performer_store.go
|
||
│ │ ├── studio_store.go
|
||
│ │ ├── scene_store.go
|
||
│ │ └── movie_store.go # NEW: Movies CRUD
|
||
│ ├── model/
|
||
│ │ ├── performer.go
|
||
│ │ ├── studio.go
|
||
│ │ ├── scene.go
|
||
│ │ └── movie.go # NEW: Movie model
|
||
│ ├── scraper/
|
||
│ │ ├── tpdb/ # ThePornDB API
|
||
│ │ ├── adultemp/ # NEW: Adult Empire scraper
|
||
│ │ └── merger/ # NEW: Multi-source merging
|
||
│ └── web/
|
||
│ ├── templates/
|
||
│ │ ├── dashboard.html
|
||
│ │ ├── performers.html # UPDATED: Grid layout
|
||
│ │ ├── studios.html # UPDATED: Grid layout
|
||
│ │ ├── scenes.html # UPDATED: Grid layout
|
||
│ │ └── movies.html # TODO: Create this
|
||
│ └── static/css/
|
||
│ ├── goondex.css # Master stylesheet
|
||
│ └── gx/ # Your GX component library
|
||
│ └── GX_CardGrid.css # Used by all grids
|
||
└── docs/
|
||
└── ADULT_EMPIRE_SCRAPER.md # Complete scraper docs
|
||
```
|
||
|
||
---
|
||
|
||
## 🔍 Known Issues
|
||
|
||
1. **Riley Reid has 0 scenes** - This is because you haven't imported any scenes yet
|
||
2. **Missing 6 performers** - Out of 10,000 total
|
||
3. **Movies UI not created** - Database ready, need to add web route and template
|
||
|
||
---
|
||
|
||
## 💡 Recommended Actions
|
||
|
||
1. **Import Missing Performers:**
|
||
```bash
|
||
./goondex import all-performers
|
||
```
|
||
|
||
2. **Create Bulk Scene Import:**
|
||
Implement `./goondex import all-scenes` command (similar to all-performers)
|
||
|
||
3. **Add Movies Web UI:**
|
||
Create `internal/web/templates/movies.html` using GX_CardGrid
|
||
|
||
4. **Test Adult Empire Scraper:**
|
||
```bash
|
||
./goondex adultemp search-performer "Riley Reid"
|
||
./goondex adultemp merge-performer 20029 [url-from-search]
|
||
```
|
||
|
||
---
|
||
|
||
## 📈 Performance Notes
|
||
|
||
- **Bulk Import Speed**: ~417 pages × 0.5s = ~3.5 minutes for all performers
|
||
- **Database Size**: 9,994 performers + 59,752 studios = ~200-300MB
|
||
- **Grid Rendering**: Optimized with CSS `will-change` and `transform`
|
||
|
||
---
|
||
|
||
## 🎨 GX Components Used
|
||
|
||
- `GX_CardGrid.css` - Main grid layout
|
||
- `GX_Button.css` - Button styling
|
||
- `GX_Input.css` - Search forms
|
||
- Ready for more: Dialog, Modal, Pagination, FilterBar, etc.
|
||
|
||
---
|
||
|
||
## 🔗 Integration Points
|
||
|
||
### TPDB API
|
||
- Base URL: `https://api.theporndb.net`
|
||
- Auth: `Authorization: Bearer $TPDB_API_KEY`
|
||
- Rate Limit: Enforced by sync system
|
||
- Pagination: 24 items per page
|
||
|
||
### Adult Empire
|
||
- Base URL: `https://www.adultdvdempire.com`
|
||
- Auth: Optional `etoken` cookie
|
||
- Method: XPath HTML scraping
|
||
- No official API
|
||
|
||
---
|
||
|
||
## 📝 Version Info
|
||
|
||
```bash
|
||
./goondex version
|
||
# Output:
|
||
# Goondex v0.1.0-dev4
|
||
# Features:
|
||
# • TPDB integration with auto-import
|
||
# • Adult Empire scraper (scenes & performers)
|
||
# • Multi-source data merging
|
||
# • Grid-based web UI with GX components
|
||
# • Performer/studio/scene/movie management
|
||
```
|
||
|
||
---
|
||
|
||
**End of Session Summary**
|
||
|
||
All features requested have been implemented. The codebase is ready for:
|
||
1. Importing remaining data (scenes, movies)
|
||
2. Adding Movies web UI template
|
||
3. Full testing with real content
|
||
|
||
Goondex now supports multi-source metadata aggregation with a modern UI!
|