VideoTools/docs/VT_PLAYER_INTEGRATION_NOTES.md

VT_Player Integration Notes for Lead Developer

Project Context

VideoTools Repository: https://git.leaktechnologies.dev/Leak_Technologies/VideoTools.git VT_Player: Forked video player component for independent development

VT_Player was forked from VideoTools to enable dedicated development of video playback controls and features without impacting the main VideoTools codebase.

Current Integration Points

VideoTools Modules Using VT_Player

1. Convert Module - Preview video before/during conversion
2. Compare Module - Side-by-side video comparison (2 players)
3. Inspect Module - Single video playback with metadata display
4. Compare Fullscreen - Larger side-by-side view (planned: synchronized playback)

Current VT_Player Usage Pattern

// VideoTools calls buildVideoPane() which creates player
videoPane := buildVideoPane(state, fyne.NewSize(320, 180), videoSource, updateCallback)

// buildVideoPane internally:
// - Creates player.Controller
// - Sets up playback controls
// - Returns fyne.CanvasObject with player UI

Priority Features Needed in VT_Player

1. Keyframing API (HIGHEST PRIORITY)

Required for: Trim Module, Chapter Module

// Proposed API
type KeyframeController interface {
    // Set keyframe markers
    SetInPoint(position time.Duration) error
    SetOutPoint(position time.Duration) error
    ClearInPoint()
    ClearOutPoint()
    ClearAllKeyframes()

    // Get keyframe data
    GetInPoint() (time.Duration, bool)  // Returns position and hasInPoint
    GetOutPoint() (time.Duration, bool)
    GetSegmentDuration() time.Duration  // Duration between In and Out

    // Visual feedback
    ShowKeyframeMarkers(show bool)      // Toggle marker visibility on timeline
    HighlightSegment(in, out time.Duration) // Highlight region between markers
}

Use Case: User scrubs video, presses I to set In point, scrubs to end, presses O to set Out point. Visual markers show on timeline. VideoTools reads timestamps for FFmpeg trim command.

2. Frame-Accurate Navigation (HIGH PRIORITY)

Required for: Trim Module, Compare sync

type FrameNavigationController interface {
    // Step through video frame-by-frame
    StepForward() error   // Advance exactly 1 frame
    StepBackward() error  // Go back exactly 1 frame

    // Frame info
    GetCurrentFrame() int64              // Current frame number
    GetFrameAtTime(time.Duration) int64  // Frame number at timestamp
    GetTimeAtFrame(int64) time.Duration  // Timestamp of frame number
    GetTotalFrames() int64

    // Seek to exact frame
    SeekToFrame(frameNum int64) error
}

Use Case: User finds exact frame for cut point using arrow keys (←/→), sets In/Out markers precisely.

3. Synchronized Playback API (MEDIUM PRIORITY)

Required for: Compare Fullscreen, Compare Module sync

type SyncController interface {
    // Link two players together
    SyncWith(otherPlayer player.Controller) error
    Unsync()
    IsSynced() bool
    GetSyncMaster() player.Controller

    // Callbacks for sync events
    OnPlayStateChanged(callback func(playing bool))
    OnPositionChanged(callback func(position time.Duration))

    // Sync with offset (for videos that don't start at same time)
    SetSyncOffset(offset time.Duration)
    GetSyncOffset() time.Duration
}

Use Case: Compare module loads two videos. User clicks "Play Both" button. Both players play in sync. When one player is paused/seeked, other follows.

4. Playback Speed Control (MEDIUM PRIORITY)

Required for: Trim Module, general UX improvement

type PlaybackSpeedController interface {
    SetPlaybackSpeed(speed float64) error  // 0.25x to 2.0x
    GetPlaybackSpeed() float64
    GetSupportedSpeeds() []float64         // [0.25, 0.5, 0.75, 1.0, 1.25, 1.5, 2.0]
}

Use Case: User slows playback to 0.25x to find exact frame for trim point.

Integration Architecture

Current Pattern

VideoTools (main.go)
    └─> buildVideoPane()
        └─> player.New()
        └─> player.Controller interface
        └─> Returns fyne.CanvasObject

Proposed Enhanced Pattern

VideoTools (main.go)
    └─> buildVideoPane()
        └─> player.NewEnhanced()
            ├─> player.Controller (basic playback)
            ├─> player.KeyframeController (trim support)
            ├─> player.FrameNavigationController (frame stepping)
            ├─> player.SyncController (multi-player sync)
            └─> player.PlaybackSpeedController (speed control)
        └─> Returns fyne.CanvasObject

Backward Compatibility

• Keep existing player.Controller interface unchanged
• Add new optional interfaces
• VideoTools checks if player implements enhanced interfaces:

if keyframer, ok := player.(KeyframeController); ok {
    // Use keyframe features
}

Technical Requirements

1. Timeline Visual Enhancements

Current timeline needs:

• In/Out Point Markers: Visual indicators (⬇️ symbols or colored bars)
• Segment Highlight: Show region between In and Out with different color
• Frame Number Display: Show current frame number alongside timestamp
• Marker Drag Support: Allow dragging markers to adjust In/Out points

2. Keyboard Shortcuts

Essential shortcuts for VT_Player:

Key	Action	Notes
Space	Play/Pause	Standard
←	Step backward 1 frame	Frame-accurate
→	Step forward 1 frame	Frame-accurate
Shift+←	Jump back 1 second	Quick navigation
Shift+→	Jump forward 1 second	Quick navigation
I	Set In Point	Trim support
O	Set Out Point	Trim support
C	Clear keyframes	Reset markers
K	Pause	Video editor standard
J	Rewind	Video editor standard
L	Fast forward	Video editor standard
0-9	Seek to %	0=start, 5=50%, 9=90%

3. Performance Considerations

• Frame stepping: Must be instant, no lag
• Keyframe display: Update timeline without stuttering
• Sync: Maximum 1-frame drift between synced players
• Memory: Don't load entire video into RAM for frame navigation

4. FFmpeg Integration

VT_Player should expose frame-accurate timestamps that VideoTools can use:

# Example: VideoTools gets In=83.456s, Out=296.789s from VT_Player
ffmpeg -ss 83.456 -to 296.789 -i input.mp4 -c copy output.mp4

Frame-accurate seeking requires:

• Seek to nearest keyframe before target
• Decode frames until exact target reached
• Display correct frame with minimal latency

Data Flow Examples

Trim Module Workflow

1. User loads video in Trim module
2. VideoTools creates VT_Player with keyframe support
3. User navigates with arrow keys (VT_Player handles frame stepping)
4. User presses 'I' → VT_Player sets In point marker
5. User navigates to end point
6. User presses 'O' → VT_Player sets Out point marker
7. User clicks "Preview Trim" → VT_Player plays segment between markers
8. User clicks "Add to Queue"
9. VideoTools reads keyframes: in = player.GetInPoint(), out = player.GetOutPoint()
10. VideoTools builds FFmpeg command with timestamps
11. FFmpeg trims video

Compare Sync Workflow

1. User loads 2 videos in Compare module
2. VideoTools creates 2 VT_Player instances
3. User clicks "Play Both"
4. VideoTools calls: player1.SyncWith(player2)
5. VideoTools calls: player1.Play()
6. VT_Player automatically plays player2 in sync
7. User pauses player1 → VT_Player pauses player2
8. User seeks player1 → VT_Player seeks player2 to same position

Testing Requirements

VT_Player should include tests for:

1. Keyframe Accuracy

   • Set In/Out points, verify exact timestamps returned
   • Clear markers, verify they're removed
   • Test edge cases (In > Out, negative times, beyond duration)

2. Frame Navigation

   • Step forward/backward through entire video
   • Verify frame numbers are sequential
   • Test at video start (can't go back) and end (can't go forward)

3. Sync Reliability

   • Play two videos for 30 seconds, verify max drift < 1 frame
   • Pause/seek operations propagate correctly
   • Unsync works properly

4. Performance

   • Frame step latency < 50ms
   • Timeline marker updates < 16ms (60fps)
   • Memory usage stable during long playback sessions

Communication Protocol

VideoTools → VT_Player

VideoTools will request features through interface methods:

// Example: VideoTools wants to enable trim mode
if trimmer, ok := player.(TrimController); ok {
    trimmer.EnableTrimMode(true)
    trimmer.OnInPointSet(func(t time.Duration) {
        // Update VideoTools UI to show In point timestamp
    })
    trimmer.OnOutPointSet(func(t time.Duration) {
        // Update VideoTools UI to show Out point timestamp
    })
}

VT_Player → VideoTools

VT_Player communicates state changes through callbacks:

player.OnPlaybackStateChanged(func(playing bool) {
    // VideoTools updates UI (play button ↔ pause button)
})

player.OnPositionChanged(func(position time.Duration) {
    // VideoTools updates position display
})

player.OnKeyframeSet(func(markerType string, position time.Duration) {
    // VideoTools logs keyframe for FFmpeg command
})

Migration Strategy

Phase 1: Core API (Immediate)

• Define interfaces for keyframe, frame nav, sync
• Implement basic keyframe markers (In/Out points)
• Add frame stepping (←/→ keys)
• Document API for VideoTools integration

Phase 2: Visual Enhancements (Week 2)

• Enhanced timeline with marker display
• Segment highlighting between In/Out
• Frame number display
• Keyboard shortcuts

Phase 3: Sync Features (Week 3)

• Implement synchronized playback API
• Master-slave pattern for linked players
• Offset compensation for non-aligned videos

Phase 4: Advanced Features (Week 4+)

• Playback speed control
• Timeline zoom for precision editing
• Thumbnail preview on hover
• Chapter markers

Notes for VT_Player Developer

1. Keep backward compatibility: Existing VideoTools code using basic player.Controller should continue working

2. Frame-accurate is critical: Trim module requires exact frame positioning. Off-by-one frame errors are unacceptable.

3. Performance over features: Frame stepping must be instant. Users will hold arrow keys to scrub through video.

4. Visual feedback matters: Keyframe markers must be immediately visible. Timeline updates should be smooth.

5. Cross-platform testing: VT_Player must work on Linux (GNOME/X11/Wayland), macOS, and Windows

6. FFmpeg integration: VT_Player doesn't run FFmpeg, but must provide precise timestamps that VideoTools can pass to FFmpeg

7. Minimize dependencies: Keep VT_Player focused on playback/navigation. VideoTools handles video processing.

Questions to Consider

1. Keyframe storage: Should keyframes be stored in VT_Player or passed back to VideoTools immediately?

2. Sync drift handling: If synced players drift apart, which one is "correct"? Should we periodically resync?

3. Frame stepping during playback: Can user step frame-by-frame while video is playing, or must they pause first?

4. Memory management: For long videos (hours), how do we efficiently support frame-accurate navigation without excessive memory?

5. Hardware acceleration: Should frame stepping use GPU decoding, or is CPU sufficient for single frames?

Current VideoTools Status

Working Modules

• ✅ Convert - Video conversion with preview
• ✅ Compare - Side-by-side comparison (basic)
• ✅ Inspect - Single video with metadata
• ✅ Compare Fullscreen - Larger view (sync placeholders added)

Planned Modules Needing VT_Player Features

• ⏳ Trim - Needs: Keyframing + frame navigation
• ⏳ Chapter - Needs: Multiple keyframe markers on timeline
• ⏳ Merge - May need synchronized preview of multiple clips

Auto-Compare Feature (NEW)

• ✅ Checkbox in Convert module: "Compare After"
• ✅ After conversion completes, automatically loads:
  • File 1 (Original) = source video
  • File 2 (Converted) = output video
• ✅ User can immediately inspect conversion quality

Contact & Coordination

For questions about VideoTools integration:

• Review this document
• Check /docs/VIDEO_PLAYER_FORK.md for fork strategy
• Check /docs/TRIM_MODULE_DESIGN.md for detailed trim module requirements
• Check /docs/COMPARE_FULLSCREEN.md for sync requirements

VideoTools will track VT_Player changes and update integration code as new features become available.