WeAreSyncing/PROJECT_BRIEF.md

# Project: Lossless A/V Sync Marker Tool

## Goal

Build a Linux desktop GUI app — think "LosslessCut, but for fixing audio/video
sync" — that lets a user:

1. Load a video file (it has both a video track and an audio track, currently
   out of sync with each other).
2. Scrub the video track to a specific frame where a known event happens
   (e.g. a clap, a flash, a beep) and click "Mark Video".
3. Scrub the audio track to the same event and click "Mark Audio".
4. The app computes the offset between the two markers (in ms).
5. The app exports a new file with the audio shifted to match the video,
   **losslessly** (stream copy, no re-encoding) via ffmpeg.

This replaces a manual workflow of: scrub in a player, write down two
timestamps, subtract them by hand, then run an ffmpeg command with
`-itsoffset`.

## Why frame-accurate scrubbing matters

Sync events are often only 1-2 frames wide. HTML5 `<video>` seeking is not
reliable enough at that precision. Use **PyQt6 (or PySide6) + libmpv**
(via `python-mpv`) for playback — mpv supports frame-stepping and precise
seeking (`--hr-seek`), which is what makes this usable.

If python-mpv/libmpv setup proves too painful on the target system, python-vlc
is an acceptable fallback, but frame-step precision should be verified early —
this is the main technical risk in the project, so validate it first before
building the rest of the UI around it.

## Suggested architecture

- `main.py` — app entry point, PyQt6 window
- `player_widget.py` — embeds mpv (or vlc) render surface in a Qt widget,
  exposes: `load(path)`, `seek(seconds)`, `step_frame(n)`, `get_position()`,
  `play()`, `pause()`
- `sync_logic.py` — pure functions: given two marker timestamps (video_mark,
  audio_mark), compute delay_ms = (video_mark - audio_mark) * 1000. Sign
  convention: positive delay_ms means audio needs to be delayed (shifted
  later) to match video; negative means video needs to be delayed instead
  (see ffmpeg_export.py for how each case is handled).
- `ffmpeg_export.py` — wraps the ffmpeg calls below; takes (input_path,
  delay_ms, output_path) and runs the correct lossless command depending on
  sign of delay_ms.
- `ui/` — Qt Designer .ui files or hand-built widgets: two player panes (or
  one pane with a toggle for "viewing as video-sync-target" vs
  "audio-sync-target" — TBD based on whether we want separate video/audio
  scrubbers or one shared transport with two mark buttons), marker readouts,
  offset display, "Export" button with output path picker.

## UI concept (rough, adjust as needed during build)

- Single video preview (shows both video+audio playing together, since it's
  one file) OR two loaded instances of the same file if the user wants to
  scrub each track independently — needs a decision during implementation
  based on what feels usable. Likely simplest: one player, but let the user
  mute video/audio independently while scrubbing so they can isolate the
  track they're marking.
- Timeline scrubber with frame-step buttons (prev frame / next frame) and
  keyboard shortcuts (e.g. `,` / `.` like most NLEs).
- Two "Mark" buttons: "Mark Video Event" and "Mark Audio Event" — each
  captures the current playhead position in ms.
- Live offset readout: "Offset: +503 ms (audio will be delayed)".
- Optional: manual override field to nudge the computed offset by hand
  before export.
- Export button → runs ffmpeg lossless shift, shows progress/completion.

## Existing working ffmpeg logic (reuse this, don't reinvent)

This bash script (`ffdelay`) already implements the correct lossless
ffmpeg commands for both positive and negative delays, including an
optional clip mode (start timestamp + duration) for fast testing on a
short segment instead of the whole file. Port this logic into
`ffmpeg_export.py` (as subprocess calls or via a Python ffmpeg wrapper —
subprocess is simplest and matches what's already proven to work):

```bash
#!/bin/bash

# ffdelay - Easy audio delay tool for ffmpeg/ffplay
# Usage:
#   ./ffdelay input.mp4 fix 500
#   ./ffdelay input.mp4 fix 500 00:45:00 30    # Quick 30s test from 45min mark
#   ./ffdelay input.mp4 play 500
#   ./ffdelay input.mp4 play 500 00:45:00

set -e

INPUT="$1"
MODE="$2"
DELAY_MS="$3"
TIMESTAMP="$4"
DURATION="$5"

if [ -z "$INPUT" ] || [ -z "$MODE" ] || [ -z "$DELAY_MS" ]; then
    echo "Usage:"
    echo "  $0 <input.mp4> fix <delay_ms> [start_time] [duration_sec]"
    echo "  $0 <input.mp4> play <delay_ms> [start_time]"
    exit 1
fi

if [ ! -f "$INPUT" ]; then
    echo "Error: Input file '$INPUT' not found"
    exit 1
fi

ABS_DELAY_MS=${DELAY_MS#-}
DELAY_SEC=$(awk "BEGIN {printf \"%.3f\", $DELAY_MS/1000}")
ABS_DELAY_SEC=$(awk "BEGIN {printf \"%.3f\", $ABS_DELAY_MS/1000}")

case "$MODE" in
    fix)
        BASENAME="${INPUT%.*}"
        EXTENSION="${INPUT##*.}"

        if [ -n "$TIMESTAMP" ] && [ -n "$DURATION" ]; then
            OUTPUT="${BASENAME}_delayed${DELAY_MS}ms_clip.${EXTENSION}"
        else
            OUTPUT="${BASENAME}_delayed${DELAY_MS}ms.${EXTENSION}"
        fi

        if [ "$DELAY_MS" -lt 0 ]; then
            # Negative delay = offset the VIDEO instead (delaying video
            # relative to audio has the same effect as advancing audio)
            if [ -n "$TIMESTAMP" ] && [ -n "$DURATION" ]; then
                ffmpeg -itsoffset "$ABS_DELAY_SEC" -ss "$TIMESTAMP" -t "$DURATION" -i "$INPUT" \
                       -ss "$TIMESTAMP" -t "$DURATION" -i "$INPUT" \
                       -map 0:v -map 1:a -c:v copy -c:a copy "$OUTPUT"
            else
                ffmpeg -itsoffset "$ABS_DELAY_SEC" -i "$INPUT" -i "$INPUT" \
                       -map 0:v -map 1:a -c:v copy -c:a copy "$OUTPUT"
            fi
        else
            # Positive delay = offset the AUDIO
            if [ -n "$TIMESTAMP" ] && [ -n "$DURATION" ]; then
                ffmpeg -ss "$TIMESTAMP" -t "$DURATION" -i "$INPUT" \
                       -itsoffset "$DELAY_SEC" -ss "$TIMESTAMP" -t "$DURATION" -i "$INPUT" \
                       -map 0:v -map 1:a -c:v copy -c:a copy "$OUTPUT"
            else
                ffmpeg -i "$INPUT" -itsoffset "$DELAY_SEC" -i "$INPUT" \
                       -map 0:v -map 1:a -c:v copy -c:a copy "$OUTPUT"
            fi
        fi
        echo "Done! Created: $OUTPUT"
        ;;

    play)
        if [ "$DELAY_MS" -lt 0 ]; then
            if [ -n "$TIMESTAMP" ]; then
                ffplay -ss "$TIMESTAMP" -t 30 -i "$INPUT" -vf "setpts=PTS+${ABS_DELAY_SEC}/TB"
            else
                ffplay -t 30 -i "$INPUT" -vf "setpts=PTS+${ABS_DELAY_SEC}/TB"
            fi
        else
            if [ -n "$TIMESTAMP" ]; then
                ffplay -ss "$TIMESTAMP" -t 30 -i "$INPUT" -af "adelay=${DELAY_MS}|${DELAY_MS}"
            else
                ffplay -t 30 -i "$INPUT" -af "adelay=${DELAY_MS}|${DELAY_MS}"
            fi
        fi
        ;;

    *)
        echo "Error: Mode must be 'fix' or 'play'"
        exit 1
        ;;
esac
```

### Known quirks already discovered during CLI development (avoid re-learning these):

- `bc` produces `.500` instead of `0.500` for sub-1 values, which ffmpeg
  rejects as an invalid duration for `-itsoffset`. Use `awk` (or Python's
  own float formatting) instead — always emit a leading zero.
- Negative delay must NOT be applied by naively negating inside `-itsoffset`
  (ffmpeg's `-itsoffset` doesn't take negative values the way you'd hope for
  this use case) — instead, restructure which of the two inputs gets the
  offset and which stream is mapped from which input. See the two branches
  above (`-lt 0` vs positive).
- When combining `-itsoffset` with `-ss`/`-t` (for fast clip-based testing),
  the offset flag must be placed correctly relative to the specific input
  it applies to, and `-ss`/`-t` should be applied to BOTH inputs so the
  extracted clip covers the same real-world time window in each.
- Warnings like `sps_id out of range`, `Invalid NAL unit size`, or
  `missing picture in access unit` during stream-copy operations are
  generally harmless source-file quirks (common with screen recordings/live
  captures) and don't indicate a problem with the sync fix itself — don't
  treat these as fatal errors in the export UI, just log them.
- `ffplay`'s `adelay` audio filter is cheap (just inserts silence) but the
  video-delay equivalent (`setpts=PTS+offset/TB`) is comparatively expensive
  because it re-times every frame — this made live preview of negative
  offsets slow at full resolution in the CLI version. Worth keeping in mind
  if the GUI adds a "preview shifted result before export" feature; may
  want to preview on a downscaled/short clip rather than the full file.

## Non-goals for v1

- No automatic cross-correlation/audio-fingerprint sync detection (that's a
  different, harder feature — separate tools like SyncSink exist for that).
  This tool is specifically for the case where the user can see/hear the
  sync event and just needs to mark it on each track.
- No re-encoding/transcoding support — lossless stream copy only. If the
  input codec/container doesn't support clean stream copy for some reason,
  surface the ffmpeg error rather than silently falling back to re-encode.
- No batch processing in v1 — single file in, single file out.

## Suggested build order

1. Validate frame-accurate seeking works acceptably with python-mpv (or
   python-vlc) in a minimal Qt window — this is the highest-risk piece,
   confirm it early before investing in the rest of the UI.
2. Build the marker + offset calculation logic (pure Python, easily
   testable without any UI).
3. Port the ffmpeg export logic from the bash script above.
4. Wire up the full UI: player, scrub controls, mark buttons, offset
   display, export button.
5. Test end-to-end on a real out-of-sync file with a clear sync event
   (e.g. record a clap on both a camera and a separate audio recorder).