Add project brief

2026-07-25 23:09:55 +02:00
commit e5abb1293b
1 changed files with 220 additions and 0 deletions
@@ -0,0 +1,220 @@
+# Project: Lossless A/V Sync Marker Tool
+
+## Goal
+
+Build a Linux desktop GUI app — think "LosslessCut, but for fixing audio/video
+sync" — that lets a user:
+
+1. Load a video file (it has both a video track and an audio track, currently
+   out of sync with each other).
+2. Scrub the video track to a specific frame where a known event happens
+   (e.g. a clap, a flash, a beep) and click "Mark Video".
+3. Scrub the audio track to the same event and click "Mark Audio".
+4. The app computes the offset between the two markers (in ms).
+5. The app exports a new file with the audio shifted to match the video,
+   **losslessly** (stream copy, no re-encoding) via ffmpeg.
+
+This replaces a manual workflow of: scrub in a player, write down two
+timestamps, subtract them by hand, then run an ffmpeg command with
+`-itsoffset`.
+
+## Why frame-accurate scrubbing matters
+
+Sync events are often only 1-2 frames wide. HTML5 `<video>` seeking is not
+reliable enough at that precision. Use **PyQt6 (or PySide6) + libmpv**
+(via `python-mpv`) for playback — mpv supports frame-stepping and precise
+seeking (`--hr-seek`), which is what makes this usable.
+
+If python-mpv/libmpv setup proves too painful on the target system, python-vlc
+is an acceptable fallback, but frame-step precision should be verified early —
+this is the main technical risk in the project, so validate it first before
+building the rest of the UI around it.
+
+## Suggested architecture
+
+- `main.py` — app entry point, PyQt6 window
+- `player_widget.py` — embeds mpv (or vlc) render surface in a Qt widget,
+  exposes: `load(path)`, `seek(seconds)`, `step_frame(n)`, `get_position()`,
+  `play()`, `pause()`
+- `sync_logic.py` — pure functions: given two marker timestamps (video_mark,
+  audio_mark), compute delay_ms = (video_mark - audio_mark) * 1000. Sign
+  convention: positive delay_ms means audio needs to be delayed (shifted
+  later) to match video; negative means video needs to be delayed instead
+  (see ffmpeg_export.py for how each case is handled).
+- `ffmpeg_export.py` — wraps the ffmpeg calls below; takes (input_path,
+  delay_ms, output_path) and runs the correct lossless command depending on
+  sign of delay_ms.
+- `ui/` — Qt Designer .ui files or hand-built widgets: two player panes (or
+  one pane with a toggle for "viewing as video-sync-target" vs
+  "audio-sync-target" — TBD based on whether we want separate video/audio
+  scrubbers or one shared transport with two mark buttons), marker readouts,
+  offset display, "Export" button with output path picker.
+
+## UI concept (rough, adjust as needed during build)
+
+- Single video preview (shows both video+audio playing together, since it's
+  one file) OR two loaded instances of the same file if the user wants to
+  scrub each track independently — needs a decision during implementation
+  based on what feels usable. Likely simplest: one player, but let the user
+  mute video/audio independently while scrubbing so they can isolate the
+  track they're marking.
+- Timeline scrubber with frame-step buttons (prev frame / next frame) and
+  keyboard shortcuts (e.g. `,` / `.` like most NLEs).
+- Two "Mark" buttons: "Mark Video Event" and "Mark Audio Event" — each
+  captures the current playhead position in ms.
+- Live offset readout: "Offset: +503 ms (audio will be delayed)".
+- Optional: manual override field to nudge the computed offset by hand
+  before export.
+- Export button → runs ffmpeg lossless shift, shows progress/completion.
+
+## Existing working ffmpeg logic (reuse this, don't reinvent)
+
+This bash script (`ffdelay`) already implements the correct lossless
+ffmpeg commands for both positive and negative delays, including an
+optional clip mode (start timestamp + duration) for fast testing on a
+short segment instead of the whole file. Port this logic into
+`ffmpeg_export.py` (as subprocess calls or via a Python ffmpeg wrapper —
+subprocess is simplest and matches what's already proven to work):
+
+```bash
+#!/bin/bash
+
+# ffdelay - Easy audio delay tool for ffmpeg/ffplay
+# Usage:
+#   ./ffdelay input.mp4 fix 500
+#   ./ffdelay input.mp4 fix 500 00:45:00 30    # Quick 30s test from 45min mark
+#   ./ffdelay input.mp4 play 500
+#   ./ffdelay input.mp4 play 500 00:45:00
+
+set -e
+
+INPUT="$1"
+MODE="$2"
+DELAY_MS="$3"
+TIMESTAMP="$4"
+DURATION="$5"
+
+if [ -z "$INPUT" ] || [ -z "$MODE" ] || [ -z "$DELAY_MS" ]; then
+    echo "Usage:"
+    echo "  $0 <input.mp4> fix <delay_ms> [start_time] [duration_sec]"
+    echo "  $0 <input.mp4> play <delay_ms> [start_time]"
+    exit 1
+fi
+
+if [ ! -f "$INPUT" ]; then
+    echo "Error: Input file '$INPUT' not found"
+    exit 1
+fi
+
+ABS_DELAY_MS=${DELAY_MS#-}
+DELAY_SEC=$(awk "BEGIN {printf \"%.3f\", $DELAY_MS/1000}")
+ABS_DELAY_SEC=$(awk "BEGIN {printf \"%.3f\", $ABS_DELAY_MS/1000}")
+
+case "$MODE" in
+    fix)
+        BASENAME="${INPUT%.*}"
+        EXTENSION="${INPUT##*.}"
+
+        if [ -n "$TIMESTAMP" ] && [ -n "$DURATION" ]; then
+            OUTPUT="${BASENAME}_delayed${DELAY_MS}ms_clip.${EXTENSION}"
+        else
+            OUTPUT="${BASENAME}_delayed${DELAY_MS}ms.${EXTENSION}"
+        fi
+
+        if [ "$DELAY_MS" -lt 0 ]; then
+            # Negative delay = offset the VIDEO instead (delaying video
+            # relative to audio has the same effect as advancing audio)
+            if [ -n "$TIMESTAMP" ] && [ -n "$DURATION" ]; then
+                ffmpeg -itsoffset "$ABS_DELAY_SEC" -ss "$TIMESTAMP" -t "$DURATION" -i "$INPUT" \
+                       -ss "$TIMESTAMP" -t "$DURATION" -i "$INPUT" \
+                       -map 0:v -map 1:a -c:v copy -c:a copy "$OUTPUT"
+            else
+                ffmpeg -itsoffset "$ABS_DELAY_SEC" -i "$INPUT" -i "$INPUT" \
+                       -map 0:v -map 1:a -c:v copy -c:a copy "$OUTPUT"
+            fi
+        else
+            # Positive delay = offset the AUDIO
+            if [ -n "$TIMESTAMP" ] && [ -n "$DURATION" ]; then
+                ffmpeg -ss "$TIMESTAMP" -t "$DURATION" -i "$INPUT" \
+                       -itsoffset "$DELAY_SEC" -ss "$TIMESTAMP" -t "$DURATION" -i "$INPUT" \
+                       -map 0:v -map 1:a -c:v copy -c:a copy "$OUTPUT"
+            else
+                ffmpeg -i "$INPUT" -itsoffset "$DELAY_SEC" -i "$INPUT" \
+                       -map 0:v -map 1:a -c:v copy -c:a copy "$OUTPUT"
+            fi
+        fi
+        echo "Done! Created: $OUTPUT"
+        ;;
+
+    play)
+        if [ "$DELAY_MS" -lt 0 ]; then
+            if [ -n "$TIMESTAMP" ]; then
+                ffplay -ss "$TIMESTAMP" -t 30 -i "$INPUT" -vf "setpts=PTS+${ABS_DELAY_SEC}/TB"
+            else
+                ffplay -t 30 -i "$INPUT" -vf "setpts=PTS+${ABS_DELAY_SEC}/TB"
+            fi
+        else
+            if [ -n "$TIMESTAMP" ]; then
+                ffplay -ss "$TIMESTAMP" -t 30 -i "$INPUT" -af "adelay=${DELAY_MS}|${DELAY_MS}"
+            else
+                ffplay -t 30 -i "$INPUT" -af "adelay=${DELAY_MS}|${DELAY_MS}"
+            fi
+        fi
+        ;;
+
+    *)
+        echo "Error: Mode must be 'fix' or 'play'"
+        exit 1
+        ;;
+esac
+```
+
+### Known quirks already discovered during CLI development (avoid re-learning these):
+
+- `bc` produces `.500` instead of `0.500` for sub-1 values, which ffmpeg
+  rejects as an invalid duration for `-itsoffset`. Use `awk` (or Python's
+  own float formatting) instead — always emit a leading zero.
+- Negative delay must NOT be applied by naively negating inside `-itsoffset`
+  (ffmpeg's `-itsoffset` doesn't take negative values the way you'd hope for
+  this use case) — instead, restructure which of the two inputs gets the
+  offset and which stream is mapped from which input. See the two branches
+  above (`-lt 0` vs positive).
+- When combining `-itsoffset` with `-ss`/`-t` (for fast clip-based testing),
+  the offset flag must be placed correctly relative to the specific input
+  it applies to, and `-ss`/`-t` should be applied to BOTH inputs so the
+  extracted clip covers the same real-world time window in each.
+- Warnings like `sps_id out of range`, `Invalid NAL unit size`, or
+  `missing picture in access unit` during stream-copy operations are
+  generally harmless source-file quirks (common with screen recordings/live
+  captures) and don't indicate a problem with the sync fix itself — don't
+  treat these as fatal errors in the export UI, just log them.
+- `ffplay`'s `adelay` audio filter is cheap (just inserts silence) but the
+  video-delay equivalent (`setpts=PTS+offset/TB`) is comparatively expensive
+  because it re-times every frame — this made live preview of negative
+  offsets slow at full resolution in the CLI version. Worth keeping in mind
+  if the GUI adds a "preview shifted result before export" feature; may
+  want to preview on a downscaled/short clip rather than the full file.
+
+## Non-goals for v1
+
+- No automatic cross-correlation/audio-fingerprint sync detection (that's a
+  different, harder feature — separate tools like SyncSink exist for that).
+  This tool is specifically for the case where the user can see/hear the
+  sync event and just needs to mark it on each track.
+- No re-encoding/transcoding support — lossless stream copy only. If the
+  input codec/container doesn't support clean stream copy for some reason,
+  surface the ffmpeg error rather than silently falling back to re-encode.
+- No batch processing in v1 — single file in, single file out.
+
+## Suggested build order
+
+1. Validate frame-accurate seeking works acceptably with python-mpv (or
+   python-vlc) in a minimal Qt window — this is the highest-risk piece,
+   confirm it early before investing in the rest of the UI.
+2. Build the marker + offset calculation logic (pure Python, easily
+   testable without any UI).
+3. Port the ffmpeg export logic from the bash script above.
+4. Wire up the full UI: player, scrub controls, mark buttons, offset
+   display, export button.
+5. Test end-to-end on a real out-of-sync file with a clear sync event
+   (e.g. record a clap on both a camera and a separate audio recorder).