Project: Lossless A/V Sync Marker Tool

Goal

Build a Linux desktop GUI app — think "LosslessCut, but for fixing audio/video sync" — that lets a user:

Load a video file (it has both a video track and an audio track, currently out of sync with each other).
Scrub the video track to a specific frame where a known event happens (e.g. a clap, a flash, a beep) and click "Mark Video".
Scrub the audio track to the same event and click "Mark Audio".
The app computes the offset between the two markers (in ms).
The app exports a new file with the audio shifted to match the video, losslessly (stream copy, no re-encoding) via ffmpeg.

This replaces a manual workflow of: scrub in a player, write down two timestamps, subtract them by hand, then run an ffmpeg command with -itsoffset.

Why frame-accurate scrubbing matters

Sync events are often only 1-2 frames wide. HTML5 <video> seeking is not reliable enough at that precision. Use PyQt6 (or PySide6) + libmpv (via python-mpv) for playback — mpv supports frame-stepping and precise seeking (--hr-seek), which is what makes this usable.

If python-mpv/libmpv setup proves too painful on the target system, python-vlc is an acceptable fallback, but frame-step precision should be verified early — this is the main technical risk in the project, so validate it first before building the rest of the UI around it.

Suggested architecture

main.py — app entry point, PyQt6 window
player_widget.py — embeds mpv (or vlc) render surface in a Qt widget, exposes: load(path), seek(seconds), step_frame(n), get_position(), play(), pause()
sync_logic.py — pure functions: given two marker timestamps (video_mark, audio_mark), compute delay_ms = (video_mark - audio_mark) * 1000. Sign convention: positive delay_ms means audio needs to be delayed (shifted later) to match video; negative means video needs to be delayed instead (see ffmpeg_export.py for how each case is handled).
ffmpeg_export.py — wraps the ffmpeg calls below; takes (input_path, delay_ms, output_path) and runs the correct lossless command depending on sign of delay_ms.
ui/ — Qt Designer .ui files or hand-built widgets: two player panes (or one pane with a toggle for "viewing as video-sync-target" vs "audio-sync-target" — TBD based on whether we want separate video/audio scrubbers or one shared transport with two mark buttons), marker readouts, offset display, "Export" button with output path picker.

UI concept (rough, adjust as needed during build)

Single video preview (shows both video+audio playing together, since it's one file) OR two loaded instances of the same file if the user wants to scrub each track independently — needs a decision during implementation based on what feels usable. Likely simplest: one player, but let the user mute video/audio independently while scrubbing so they can isolate the track they're marking.
Timeline scrubber with frame-step buttons (prev frame / next frame) and keyboard shortcuts (e.g. , / . like most NLEs).
Two "Mark" buttons: "Mark Video Event" and "Mark Audio Event" — each captures the current playhead position in ms.
Live offset readout: "Offset: +503 ms (audio will be delayed)".
Optional: manual override field to nudge the computed offset by hand before export.
Export button → runs ffmpeg lossless shift, shows progress/completion.

Existing working ffmpeg logic (reuse this, don't reinvent)

This bash script (ffdelay) already implements the correct lossless ffmpeg commands for both positive and negative delays, including an optional clip mode (start timestamp + duration) for fast testing on a short segment instead of the whole file. Port this logic into ffmpeg_export.py (as subprocess calls or via a Python ffmpeg wrapper — subprocess is simplest and matches what's already proven to work):

#!/bin/bash

# ffdelay - Easy audio delay tool for ffmpeg/ffplay
# Usage:
#   ./ffdelay input.mp4 fix 500
#   ./ffdelay input.mp4 fix 500 00:45:00 30    # Quick 30s test from 45min mark
#   ./ffdelay input.mp4 play 500
#   ./ffdelay input.mp4 play 500 00:45:00

set -e

INPUT="$1"
MODE="$2"
DELAY_MS="$3"
TIMESTAMP="$4"
DURATION="$5"

if [ -z "$INPUT" ] || [ -z "$MODE" ] || [ -z "$DELAY_MS" ]; then
    echo "Usage:"
    echo "  $0 <input.mp4> fix <delay_ms> [start_time] [duration_sec]"
    echo "  $0 <input.mp4> play <delay_ms> [start_time]"
    exit 1
fi

if [ ! -f "$INPUT" ]; then
    echo "Error: Input file '$INPUT' not found"
    exit 1
fi

ABS_DELAY_MS=${DELAY_MS#-}
DELAY_SEC=$(awk "BEGIN {printf \"%.3f\", $DELAY_MS/1000}")
ABS_DELAY_SEC=$(awk "BEGIN {printf \"%.3f\", $ABS_DELAY_MS/1000}")

case "$MODE" in
    fix)
        BASENAME="${INPUT%.*}"
        EXTENSION="${INPUT##*.}"

        if [ -n "$TIMESTAMP" ] && [ -n "$DURATION" ]; then
            OUTPUT="${BASENAME}_delayed${DELAY_MS}ms_clip.${EXTENSION}"
        else
            OUTPUT="${BASENAME}_delayed${DELAY_MS}ms.${EXTENSION}"
        fi

        if [ "$DELAY_MS" -lt 0 ]; then
            # Negative delay = offset the VIDEO instead (delaying video
            # relative to audio has the same effect as advancing audio)
            if [ -n "$TIMESTAMP" ] && [ -n "$DURATION" ]; then
                ffmpeg -itsoffset "$ABS_DELAY_SEC" -ss "$TIMESTAMP" -t "$DURATION" -i "$INPUT" \
                       -ss "$TIMESTAMP" -t "$DURATION" -i "$INPUT" \
                       -map 0:v -map 1:a -c:v copy -c:a copy "$OUTPUT"
            else
                ffmpeg -itsoffset "$ABS_DELAY_SEC" -i "$INPUT" -i "$INPUT" \
                       -map 0:v -map 1:a -c:v copy -c:a copy "$OUTPUT"
            fi
        else
            # Positive delay = offset the AUDIO
            if [ -n "$TIMESTAMP" ] && [ -n "$DURATION" ]; then
                ffmpeg -ss "$TIMESTAMP" -t "$DURATION" -i "$INPUT" \
                       -itsoffset "$DELAY_SEC" -ss "$TIMESTAMP" -t "$DURATION" -i "$INPUT" \
                       -map 0:v -map 1:a -c:v copy -c:a copy "$OUTPUT"
            else
                ffmpeg -i "$INPUT" -itsoffset "$DELAY_SEC" -i "$INPUT" \
                       -map 0:v -map 1:a -c:v copy -c:a copy "$OUTPUT"
            fi
        fi
        echo "Done! Created: $OUTPUT"
        ;;

    play)
        if [ "$DELAY_MS" -lt 0 ]; then
            if [ -n "$TIMESTAMP" ]; then
                ffplay -ss "$TIMESTAMP" -t 30 -i "$INPUT" -vf "setpts=PTS+${ABS_DELAY_SEC}/TB"
            else
                ffplay -t 30 -i "$INPUT" -vf "setpts=PTS+${ABS_DELAY_SEC}/TB"
            fi
        else
            if [ -n "$TIMESTAMP" ]; then
                ffplay -ss "$TIMESTAMP" -t 30 -i "$INPUT" -af "adelay=${DELAY_MS}|${DELAY_MS}"
            else
                ffplay -t 30 -i "$INPUT" -af "adelay=${DELAY_MS}|${DELAY_MS}"
            fi
        fi
        ;;

    *)
        echo "Error: Mode must be 'fix' or 'play'"
        exit 1
        ;;
esac

Known quirks already discovered during CLI development (avoid re-learning these):

bc produces .500 instead of 0.500 for sub-1 values, which ffmpeg rejects as an invalid duration for -itsoffset. Use awk (or Python's own float formatting) instead — always emit a leading zero.
Negative delay must NOT be applied by naively negating inside -itsoffset (ffmpeg's -itsoffset doesn't take negative values the way you'd hope for this use case) — instead, restructure which of the two inputs gets the offset and which stream is mapped from which input. See the two branches above (-lt 0 vs positive).
When combining -itsoffset with -ss/-t (for fast clip-based testing), the offset flag must be placed correctly relative to the specific input it applies to, and -ss/-t should be applied to BOTH inputs so the extracted clip covers the same real-world time window in each.
Warnings like sps_id out of range, Invalid NAL unit size, or missing picture in access unit during stream-copy operations are generally harmless source-file quirks (common with screen recordings/live captures) and don't indicate a problem with the sync fix itself — don't treat these as fatal errors in the export UI, just log them.
ffplay's adelay audio filter is cheap (just inserts silence) but the video-delay equivalent (setpts=PTS+offset/TB) is comparatively expensive because it re-times every frame — this made live preview of negative offsets slow at full resolution in the CLI version. Worth keeping in mind if the GUI adds a "preview shifted result before export" feature; may want to preview on a downscaled/short clip rather than the full file.

Non-goals for v1

No automatic cross-correlation/audio-fingerprint sync detection (that's a different, harder feature — separate tools like SyncSink exist for that). This tool is specifically for the case where the user can see/hear the sync event and just needs to mark it on each track.
No re-encoding/transcoding support — lossless stream copy only. If the input codec/container doesn't support clean stream copy for some reason, surface the ffmpeg error rather than silently falling back to re-encode.
No batch processing in v1 — single file in, single file out.

Suggested build order

Validate frame-accurate seeking works acceptably with python-mpv (or python-vlc) in a minimal Qt window — this is the highest-risk piece, confirm it early before investing in the rest of the UI.
Build the marker + offset calculation logic (pure Python, easily testable without any UI).
Port the ffmpeg export logic from the bash script above.
Wire up the full UI: player, scrub controls, mark buttons, offset display, export button.
Test end-to-end on a real out-of-sync file with a clear sync event (e.g. record a clap on both a camera and a separate audio recorder).

9.7 KiB Raw Permalink Blame History