ThePendulum
76b5f827be
Root cause of the reported 'timestamps off by different amounts, sometimes early, sometimes late' bug: _extract_audio_wav() never passed an explicit -map to ffmpeg, so on multi-audio-track files it relied on ffmpeg's own default-stream heuristic -- which has no knowledge of, and isn't guaranteed to agree with, whichever audio track mpv is actually configured to play. A wrong-track clap is essentially unrelated to the true one (not a small timestamp-math error), which explains the inconsistent, file-dependent symptom -- and our single-audio-track synthetic fixtures never exercised this path at all, since they only ever had one track to pick. Fixed by threading an explicit ffmpeg stream index through detect_claps() -> _extract_audio_wav(), sourced in main.py from mpv's own track-list (the ff-index of whichever audio track has selected=True). Detection now always analyzes the exact track the user is listening to, on both 'Detect All' and 'Detect Next'. Also: _probe_audio_start_time() used to silently fall back to 0.0 if ffprobe couldn't parse start_time as a plain float (e.g. N/A) -- same failure class (silently wrong instead of loud). Now emits a Python warning instead, so a genuinely unparseable/unusual container is at least visible rather than silently mis-corrected. Verified two ways: (1) ruled out an initial alternate hypothesis (video stream's own start_time mattering) empirically against the existing single-track fixtures before ruling it out. (2) Built a new two-audio-track fixture (sync_test_MULTITRACK.mp4: track 1 has a beep at 3s, track 2 at 7s) and confirmed end-to-end through the real app -- switching mpv's active track from aid=1 to aid=2 correctly flips which clap gets detected, which is exactly the mechanism this bug required.
Synthetic test clips
Generated with ffmpeg (see git history / commit for exact commands). Ground truth for validating marker capture, offset math, and lossless export.
-
sync_test_INSYNC.mp4— 10s, 640x480 @30fps, black video with a 33ms white flash at t=5.000s; near-silent 1kHz tone that jumps to full volume for the same 33ms window at t=5.000s. Video and audio events are aligned (offset = 0ms). -
sync_test_OUTOFSYNC_audio+350ms.mp4— same content, but the audio track has been shifted +350ms later relative to video (via-itsoffset 0.350on the audio input, stream-copied). The flash event stays at video t=5.000s; the beep event now appears at t=5.350s. Exercises the negative-delay_ms / "delay video" branch in ffmpeg_export.py. -
sync_test_OUTOFSYNC_video+350ms.mp4— same content, but the video track has been shifted +350ms later instead (via-itsoffset 0.350on the video input). The beep stays at audio t=5.000s; the flash now appears at t=5.350s. Exercises the positive-delay_ms / "delay audio" branch in ffmpeg_export.py.
Ground truth for the app's marker/offset workflow:
- Mark Video at the flash → ~5.000s
- Mark Audio at the beep → ~5.350s
- Expected computed
delay_ms= (video_mark - audio_mark) * 1000 = -350 (audio is already the late one here, so the fix is to delay the video by 350ms to catch up to it -- not to delay the audio further. Negative delay_ms → offset the video input, per sync_logic.py's sign convention.) - Exporting
sync_test_OUTOFSYNC_audio+350ms.mp4with delay_ms = -350 should land both events at t=5.350s in the output (not back at 5.000s -- the absolute timeline shifts, only the relative sync matters).
Frame at exactly t=5.000s is frame index 150 (30fps) — useful for checking frame-step precision lands exactly on it.
sync_test_MULTITRACK.mp4— 10s, blue video, two AAC audio tracks: track index 1 (default) has a beep at t=3.000s, track index 2 (not default) has a beep at t=7.000s. Used to verify clap detection follows whichever audio track mpv is actually playing (itsff-index), not whichever one ffmpeg's own default-stream selection would pick -- see scripts/smoke_test_multitrack_clap_detection.py.