ThePendulum
76b5f827be
Root cause of the reported 'timestamps off by different amounts, sometimes early, sometimes late' bug: _extract_audio_wav() never passed an explicit -map to ffmpeg, so on multi-audio-track files it relied on ffmpeg's own default-stream heuristic -- which has no knowledge of, and isn't guaranteed to agree with, whichever audio track mpv is actually configured to play. A wrong-track clap is essentially unrelated to the true one (not a small timestamp-math error), which explains the inconsistent, file-dependent symptom -- and our single-audio-track synthetic fixtures never exercised this path at all, since they only ever had one track to pick. Fixed by threading an explicit ffmpeg stream index through detect_claps() -> _extract_audio_wav(), sourced in main.py from mpv's own track-list (the ff-index of whichever audio track has selected=True). Detection now always analyzes the exact track the user is listening to, on both 'Detect All' and 'Detect Next'. Also: _probe_audio_start_time() used to silently fall back to 0.0 if ffprobe couldn't parse start_time as a plain float (e.g. N/A) -- same failure class (silently wrong instead of loud). Now emits a Python warning instead, so a genuinely unparseable/unusual container is at least visible rather than silently mis-corrected. Verified two ways: (1) ruled out an initial alternate hypothesis (video stream's own start_time mattering) empirically against the existing single-track fixtures before ruling it out. (2) Built a new two-audio-track fixture (sync_test_MULTITRACK.mp4: track 1 has a beep at 3s, track 2 at 7s) and confirmed end-to-end through the real app -- switching mpv's active track from aid=1 to aid=2 correctly flips which clap gets detected, which is exactly the mechanism this bug required.
41 lines
2.2 KiB
Markdown
41 lines
2.2 KiB
Markdown
# Synthetic test clips
|
|
|
|
Generated with ffmpeg (see git history / commit for exact commands). Ground truth for
|
|
validating marker capture, offset math, and lossless export.
|
|
|
|
- `sync_test_INSYNC.mp4` — 10s, 640x480 @30fps, black video with a 33ms white flash
|
|
at t=5.000s; near-silent 1kHz tone that jumps to full volume for the same 33ms
|
|
window at t=5.000s. Video and audio events are aligned (offset = 0ms).
|
|
|
|
- `sync_test_OUTOFSYNC_audio+350ms.mp4` — same content, but the audio track has
|
|
been shifted +350ms later relative to video (via `-itsoffset 0.350` on the audio
|
|
input, stream-copied). The flash event stays at video t=5.000s; the beep event
|
|
now appears at t=5.350s. Exercises the negative-delay_ms / "delay video" branch
|
|
in ffmpeg_export.py.
|
|
|
|
- `sync_test_OUTOFSYNC_video+350ms.mp4` — same content, but the *video* track has
|
|
been shifted +350ms later instead (via `-itsoffset 0.350` on the video input).
|
|
The beep stays at audio t=5.000s; the flash now appears at t=5.350s. Exercises
|
|
the positive-delay_ms / "delay audio" branch in ffmpeg_export.py.
|
|
|
|
**Ground truth for the app's marker/offset workflow:**
|
|
- Mark Video at the flash → ~5.000s
|
|
- Mark Audio at the beep → ~5.350s
|
|
- Expected computed `delay_ms` = (video_mark - audio_mark) * 1000 = **-350**
|
|
(audio is already the *late* one here, so the fix is to delay the video by
|
|
350ms to catch up to it -- not to delay the audio further. Negative
|
|
delay_ms → offset the video input, per sync_logic.py's sign convention.)
|
|
- Exporting `sync_test_OUTOFSYNC_audio+350ms.mp4` with delay_ms = -350 should
|
|
land both events at t=5.350s in the output (not back at 5.000s -- the
|
|
absolute timeline shifts, only the *relative* sync matters).
|
|
|
|
Frame at exactly t=5.000s is frame index 150 (30fps) — useful for checking
|
|
frame-step precision lands exactly on it.
|
|
|
|
- `sync_test_MULTITRACK.mp4` — 10s, blue video, two AAC audio tracks: track
|
|
index 1 (default) has a beep at t=3.000s, track index 2 (not default) has
|
|
a beep at t=7.000s. Used to verify clap detection follows whichever audio
|
|
track mpv is actually playing (its `ff-index`), not whichever one
|
|
ffmpeg's own default-stream selection would pick -- see
|
|
scripts/smoke_test_multitrack_clap_detection.py.
|