If automatic audio sync keeps getting it wrong: the static screen that scene cut detection can't see

This is a translated article. Read the original in Korean ↗

I was building personaverse, a pipeline that automatically produces Shorts videos of AI characters bickering about the day's news, all the way from script to render. One feature takes the original audio from the source clip used as the background and syncs it onto the recorded output automatically. On episodes where a single character stands almost motionless in a small player area, the audio was off every time.

Conclusion

If your pipeline measures audio offset automatically by lining up scene cuts and the values keep coming out wrong, first check whether the input video is low-motion (close to a static screen). Scene cut detection relies on how much the picture changes between frames. When most of the frame is white background and the subject barely moves inside a small area, the change signal gets buried. Before you reach for "just lower the sensitivity," check whether the recording has any cut signal to compare against in the first place .

You can check for a cut signal directly with FFmpeg's scene change detection filter, scdet. Note that this command was not actually run in this session. It's a general procedure researched separately from the official FFmpeg documentation, added for readers who want to reproduce the situation.

ffmpeg -i input.mp4 -vf "scdet=threshold=10,metadata=print" -f null -

If lavfi.scd.time never appears in the log, zero cuts were detected at that threshold. The FFmpeg docs recommend a threshold between 8.0 and 14.0, but nobody actually tuned the threshold against this case's video, so don't treat that range as the answer for this incident. The only takeaway is the direction: check first whether the video produces any cuts at all.

It gets more dangerous when the code silently papers over that failure with a fixed fallback value. While the fallback happens to be close enough, the bug stays hidden, and the drift only shows up once conditions change. Here, on the first render (no title card), the fallback of 0.3s was close to the usual delay, so nothing looked wrong. Once a 2-second title card was added, the fallback of 2.82s and the real offset of 2.57s were 0.25s apart, and the audio came in late .

The fix was to stop trusting the automatic measurement. Instead, frame grids were built from both the source clip and the final video, the points where the content clearly changes were found by eye , and the offset was worked out by hand by comparing when the same scene appears in each. The exact frame-extraction command isn't in the session record either, but with standard FFmpeg features the same job looks like this:

ffmpeg -i input.mp4 -vf fps=2 frame_%05d.png

Extracting numbered frames at 2 per second means file number n sits at (n-1)/2 seconds. To skim many frames in one image, you can also build a grid with the tile filter.

Clip time 17.1s showed up at 19.67s in the final video, and clip time 51.2s at 53.8s. Both points gave the same offset: 2.57s .

Once the offset is settled, only the audio gets re-encoded. There's no re-recording.

python video-engine/make.py --scene work/<slug>/scene.json --skip-record --clip-audio-delay 2.57

--skip-record lets you re-sync just the audio without re-recording .

The offset's sign is (time the scene appears in the final video) − (time the same scene appears in the source clip). If the value is positive, delay the audio by that amount. Both points in this case were positive, at +2.57s . A negative value (where the audio needs to start earlier) is a different story: the source of video-engine/make.py isn't in this repository, so it couldn't be confirmed whether --clip-audio-delay actually accepts negative values. If you run into a case that needs a negative value, check the source before passing a negative number to the flag.

Situation

The pipeline records HTML-built scenes with headless Chromium (Playwright) to produce the raw video (raw webm). It then lays the background clip's original audio on top, delayed by a set amount. That delay (lead time / offset) isn't measured by a person. It's measured automatically by finding the points where the scene changes in both the source clip and the recording, then lining them up .

The problem video was a 53-second edited clip: a Short that stitched together remarks from Liv's live stream, Yoo Youngwoo's joke singing, and a twist section where he turns out to actually sing well. The final Short was rendered in the canvas-white format, where a character introduces a small player .

Symptom

• Rendering a Short with a 109MB edited clip as the background left the audio delay unmeasured. It fell back to 0.3s. That value was close to the usual delay, so it didn't surface as a problem at the time
• After a 2-second title card was added at the very start, the audio delay again came from a fallback value of 2.82s instead of the automatic measurement
• Comparing scene changes at two points showed the real offset was 2.57s. That was 0.25s less than the 2.82s applied, so the audio came in that much late

Cause

The first suspect was that the title card's fade effect was being misread as a scene cut. The idea was that a fade to dark and back could look like a scene change, so the matching logic was fixed. After the fix, though, it still produced the fallback value (2.82) .

With the matching logic ruled out, the raw recording (raw webm) itself was checked frame by frame. No cuts were being detected at all. In this video, the player area where the character stands takes up only a small part of the screen. Everything else is white background, and the character barely moves, so the scene-change signal between frames got buried .

So the real cause wasn't a bug in the matching logic. The input video didn't have enough on-screen change for the scene detection algorithm to react to. When scene cut detection fails, the code falls back to a fixed value. On the first render (no title card), the fallback (0.3s) happened to be close to the real value, which hid the problem. Once the title card was added and the needed offset grew, the fallback (2.82s) and the real value (2.57s) drifted apart, and only then did the misalignment show .

Fix

The automatic measurement was abandoned. Frame grids were built from both the source clip and the final video, and the points where the same scene changes were picked out by eye . The session record only says "checked frame by frame directly" and doesn't say which command was used. To do the same with standard FFmpeg features, extract frames at a fixed interval with the fps filter (ffmpeg -i input.mp4 -vf fps=2 frame_%05d.png) and multiply the file number by the interval in seconds to get each frame's time. To skim many frames in one image, build a grid with the tile filter.

Next came comparing when the same scene appears in each video. Clip time 17.1s appeared at 19.67s in the final video, and clip time 51.2s at 53.8s. Both points gave exactly the same offset, 2.57s, so the value was trustworthy .

python video-engine/make.py --scene work/<slug>/scene.json --skip-record --clip-audio-delay 2.57

Only the audio was re-encoded with this value . The offset's sign is the final-video time minus the clip time, and both points here were positive (+2.57s), meaning the audio just needs to be delayed by that much. Whether --clip-audio-delay accepts a negative value as-is was not confirmed.

The session concluded that "the small player + low-motion solo character shot format doesn't work well with automatic measurement, so from now on, episodes in this format will have their offset measured and set by hand after rendering" .

Bonus — a fallback value that happens to be right hides the bug

Here too, at first nobody noticed the automatic measurement had failed, because the 0.3s fallback was close to the real value. It only came to light once the title card pushed the needed offset up and the gap between the fallback (2.82) and the real value (2.57) opened up .

Reporter's Notes — the fallback just got lucky

Park Doeun

You know our Shorts engine, the feature that auto-syncs the background clip's audio? This episode's audio keeps drifting late.

Jung Bada

Lead time formula's probably wrong again. Last time, the formula that subtracted playback length from raw length also ate the trailing buffer and pushed it a full second late .

Jung Haneul

We already fixed that by switching to scene cut matching. What makes you think it's that formula again?

Jung Bada

Hm... nothing really. I just figured the math was off again.

Park Doeun

Let's check the logs first. Huh, this one says it wasn't auto-measured, it used the 0.3s fallback .

Jung Haneul

Fallback means the auto-measurement failed outright. So why didn't anyone notice on the earlier render?

Park Dohyun

Because the fallback happened to be close. 0.3s isn't far off the usual delay.

Jung Bada

Oh, so this time adding the 2-second title card up front made the needed offset bigger, the fallback (2.82) and the real value drifted apart, and that's how it got caught .

Park Doeun

But shouldn't we first figure out why the measurement itself fails? Start with why the fallback kicks in.

Jung Bada

I think it's mistaking the title card's fade for a scene cut. Going dark and coming back can look like a cut .

Jung Haneul

Sounds plausible. Let's fix the matching logic and measure again.

The matching logic is patched so fades no longer count as scene cuts.

Park Dohyun

So what happened when you re-measured? If it's still the fallback, the fade theory is out.

Park Doeun

Still the 2.82s fallback. The logic fix did nothing.

Park Dohyun

Then before blaming the matching, let me ask: does the raw even have cuts in it? Did you check?

Park Doeun

No, I didn't. Opening raw.webm right now.

The raw recording is stepped through frame by frame.

Jung Bada

Well? See anything?

Park Doeun

No cuts get picked up at all. Most of the screen is white background and the character barely moves .

Jung Haneul

That's it, the real cause. Scene cut detection looks at how much the picture changes between frames. If nothing changes, the signal itself is weak.

Park Doeun

Wait, what exactly does "scene cut detection" look at? How is it different from what we see with our eyes?

Jung Bada

It finds the points where the picture changes sharply in both the source clip and the recording, then lines those points up to get the delay. No change points means nothing to line up .

Park Dohyun

Then this format just doesn't work with auto-measurement. It's a solo shot, one character standing in a small player.

Jung Haneul

Then let's drop the auto part and measure it ourselves. Compare the same scene's timestamps in the clip and the final cut with frame grids.

Park Doeun

Measured it. Clip 17.1s lands at 19.67s in the final, and 51.2s lands at 53.8s. Both give the same offset, 2.57s .

Jung Bada

If two points agree, it's trustworthy. That's 0.25s less than the 2.82 we applied.

Park Doeun

Re-applied just the audio with --clip-audio-delay 2.57. No re-recording needed, so it's fast .

Park Dohyun

But is this actually fixed, or did we just patch this one episode? The next similar format will hit the same fallback.

Jung Haneul

Right. Auto-measurement structurally doesn't work well on this canvas-white format, so let's make it the rule that episodes in this format get measured by hand after rendering .

Jung Bada

So my first theory (mistaking the fade) wasn't totally wrong. The fade was a problem too.

Park Dohyun

Even if it was half right, you should have dropped it the moment the fallback came back after the fix. Same symptom doesn't mean a single cause.

A final wrap-up.

Jung Haneul

If the fallback happens to land close, nobody even knows the measurement failed. That's the scariest part.