When iPhone video sync keeps jumping and stays 0.5 seconds behind

This is a translated article. Read the original in Korean ↗

We were building a web app (formation) for dance rehearsals: a leader's laptop plays a choreography video with music, while members use their own phones to follow the same point in the video and the formation (movement paths). On an iPhone 12 Pro, the video would repeatedly sync with the leader, then stutter and fall behind. Working from the hypothesis that the leader advances while a seek (an instant jump to a playback position) completes, we added compensation to the seek target and received confirmation that it worked. However, this was not an experiment that measured seek duration itself or fully ruled out device performance as a cause.

Conclusion

If an iPhone's <video> keeps drifting out of sync with the leader and stuttering, playback-rate correction and preloading the stream did not help in this case. What worked was moving the seek target ahead from the leader's estimated current playback time, exp, to exp + seekComp. Switching to preloading (blob) left the stuttering unchanged , and removing rate correction or changing it to ±6% still left stuttering or uncorrected drift . When the error observed with ?debug=1 stayed at 0.45–0.5 seconds, we hypothesized that the delay until each seek finished was leaving the viewer behind every time. After implementing compensation that started at 0.45 seconds and updated from the error remaining after each seek, we received an "It worked!" from the iPhone user.

Expressed as formulas, the core change looks like this. This summarizes the implementation description rather than reconstructing the actual source.

이전 seek 목표 = exp
변경 seek 목표 = exp + seekComp
seekComp 초기값 = 0.45초
이후 seek 착지 잔차로 seekComp 갱신

exp is the leader's current playback time as estimated by the viewer from sync messages. The record only mentions smoothing received samples with an EMA (exponential moving average); it does not preserve the coefficient . The sketch below is an example for anyone implementing the same idea, not code reconstructed from the record. You will need to tune the coefficient 0.5 and the measurement timing yourself.

// Example (not a reconstruction): estimate leader time + learn seekComp
// In applySync: smooth the offset using received leader time t and receipt time now (seconds)
offset = offset === null ? t - now : offset * 0.8 + (t - now) * 0.2;
const exp = () => performance.now() / 1000 + offset;   // Estimated current leader time

function hardSeek() {
  video.currentTime = exp() + seekComp;
  landAt = performance.now() + 800;   // Measure landing error 0.8 seconds after the request (assumption)
}
// When not paused, once after landAt:
const resid = exp() - video.currentTime;     // Positive = still behind
seekComp = Math.max(0, seekComp + resid * 0.5);

0.47 seconds was not a direct measurement of seek duration. What we actually observed was a range of sync errors; interpreting it as seek delay was a hypothesis. The actual post-fix Δ and comp values were not recorded. We therefore cannot claim that the error fell to 0.05 seconds on the physical device.

Situation

We were building real-time rehearsal-room synchronization for formation, a dance formation and movement-path editor. The setup has three parts.

• Leader page: The leader's laptop plays a choreography video with music and sends its current playback time and state to the server every 200ms (five times per second).
• Server: A Node server running on the laptop. It relays the leader's sync messages to all viewers over SSE (a mechanism for the server to continuously push events to browsers) and serves the video file.
• Viewer page (viewer.html): Runs on members' phones. It estimates the leader's current playback time (exp) from received sync messages and aligns the phone's <video> with that time. A canvas draws the formation (movement paths) over the video. There was also a mode that turned off the video and showed only movement paths.

The problem was in the viewer's correction logic for aligning <video> with the leader. There are two tools available when the video drifts away from the leader. One is to adjust playbackRate (playback speed) slightly to catch up gradually; the other is to seek by changing currentTime, jumping to the target position.

Symptom

Even a 360p streaming video stuttered intermittently on the iPhone 12 Pro. Preloading it before playback made no difference, while the mode that removed the video and showed only movement paths was smooth. The recurring symptom was: "Just when it seems in sync, it stutters again and falls behind."

We initially suspected rebuffering during streaming seeks and tried preloading. But the very next test reproduced the problem. This was only a clue that streaming alone could not readily explain the symptom; it was not an experiment that tested and ruled out decoding, memory, and device performance individually. The investigation shifted toward video sync correction.

Even then, each change to the correction method left a different problem.

Observations and hypotheses that narrowed down the cause

The turning point was the question, "Can we check whether the gap is growing or shrinking?" In response, we added a ?debug=1 display showing the error Δ, whether it was growing, shrinking, or holding steady, the playback rate, and the seek threshold. Under the definition used at the time, a positive Δ meant the viewer was behind the leader. We also added a guard against seeking while a seek was already in progress, and a condition that allowed seeking only when the gap was growing.

The value seen on the iPhone "kept going back and forth between 0.45 and 0.5." The observation was that the error stayed within a fixed range rather than accumulating. We interpreted that range as an error floor left by seek delay.

Here is that interpretation as a simple model at 1× playback speed. Suppose a seek to 10.00 seconds is requested when the leader is at that position, and the viewer takes 0.47 seconds to resume playback there. Meanwhile, the leader has reached 10.47 seconds. The viewer has reached the target specified when the request was made, but by the time it arrives, it is already 0.47 seconds behind again. This is an assumption used to explain the cause, not a result measured with a separate timer.

In this model, the 0.35-second threshold is smaller than the remaining error, so correction repeats indefinitely. Raising the threshold, on the other hand, reduces jumps without closing the gap. A change raising the threshold to 1.0 seconds was reported first, with the expectation that playback would stay smooth while remaining 0.5 seconds behind. No separate result from a physical device was reported for this step.

There was also a proposal to delay the leader by 0.5 seconds. It was rejected on the grounds that delaying the reference would leave the viewer lagging behind that reference again, preserving the relative gap. What was considered here was delaying the broadcast reference or playback as a whole; this did not establish that delaying only the audio would also be impossible. The final goal also became clear: a 0.5-second offset in movement paths was acceptable, but the music and the movements in the video had to match, and half a second is a large difference in a dance video.

The observation that playback-rate correction caused stuttering also needs to be read within its limits. The physical-device test with hysteresis failed, but it did not establish that "anything other than 1× speed always stutters on an iPhone." An experiment with preservesPitch=false, mentioned as an alternative, was not carried out during this fix either.

Fix

We changed the target from the leader's position when the seek is requested to the position the leader is expected to reach when the seek finishes. According to the implementation description, seekComp was added to the target in hardSeek, and the compensation was then learned from the residual error after landing. It was designed to account for that residual rather than apply a fixed 0.5 seconds to every device.

The settings at the time were as follows. These numbers summarize that implementation; they were not verified as optimal for other devices.

All the record preserves is the description "learn seekComp from the landing residual" and the timing "measure 0.8 seconds after the seek." The actual update coefficient, bounds, and reference point for those 0.8 seconds (the seek request or the seeked event) were not preserved . The sketch above fills in that gap by using the request time as its reference. To implement the same approach, you need to log the seek request time and the residual 0.8 seconds later yourself, then choose an update coefficient. Implementers also need to check whether the residual could be distorted if a slow device does not finish seeking within 0.8 seconds. This is a caveat, not a fact established at the time.

Looking only at the arithmetic, if the actual delay is 0.5 seconds and seekComp is 0.45 seconds, the first seek leaves a residual of 0.05 seconds.

In the simulation, setting the delay to 0.5 seconds and applying 0.45 seconds of compensation produced a first-seek residual of 0.05 seconds. The "10× improvement" was a simulation result. Confirmation on the actual device was "It worked!"; post-fix Δ and comp values were not recorded. The Δ +0.03s and comp 0.47 shown in the example debug display were not measurements either.

Afterward, the decision was made to remove the movement-paths-only button because there no longer seemed to be a need to turn off the video. That supports the judgment that the issue felt resolved, but it does not quantify the remaining error or prove the accuracy of simultaneous playback across multiple devices. The confirmed outcome in this case goes as far as receiving confirmation that the compensation resolved the problem.

Reporter's Notes — arrived, but late again

The script below reconstructs the investigation through four characters.

Park Doeun

You know that web app for rehearsals, where the leader's laptop plays a choreography video with music and everyone follows the same scene and formation on their own phone? On my iPhone 12 Pro, the video keeps syncing with the leader, then stuttering and falling behind again.

Jung Haneul

Right, each phone makes its own <video> follow the playback time it receives from the leader. When it drifts, we adjust the speed slightly or seek straight to that position.

Jung Bada

If it's stuttering at 360p, streaming sounds like the culprit. That brief pause while it downloads the relevant chunk again on each seek is called rebuffering. Let's preload the file on the phone and see whether it goes away.

Jung Haneul

Then two tests should help distinguish things. Try playing the preloaded file, and compare it with the mode that turns off the video and draws only movement paths.

Park Doeun

Tried both. It stutters just the same when preloaded, and the movement-paths-only mode is perfectly smooth, with no stuttering at all.

Jung Haneul

Since it stutters even when preloaded, video streaming alone doesn't readily explain it. The movement-paths-only mode draws directly at the leader's time, so next let's change the logic that corrects <video> and compare.

Park Dohyun

The prediction that preloading alone would fix it was wrong. The comparison narrowed down what to check next, but it didn't rule out decoding or device performance entirely.

Jung Bada

Then let's look at rate correction. Whenever even a small error appears, we're constantly nudging playbackRate, the playback speed. Let's remove rate correction and jump only when the gap exceeds 0.75 seconds.

Park Doeun

The jumps are gone. But it stays a little behind the leader and never closes the gap. Looks like it leaves it alone because it's within 0.75 seconds.

Park Dohyun

Reducing jumps alone hasn't fixed synchronization. This time we need to check whether we can reduce small errors too, and whether that introduces stuttering.

Jung Bada

Then let's speed it up by just 6%, changing the rate once when we start catching up and once when we're fully caught up. Using separate thresholds for switching on and off like this is called hysteresis. In the simulation, it converged over nine seconds with two rate changes.

Park Doeun

On the phone, it's very choppy. Completely different from the simulation.

Park Dohyun

The simulation only shows whether the logic runs as designed. You have to run it on the phone to find out how the device handles that playback rate.

Jung Haneul

Even with infrequent rate changes, this test still stuttered. Let's remove this correction method for now.

Jung Bada

Let's give up on rate correction for this attempt. Lock it to 1× and lower the jump threshold to 0.35 seconds. At least that should close the gap.

Park Doeun

Now it keeps jumping nonstop. It seems like it can never get below 0.35. Could we first see whether the error is growing or shrinking?

Jung Haneul

Good, let's look at numbers instead of guessing. I'll make the error from the leader, Δ, and its trend appear at the bottom of the screen when you append ?debug=1 to the URL. A positive Δ means the phone is behind.

Park Doeun

Checked it. It keeps going back and forth between 0.45 and 0.5 seconds. It isn't growing, and it isn't dropping toward zero either.

Jung Haneul

Staying in one range is a clue. Suppose we jump to 10 seconds when the leader is at 10 seconds, but it takes about 0.47 seconds to arrive. By then, the leader is already at 10.47 seconds. Since the threshold is 0.35, we jump again and arrive late again.

Park Dohyun

Under that hypothesis, the 0.35-second threshold is smaller than the remaining error, so it keeps triggering jumps. But what we observed was an error range. We haven't directly measured whether 0.47 seconds is the actual seek duration.

Jung Bada

Then let's raise the threshold to one second and let it stay smooth while half a second behind. Or couldn't we just delay the leader by 0.5 seconds so it looks like the phone has caught up?

Jung Haneul

Raising the threshold stops the jumps, but the gap stays. If we delay playback and broadcasting together, the phone follows that reference too, so the gap would stay. Delaying only the audio is a different approach, but let's fix the phone's correction first here.

Park Dohyun

Besides, our goal isn't to eliminate stuttering; it's to align the music with the movements in the video. A half-second error in movement paths is acceptable, but in dance, half a second between the music and the movement is far too much.

Jung Haneul

Then let's change the destination. Instead of jumping to the leader's current position, exp, we jump to exp + seekComp, where the leader should be when the jump finishes. Start seekComp at 0.45 seconds, then keep updating it from the error left after arrival.

Park Doeun

It works now! I don't think we need to turn off the video and look only at movement paths anymore. We can remove that button.

Jung Haneul

Let's keep the distinctions clear when we write this up. The 0.47 seconds came from interpreting the range where the error stayed, not directly timing the jump. Getting down to 0.05 seconds was a simulation result too, and we didn't record the actual Δ after the fix.

Park Dohyun

And while we got an "It worked" after seeking ahead this time, that doesn't prove seek delay was the cause. To establish the cause, we need to measure directly from the start of the seek to the seeked event.

Jung Bada

So I was wrong to blame streaming at first, wrong to blame playback rate, and our current hypothesis is still just a hypothesis. I've learned that a plausible symptom doesn't establish the cause.