Detect A/V out of sync.

[changxiangzhong]

We have a few reports regarding A/V out of sync issue, it's rarely reproducible, but we want to capture it in our analytics system, to understand how bad it is. I've looked into EventLogger and ExoPlayer's internal Listener, I couldn't find anything.

Could you please shred some light on out of sync detection?

[icbaker]

Does the player know the audio and video are out of sync?

Generally ExoPlayer runs the playback clock based on timestamps from the audio track, and aims to release video frames to the screen so they are in sync with the corresponding audio timestamp.

If you see A/V out of sync some possible causes are:

1. Video decoding is too slow, so the frames aren't ready when they need to be released to the screen, so they are released late. In this case the player knows it's out of sync. The demo app displays 'video frame processing offset' (vfpo`) in its debug text at the top of the playback screen which measures this. See more info here: https://medium.com/google-exoplayer/improved-rendering-performance-operating-mediacodec-in-asynchronous-mode-and-asynchronous-buffer-3026207850b2
2. The media has incorrect/mismatched timestamps, so the player does everything "right" but the user perceives the audio and video are not in sync.
3. There's a bug in the player logic which is aiming to keep audio and video in sync (and so by definition, the player doesn't know the audio and video are out sync).

In case 1, you can measure vfpo from DecoderCounters in the same way as the demo app.

In case 2 and 3, there's not really a way to measure this from within ExoPlayer, because if the player was able to measure a sync problem (and we're not performance-constrained, i.e. case (1)), it would just fix it.

[changxiangzhong]

Thank you so much for your explanation. We'll collect vfpo in analytic system

[changxiangzhong]

Thanks you so much Ian, I have one more question regarding this vfpo. I understand this

vfpo = (time when a frame being decoded) - (time when the frame being scheduled to render)

At which number shall we anticipate an A/V out of sync? Minus value? less than 10ms? Or at which number shall we alter the analytics system?

[icbaker]

Defining the threshold of what you consider "out of sync" is up to you I think. It probably depends a bit on the content too? e.g. in roughly descending order of visibility of sync issues:

1. An A/V sync test video with clicks aligned with clear visual cues
2. Content with lots of speech with visible mouths/lips
3. Content with a voiceover or soundtrack that doesn't directly reflect what's visible on-screen

So there's no single value we can tell you "small than this is in sync, larger than this is out of sync"