Question about onRenderedFirstFrame

[botondbutuza]

Hey guys!

More of a question than a bug. I have a player and a hidden surface. I want to show the surface the second the player is actually rendering onto it continuously.

I looked at EventListener's onPlaybackStateChanged(), and I get a callback when the player is ready, but it being ready and it rendering are two different things - it needs a keyframe to start rendering, which might be 200 milliseconds away.

Next, I looked at VideoListener's onRenderedFirstFrame() - aptly named, I figured that's what I need. Alas, there's still a visible delay between my getting that callback, and the video actually rendering on my surface.

Edit: Same with VideoRendererEventListener's onRenderedFirstFrame().

I wonder if there's a callback I am not aware of, or if there's any way of sidestepping that tiny delay I'm seeing.

Using v2.2.0, HlsMediaSource, API 22 on a Samsung SM-J200M.

Thanks!

[ojw28]

onRenderedFirstFrame is what you need. What exactly is a "hidden surface", and how are you hiding it and making it visible?

[botondbutuza]

It's actually a TextureView, not a surface, and I'm setting its visibility from VISIBLE to INVISIBLE.

I basically have two TextureViews playing two videos, and I only want to show the second TextureView once the second video is rendering onto it. If I show it when I get the onRenderedFirstFrame() callback, there's still a tiny delay between it being shown and the video rendering continuously.

I really appreciate your help with this. :)

[ojw28]

To help with debugging, I'd suggest you try posting a delayed message/runnable in onRenderedFirstFrame, that makes the TextureView visible 500ms later than if you were to make it visible immediately at that point.

Is there still an issue when the TextureView eventually is made visible? ExoPlayer will definitely have released multiple frames to the Surface by that point, so if you're still seeing the same problem then it seems likely the effect you're seeing is some lower level graphical thing around the visibility of the TextureView, that we probably don't have control over, or an issue in your application code.

If you're not seeing the problem when you add the delay, how far can you reduce the delay until the problem re-appears? If delaying 16ms or 32ms (i.e. one or two VSYNCs) reliably avoids the problem, then pragmatically I'd suggest simply doing that. If it can be proven that there's potentially a 1 or 2 VSYNC delay for the frame to be made visible, we could potentially consider adding such a delay prior to firing onRenderedFirstFrame. I think we'd need to understand what the delay is first, however.

[botondbutuza]

Okay, so here are my findings:

• with a delay of 500ms, the video is rendering correctly
• I can reduce this to 400ms, and the video is still mostly correct
• anything below 400ms and there's a noticeable delay between

Our streams have a keyframe every 0.52 seconds, and Exo waits for the first keyframe before rendering, right? Is that being taken into account when onRenderedFirstFrame is sent? That's the only thing I can think.

[ojw28]

onRenderedFirstFrame is triggered at the point where a buffer has been output from the decoder and released to the surface. So yes, it should be taken into account.

[botondbutuza]

There's a consistent 200-300ms delay between setPlayWhenReady and onRenderedFirstFrame, which is pretty much what I expected. Any ideas?

[ojw28]

Not really. Sorry! Does this happen on all devices? Is it possible for you to provide a minimal change to the ExoPlayer demo app that allows us to easily reproduce the issue?

[botondbutuza]

@ojw28 Quick q: once the player is in STATE_READY, and onRenderedFirstFrame is called, content should render continuously, right? There's no additional check or something I have to wait for?

Because there is a delay with my HLS media, and I cannot find the cause of it.

[ojw28]

We always render the first frame as soon as possible. For HLS it's possible that a segment doesn't have a key-frame near the start. It the chunk spans time [5-10], the user seeks to t=6 and the first key-frame in the chunk is at t=9, then we'll request the [5-10] chunk and find that the first frame we can actually display is the key-frame at t=9. Hence we'll display this frame, then not render any more frames until audio actually reaches t=9, then play audio and video together from that point.

This is not a problem if key-frames are positioned near the start of each segment (which I think is a recommendation somewhere in one of Apple's docs). In both DASH and SmoothStreaming it's required (in practice at least) that every segment begin with a key-frame.

[botondbutuza]

Ah okay, so that explains why audio plays with no delay whatsoever, but the video doesn't. Is there a way for me to get a callback or find out when video playback reached the keyframe and is continuing together with audio?

I want to display the surface when the video is actually playing, and neither onRenderedFirstFrame nor onPlayerStateChanged actually offers this information.

I really appreciate your continued support on this. :)

[botondbutuza]

@ojw28 Sorry mate, any update/thought/views on this?

[ojw28]

Not really, sorry. You'll probably have to do some more digging yourself. Maybe if we were to report the presentation timestamp of the rendered frame via onRenderedFirstFrame, you'd be able to compare that against the current playback position. When the playback position starts to exceed the presentation timestamp you'll know that frames are being rendered regularly. You could try making that change; I don't see any reason not to accept a pull request doing that if it helps.

It's also possible that when the first frame gets rendered may change slightly as fallout from some changes being made related to #2582. It's somewhat unclear to me how they'll affect the issue described here. Those changes will likely land in the next few weeks, so might be worth re-evaluating this with the latest dev-v2 after that issue has been closed.