Exoplayer 2.13.0 HLS playback is jerky, same stream played smooth in 2.12.3 #8560
Comments
|
Thanks for reporting. We'll provide a fix for the problem. |
This may happen for HLS live streams without program date time information. Issue: #8560 #minor-release PiperOrigin-RevId: 356227729
|
Checked #8570, live playback speed workaround also worked for me: LivePlaybackSpeedControl speedControl = new DefaultLivePlaybackSpeedControl.Builder()
.setFallbackMinPlaybackSpeed(1f)
.setFallbackMaxPlaybackSpeed(1f)
.build();
|
|
The fix for this will be included in 2.13.1, at which point the workaround will become unnecessary. |
This may happen for HLS live streams without program date time information. Issue: #8560 PiperOrigin-RevId: 356227729
|
Thanks for the fix, it works on FullHD streams but the issue still appears on some 4K streams. If I force LivePlaybackSpeedControl Min/Max playback speed to 1f - playback is smooth. If use default (do not specify) - playback stutters periodically. |
|
Reopening to address the problem described above. Could you also let us know which device you are using? It might be related to the device using too much CPU during playback for speed adjustment.
Could you also try to turn this off to see if it makes a difference? |
|
I use NVidia Shield Pro 2019, lack of CPU power shouldn’t be a problem. Stuttering appear only in 4K streams, lower resolution is ok. Forcing LivePlaybackSpeedControl to 1f also solves the problem. |
|
I tested the stream mentioned in the issue description on a Sony Android TV (2020 model). The content is at 50fps and the TV screen refresh rate is at 60Hz. When playback speed is at 1.03x (1.03x chosen because it is the live speed controller's default max), I can see periodic frame drops. I haven't figured out the root cause yet and still investigating. [tracking progress] Before testing on other devices and only having seen the drops on the TV, I investigated the adjusted release timestamps, in case there was a bug in |
|
I was not able to detect -so far- a clear indication of why this happens on the TV I'm testing with. However, this device has frame drops for any playback speed higher than 1x, not only the content provided in this issue. Playing H264 content at 24fps and increasing the speed at any value higher than 1x seems to introduce "chopiness". @pvishnyakov Can you please play some content with the demo app and try speeds 1.25x and maybe 1.5x. For example, the demo app has HD content at 24 fps, playing at 1.25x should not exceed the device capabilities (in terms of decoding). You can manually change the playback speed tapping on the coq control, at the bottom right of the screen. Do you see any chopiness on your device too? |
|
I have tried all HLS streams in demo app with speed up to 2x (when possible, Apple master streams cannot change speed), playback is smooth, if we can talk about smoothness with the content like this. 4K 60fps DASH was pretty smooth with speeds up to 1.5x (2x is ugly). Clear DASH UHD 264/265 24fps playback is smooth even with 2x speed. |
|
I have compared 2 apps with one difference - SurfaceView vs. PlayerView. The issue described above appears in SurfaceView option with HLS 4K stream and TV screen refresh rate adjusted to match the video frame rate. If TV refresh rate is not matching video frame rate, playback is not smooth for FMP4 and HLS streams of all resolutions. In the app with PlayerView, MP4 streams are playing smooth without TV screen refresh rate adjustment, HLS not tested yet, going to check later. Update: tested HLS and FMP4 with PlayerView as well, without TV screen refresh rate adjustment the playback is stuttering. |
|
We cannot reproduce locally, maybe you can do some debugging steps on your side
You should then compare each printed You should then compare each If presentation times and release times are aligned with the content and playback speed, then this is an indication the frames are being dropped by the device components, which are outside ExoPlayer's control. |
|
I’ve made it using Demo app and the streams that have this issue. Ouput is generated with this code (called from the method you’ve recommended above) private void updateRenderDebug(long presentationTimeUs, long releaseTimeNs) {
if (null!=rdText) {
long ptAdvance = presentationTimeUs-lastPT;
long rtAdvance = releaseTimeNs-lastRT;
float calculatedFps = 1000000f/ptAdvance;
float nanoAdvance = 1000000000f/25f/ExoPlayerImplInternal.adjSpeed;
String text = "Adjusted speed = "+ ExoPlayerImplInternal.adjSpeed+", Frame rate = 25\n"+
"Presentation time = "+presentationTimeUs+", PT Advance = "+ptAdvance+" (as for "+calculatedFps+" fps)\n"+
"Release time = "+releaseTimeNs+", RT Advance = "+rtAdvance+" (must be "+nanoAdvance+")\n";
lastPT = presentationTimeUs;
lastRT = releaseTimeNs;
uiHandler.post(()->rdText.setText(text));
}
}
Presentation time advance is 40000 microseconds - correspond to 25 fps, ok; This is the video of the test, you can see the stutters as well (look at the grass beyond the water): IMG_0910.MOV |
|
Thank you for your assistance! I can see in your attached video that For completeness, will you be able to log the presentationTimeUs and releaseTimeNs for a few seconds (say, 5 seconds) and share the log? We can then see how often the player decides to display a frame sooner (to match the 1.03x) and whether there is any bug there. This should be enough I think that from the ExoPlayer side things are working as intended. I can also verify that the stuttering you are observing is the same as the stuttering I observed on my TV, where I can verify the frames are rendered on expected rendering timestamps. My working theory at the moment is that the device firmware cannot handle displaying content at an arbitrary frame-rate. I will forward the issue, at least for the TV I have access to, to the manufacturer. |
|
[internal ref b/184139564] |
|
I’ll gather the logs but MP4 or DASH file don’t have this issue, only live 4K HLS does. Non-live (VOD) HLS also fine, any resolution. You’ve said that presentation and release time should be aligned with playback speed and frame rate but in my test release time is not aligned, please pay attention. |
|
Done, first log is from the channel shown on the video posted before, stuttering one: Second log is from Full HD HLS live stream (same server), the video is pretty smooth, release time is changing but not the same as above, here is almost no shortened releases: The video used to record 2nd log is here, you can see there is no stutter: |
|
One more log for 4K HLS live stream, same source as before but this time the speed is not forced to 1.03 (let player adjust): |
|
Thank you for providing the logs. Long message to follow. On a high-level:
In more detail: As I explained above, the player will send frames to be rendered on screen on time slots that are matching the screen's refresh rate times (see VideoFrameReleaseHelper if interested). On a 50Hz refresh rate, the screen is refreshed every 20ms. To play a 25fps video at 1x speed, the player will send a frame to render every 2nd screen refresh slot (vsync), which is every 40ms. To play 25fps at 1.03x speed, the player needs to render 25.75 frames per second. Roughly speaking, the player should render 25 frames for the first second, and then 26 frames on the second second, because theoretically 2x25x1.03 = 51.5 frames. And so on. For this specific use case (25fps at 1.03x), the player should render most frames every 2 vsyncs (40ms), and occasionally send a frame sooner, i.e. on the next vsync which is in 20ms. Overall, if you look at the release timestamps on longer term, the average release time diff should be close to 38.83ms. I was not clear in my description of this here, I'm sorry. On 4K_HLS_frame_render.log, most "RT advance" is 40ms but for a specific frames "RT advance" is 20ms. For example lines 1, 18, 36. You can look at this from a different perspective: how many frames are rendered per second. Take the release time of the first frame (line 1, 773371601615011) and move down until 1 second has passed. On the 26th frame, 0.98 seconds have passed, and on the 27th frame, the difference is 1.02 seconds. So the player is sending more than 25 frames on screen. I think the release timestamps on 4K_HLS_frame_render.log show that the renderer is adapting to the 1.03x speed on the 50Hz screen as intended. For the FHD_HLS_frame_render.log however, the release time diffs are mostly 40ms, sometimes the diff is 60ms (the player delays rendering a frame for one more vsync slot), but it never sends a frame to be rendered after exactly one vsync. The release time stamps are actually for playback speed slightly lower that 1x. Are you certain you collected this log while playing content at 1.03x, hard-coded here? Did you add a log line as I described on this comment to ensure the speed change was effective? Was this live content, the same as the 4K one? For 4K_HLS_No_Forced_Speed_frame_render, most of the release timestamp diffs are 40ms, it looks like the player is not adjusting the speed (it doesn't have to if it doesn't detect it needs to). You mentioned that you observed no stuttering with MP4, DASH and non-live (VOD) HLS. The logic of automatic speed adjustment applies only to live streams and the code-change I pointed in this comment applies for live streams only. Are you certain that in your tests, the player was playing at 1.03x? With regards to other questions:
This is not 100% accurate for the issue here. For live content, unless disabled by the app, the player might automatically adjust playback speed and this affects the Renderer.
I'm assuming you are interested for HLS where you observe the frame drops. This method is called before downloading an HLS chunk, you can log the Overall: Just to clarify, at this point you observe the issue only with 4K live HLS streams. Is the following still true?
Considering that we do not have access to the specific content and we cannot reproduce locally, it's hard for us to offer better help. Would you consider disabling automatic live speed adjustment as above for the 4K Live streams on your specific device? |
|
private void updatePlaybackPositions() throws ExoPlaybackException {
MediaPeriodHolder playingPeriodHolder = queue.getPlayingPeriod();
if (playingPeriodHolder == null) {
return;
}
// Update the playback position.
long discontinuityPositionUs =
playingPeriodHolder.prepared
? playingPeriodHolder.mediaPeriod.readDiscontinuity()
: C.TIME_UNSET;
if (discontinuityPositionUs != C.TIME_UNSET) {
resetRendererPosition(discontinuityPositionUs);
// A MediaPeriod may report a discontinuity at the current playback position to ensure the
// renderers are flushed. Only report the discontinuity externally if the position changed.
if (discontinuityPositionUs != playbackInfo.positionUs) {
playbackInfo =
handlePositionDiscontinuity(
playbackInfo.periodId,
discontinuityPositionUs,
playbackInfo.requestedContentPositionUs);
playbackInfoUpdate.setPositionDiscontinuity(Player.DISCONTINUITY_REASON_INTERNAL);
}
} else {
rendererPositionUs =
mediaClock.syncAndGetPositionUs(
/* isReadingAhead= */ playingPeriodHolder != queue.getReadingPeriod());
long periodPositionUs = playingPeriodHolder.toPeriodTime(rendererPositionUs);
maybeTriggerPendingMessages(playbackInfo.positionUs, periodPositionUs);
playbackInfo.positionUs = periodPositionUs;
}
// Update the buffered position and total buffered duration.
MediaPeriodHolder loadingPeriod = queue.getLoadingPeriod();
playbackInfo.bufferedPositionUs = loadingPeriod.getBufferedPositionUs();
playbackInfo.totalBufferedDurationUs = getTotalBufferedDurationUs();
// Adjust live playback speed to new position.
if (playbackInfo.playWhenReady
&& playbackInfo.playbackState == Player.STATE_READY
&& shouldUseLivePlaybackSpeedControl(playbackInfo.timeline, playbackInfo.periodId)
&& playbackInfo.playbackParameters.speed == 1f) {
float adjustedSpeed = overrideAdjSpeed?adjSpeed:
livePlaybackSpeedControl.getAdjustedPlaybackSpeed(
getCurrentLiveOffsetUs(), getTotalBufferedDurationUs());
if (mediaClock.getPlaybackParameters().speed != adjustedSpeed) {
mediaClock.setPlaybackParameters(playbackInfo.playbackParameters.withSpeed(adjustedSpeed));
handlePlaybackParameters(
playbackInfo.playbackParameters,
/* currentPlaybackSpeed= */ mediaClock.getPlaybackParameters().speed,
/* updatePlaybackInfo= */ false,
/* acknowledgeCommand= */ false);
}
}
}
public static final float adjSpeed = 1.03f;
public static final boolean overrideAdjSpeed = true;First 2 logs (4K_HLS_frame_render and FHD_HLS_frame_render) were recorded with Last log (4K_HLS_No_Forced_Speed_frame_render) was recorded with
protected void renderOutputBufferV21(
MediaCodecAdapter codec, int index, long presentationTimeUs, long releaseTimeNs) {
maybeNotifyVideoSizeChanged();
TraceUtil.beginSection("releaseOutputBuffer");
codec.releaseOutputBuffer(index, releaseTimeNs);
TraceUtil.endSection();
lastRenderRealtimeUs = SystemClock.elapsedRealtime() * 1000;
decoderCounters.renderedOutputBufferCount++;
consecutiveDroppedFrameCount = 0;
maybeNotifyRenderedFirstFrame();
updateRenderDebug(presentationTimeUs, releaseTimeNs);
}
public static int rdTextId = -1;
public static TextView rdText = null;
private static final Handler uiHandler = new Handler(Looper.getMainLooper());
private static long lastPT = -1;
private static long lastRT = -1;
private static final StringBuilder stringBuilder = new StringBuilder();
private static int maxCount = 250;
private static int countUp = 500;
private SimpleDateFormat sdf = new SimpleDateFormat("hh:mm:ss");
private void updateRenderDebug(long presentationTimeUs, long releaseTimeNs) {
if (null!=rdText) {
long ptAdvance = presentationTimeUs-lastPT;
long rtAdvance = releaseTimeNs-lastRT;
float calculatedFps = 1000000f/ptAdvance;
float nanoAdvance = 1000000000f/25f/ExoPlayerImplInternal.adjSpeed;
String text = "Adjusted speed = "+ ExoPlayerImplInternal.adjSpeed+", Frame rate = 25\n"+
"Presentation time = "+presentationTimeUs+", PT Advance = "+ptAdvance+" (as for "+calculatedFps+" fps)\n"+
"Release time = "+releaseTimeNs+", RT Advance = "+rtAdvance+" (must be "+nanoAdvance+")\n";
if (countUp>0) {
countUp--;
} else {
if (maxCount > 0) {
addLog(text);
maxCount--;
} else {
try (FileOutputStream fos = context
.openFileOutput("frame_render.log", Context.MODE_PRIVATE)) {
byte[] buf = stringBuilder.toString().getBytes(StandardCharsets.UTF_8);
fos.write(buf, 0, buf.length);
fos.flush();
} catch (Exception e) {
e.printStackTrace();
}
}
}
lastPT = presentationTimeUs;
lastRT = releaseTimeNs;
uiHandler.post(()->rdText.setText(text));
}
}
private void addLog(final String text) {
String s = /*sdf.format(new Date())+">> "+*/text.replaceAll("\n", "; ")+"\n";
stringBuilder.append(s);
}
Yes, it still is, I use this workaround in my live streaming app when the screen refresh rate is adjusted to the video FPS. |
|
Hi.
After I’ve checked some other streams I’ve found that the problem happens only with some specific streams, not with any 4K HLS live stream.
This is modified video source file from demo app: see my email
The issue is observed on first 3 streams (the first one - Love Nature 4K - is the most visible), others in my HLS collection are playing smooth. To check first 3 streams please try to modify the USER_AGENT in DemoUtil.java as follows, it worked in my case:
private static final String USER_AGENT = "Mozilla/5.0 (iPhone; CPU iPhone OS 13_4 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/80.0.3987.95 Mobile/15E148 Safari/604.1";
This is last resort, if doesn’t work - let’s leave it for better time, it’s not so critical, finally there is a workaround.
|
|
I have sent the stream sources to email, please check |
|
Hi, Thank you for sending the content. I tested the streams on a few mobile devices, a ChromeCast and an Android TV. Only the TV showed the stuttering, but this TV is stuttering on all kinds of content. All other devices played content smoothly on different playback speeds. Considering that overriding the playback adjustment on your app seems to solve the problem, I'd say you use that. From the data collected so far, I believe the issue is caused by a component outside of the player's control and is triggered when the playback speed is different than 1x. At the moment, I can't offer advice other than the workaround you have applied. Some extra details that might be helpful if you investigate further:
|
Hi.
I’m using Exoplayer in my streaming app for Android TV, I have Nvidia Shield Pro 2019 as testing device, Android TV v.9 installed.
After upgrading Exoplayer library to the latest version 2.13.0 I’ve noticed that HLS streams are not playing smoothly while same streams with same settings on the same device played smoothly with 2.12.3 and before.
I’ve tried different streams from different sources, they all behave in a similar way.
Try this stream for example - there is a marquee below, in 2.12.3 and before it was flawlessly smooth, in 2.13.0 it stutters every second or about that.
SurfaceView is used to show video, TV screen refresh is adjusted to video frame rate.
Studied Release Notes, documentation and sample app, cannot find what’s wrong, can you please help?
Player initialization code is below:
The text was updated successfully, but these errors were encountered: