Change playback rate (video + audio only use cases)

[viacheslavokolitiy]

Hello guys. Is it possible to change playback rate of video track e.g play faster or play slow ?

[runningcode]

+1 this would be very useful!

[ojw28]

To clarify, do you mean with audio (sped up but corrected to still be at the correct pitch) or without?

[viacheslavokolitiy]

@ojw28 with audio

[ojw28]

This isn't something that's supported currently. It is something you could potentially do yourself, either by extending MediaCodecAudioTrackRenderer, or by creating your own audio renderer to use instead of the one provided by the library.

Basically, you'd need to do some adjustment in your extension to get the audio playing at the desired rate (and reporting time as progressing at that rate). This may involve pitch correction. The video track will follow the speed of the audio track (provided it can decode fast enough).

[lsjwzh]

+1

[shukob]

+1

[vishwa-vyom]

+1

[emakovsky]

+1

[shukob]

I need this functionality for my current project and used libSonic to change audio speed as in Prestissimo's Track.java.

Changing reported time part requires multiplying result of framesToDurationUs and durationUsToFrame, and conditions regarding MIN_PLAYHEAD_OFFSET_SAMPLE_INTERVAL_US and MIN_TIMESTAMP_SAMPLE_INTERVAL_US by current speed on MediaCodecAudioTrackRenderer.

I think I should multiply other Usec numbers in both renderers to work without flaw.
As I had to comment out dropOutputBuffer section in processOutputBuffer on MediaCodecVideoTrackRenderer to correctly refresh video frame.

Also
" // TODO: Don't bother doing this once [redacted] is fixed.
// Work out how many bytes we can write without the risk of blocking."
part in processOutputBuffer may need some edit when actually written byte length differs from bytesToWrite because of changing pitch?

[evanhalley]

@skonb Can you provide more details. I'm attempting to incorporate variable speed playback in MediaCodecAudioTrackRenderer (subclassed it, overriding the processOutputBuffer function) . I've done some of it (passing the byte buffer into and out of sonic), but even as a speed and pitch of 1.0f, it's really static-ky. Thanks!

[shukob]

@emuneee I have uploaded current implementation here. That is based on Sep 8 version of ExoPlayer so may not work with the latest one.
It contains bug that getCurrentPositionUs returns larger/smaller value than expected if speed is up/down.

[kboyarshinov]

+1

[kboyarshinov]

@skonb Have you tried to update your implementation to latest ExoPlayer version? Thanks!

[shukob]

@kboyarshinov No. I am now working on the latest ExoPlayer, but just saw it is almost not working on 5.0. So will check on 4.x and try to port above implementation if it reliably works on them in a few days. Thanks.

[mikumi]

@skonb In my project we use a similar implementation with libsonic based on August 15 version of ExoPlayer. Currently I am trying to port it to the newest ExoPlayer release as the older version has problems with webm/ogg files. But no luck so far. Have you made progress porting yours?

[shukob]

@mikumi
I'm now working on it. It seems that at least on 4.2.2, above implementation based on Sep 8 works, but on 5.0.1, audios do not slows down. I tried to port it within exoplayer.audio.AudioTrack to preserve new AudioTrackRenderer structure without success..., I have not investigated about whether Android or ExoPlayer that has problems on 5.x and if new AudioTrack-AudioTrackRenderer implementation is related to playing webm/ogg files.

Regardless of Exoplayer version, if playback speed changes while playing, getPlaybackPositionUs must be completely rewritten to accumulate played frame counts on above implementation.

[mikumi]

@skonb Thanks for the update. I realized that the webm/ogg issues happened because of another modification we had in our project, it was unrelated to ExoPlayer version.
I have ported our previous modification to the new ExoPlayer version by mainly changing the AudioTrack and it's working, but it does not support changing speed while playing because as you said, we would have to rewrite how to calculate the playing position.
I am posting the link here but don't really recommend to use it like that, as it is modified to just suit our needs:
Invisibi#1

[shukob]

@mikumi Thanks for the info! I'll check it : )

[PaulWoitaschek]

Are there any news on this? I tried the last implementation, but it gave really bad sound. I think this is somethig that would also be great to have ExoPlayer implement it natively.

[Ood-Tsen]

We may add a new TrackRenderer as new time source and disable audioTrackRenderer by overwrite the function isTimerSource.

boolean isTimerSource() {
}

And change the play rate by overwrite function "getCurrentPositionUs"

long getCurrentPositionUs(){
}

[PaulWoitaschek]

Can you do that? Is that a big task? Im the developer of Material Audiobook Player and this whole audio-internal stuff goes way beyond my capabilities ;-)

[crossle]

+1

[Ood-Tsen]

@Ph1b This concept is for video playback, video renderer will sync video frame to the clock automatically.

But for audio is not that simple.
We need to resampling audio raw data.

[kkovach]

+1 Would love to see this as an official feature.

[jacoviza]

+1

[danikula]

+1

[Arqu]

+1

[PaulWoitaschek]

@piotrmadry its Q1 (friday)

[ojw28]

The work should be complete (or nearly complete) in dev-v2 by that point, but it may take a week or two extra before we cut a release that includes it.

[ojw28]

Playback rate and pitch adjustments are seamless again as of f7fff0d. The feature is now at the level of our previous (API 23+ only) support, only now across all API levels. Expect a release in the next week or two as mentioned above.

Subsequent changes will follow that propagate the playback rate to some other components that would benefit from it (e.g. the ABR algorithm, in the case of adaptive playbacks).

[workspace]

I've applied the new playback speed control on the dev-v2 branch, but the sync of video and audio did not fit well. I wonder if you are aware of this problem. (x2)

[andrewlewis]

@KiminRyu I'm not aware of any issues with audio/video synchronization. If you file a new issue including steps to reproduce the problem we will take a look. Thanks.

[ojw28]

Apologies we've not managed to provide a release version containing this functionality yet. The functionality will be in the 2.4 release, but we're having to wait for a few other changes to land before we can publish it. We're now targeting toward the middle of next week. Thanks!

[KirillMakarov]

@ojw28 can you notify us in this issue, when the feature will be released?

[PaulWoitaschek]

@KirillMakarov GitHub has an atom feed you can add to your RSS reader:
https://github.com/google/ExoPlayer/releases.atom

[ojw28]

You can also watch https://bintray.com/google/exoplayer/exoplayer. There are quite a few ways to be notified of new releases without us having to update issues manually ;).

[workspace]

@ojw28
The new playback speed change method seems to be heavily influenced by CPU performance. I just put this code in demo when player is initialized.

player.setPlaybackParameters(new PlaybackParameters(2.0f, 1.0f));

For PlaybackParams added in API 23, the video played back successfully on Nexus 5. However, with the new PlaybackParameter, frame drops occur on devices with low CPU performance such as Nexus 5 and Galaxy S4.

Nexus 5 with API 23
Galaxy S4 with API 21

However, the Nexus 5X, Galaxy Note 5 and Galaxy S8 Plus (Android N devices) played well without problems. I played demo in the API 23 version of the emulator to see if it was a difference in the Android API.
 
Nexus 5x emulator with API 23

So I'm worried that the new playback speed change method will work well on various devices.

Error Log in Android Studio

04-21 16:56:50.368 20130-20130/com.google.android.exoplayer2.demo E/EventLogger: internalError [21.31, audioTrackUnderrun [56448, 320, 27]]
04-21 16:56:50.443 20130-20130/com.google.android.exoplayer2.demo E/EventLogger: internalError [21.38, audioTrackUnderrun [56448, 320, 1]]
04-21 16:56:50.618 20130-20130/com.google.android.exoplayer2.demo E/EventLogger: internalError [21.55, audioTrackUnderrun [56448, 320, 22]]

[andrewlewis]

@KiminRyu Did you see these problems when running a release APK or a debuggable APK?

The new implementation uses the Java version of Sonic, so its performance is affected by what optimizations the Java runtime applies. There is a certain optimization that is turned off on debuggable builds, which causes an inner loop in Sonic to be very slow. If you build a release APK this optimization should apply (on all API versions) and I found that performance was fine on various devices I tested, including a Samsung Galaxy S4 running Android 4.2.2 (API 17).

On API 24 Android builds onwards, performance should be fine on both debuggable and release builds.

[workspace]

I got the latest version of the dev-v2 branch and built it. (so it is a debuggable apk.) So...How can I set the same environment with a release apk?

[andrewlewis]

I used the instructions for Android Studio: Sign your release build.

[workspace]

I built it with the release version, and the playback speed change worked just as you said. As shown in the video attached below, playback of Secure_xx (SD, HD, UHD) of Widevine DASH: MP4, H264 sample was not smooth. On the other hand, clear_xx samples of the same widevine drm sample worked well, and Youtube DASH also worked.

widevine secure hd -> clear hd -> youtube dash

[PaulWoitaschek]

Thanks for the release. It works really great!

[workspace]

@andrewlewis
As I mentioned above, when playing from 2.4.0 version with widevine secure path, the screen stops immediately after 10 seconds. NEXUS 5 (23), Galaxy Note 5 (24), Galaxy S8 Plus(24) and other API 23 and above work normally. On the other hand, older devices such as the Galaxy S4 (21), Galaxy Note 2 (19) and Galaxy Note (16) have a problem in changing the playback speed faster. I have built and tested a new 2.4.0 release apk.

[andrewlewis]

@PaulWoitaschek Glad to hear it's working!

@KiminRyu Please could you file a new issue to track that, including all the details requested in the new issue template? Thanks.

I'm closing this issue now as the core functionality is implemented, but as @ojw28 noted above further changes will be needed to propagate the playback speed to some other player components (like the adaptive track selection logic and LoadControl). In case it's useful: there's a blog post summarizing how to use this feature.

[gemiren]

I am seeing high CPU usage and battery drain since the 2.4.0 version when using variable playback speed. On the Nexus 5x it uses about 8% to play with Android's built-in media player while it uses about 16% CPU when using ExoPlayer with the same playback speed. Looking at the code I can see ExoPlayer now always uses Sonic to adjust the playback speed, even when the Android system supports the variable playback natively on API>=23. That doesn't seem to be an efficient way to handle the variable playback speed to me. What not only use Sonic for API<23 and use the system's built-in support for API>=23? The C version of Sonic on API>=23 is much faster than the Java version of Sonic included in ExoPlayer.

[PaulWoitaschek]

I'm seeing multiple user complaints about high battery drain too.

The C version of Sonic on API>=23 is much faster than the Java version of Sonic included in ExoPlayer.
Did you measure that? According to the author the java and c version are quite equal.

[gemiren]

@PaulWoitaschek, I didn't measure that. Maybe "faster" isn't the proper word to describe this difference in the C version and the Java version. There are could be many other factors involved here. Anyway, the CPU usage is way too high when playing with variable playback speed for API>=23 compared with using the C version Sonic library in the Android system. I just don't understand why do we need to use the Java version of Sonic for API>=23 since the Android system already has that implemented. We don't need to reinvent the wheel for API>=23 to play with variable playback speed. The Android system has that supported and it works nicely and perfectly in the versions before 2.4.0.

Sadly this is a deal break for me. I had to revert back to old ExoPlayer version to fix the battery drain.

[ojw28]

Do you have solid data backing up the claim of increased battery drain? Looking at CPU usage isn't sufficient. Neither is linking ad-hoc user reports to something you know has changed, when other things have also changed. It really needs to be actual battery drain data comparing the two whilst minimizing any other changes.

Note that CPU usage for the playback speed adjustment will likely be attributed to a different process in each case (your process when using the Java version, or the mediaserver process when using the C version). So if you're just looking at the CPU usage of your own process then that's not a correct comparison. As noted above the author of Sonic suggests the performance of the two are give-or-take equal. Note also that you need to be using a release build of your apk, as discussed above. Debug builds may yield vastly inferior performance with variable playback speed enabled.

As for why we're using the Java version everywhere: One reason is simplicity; having a single code path is just a lot simpler. Another reason is that the Android system support isn't perfect. We actually found a bug in Sonic (which is fixed in ExoPlayer's bundled version, but will only be fixed in Android from O onward). If we see solid evidence that power drain is significantly worse then we can revisit the decision.

[gemiren]

@ojw28, unfortunately I don't have the solid data to pin down what's exactly causing the huge increased battery drain. Since I updated my app to use ExoPlayer 2.4.0 I have received numerous user feedback regarding the increased battery drain. So I did some comparison between ExoPlayer and Android Media Player on a Nexus 5x with Android 7.1.2. I did see the battery usage went down significantly if I switched to use the Android Media Player.

I am using the release build compile 'com.google.android.exoplayer:exoplayer:r2.4.1' so it is not an issue of the debug version.

You are right that CPU usage isn't the only factor causing the battery drain. It's just convenient to monitor the CPU usage in the Android Studio to get the rough idea of how the battery performances.

It is true that the CPU usage should include the mediaserver process when using the Android Media player. However, that's not how the battery usage stats works. Now my app is listed at the top of the battery drain list. I can not tell users that the battery usage is probably the same, just because now it includes the battery usage from mediaserver which isn't included before. From a user point of view, when the battery usage goes up almost doubled, any explanation is pale.

[ojw28]

I don't think there's evidence in this thread (yet) of "huge increased battery drain" or that the Java version is significantly less efficient. It seems to me there are two possibilities at this point:

1. Total battery drain is actually about the same as before. Due to the way battery accounting works, battery drain associated with the variable speed adjustment is now being attributed to your app where-as previously it was attributed to mediaserver. This is causing your users to complain.
2. Total battery drain is actually higher (i.e. the user's battery will actually drain faster than before).

If it's the first of these then I don't think we're going to fix it. After all, your app is causing the battery drain that's being associated to it in this scenario. The fact the drain was not accounted to your app previously would effectively be an accounting trick, and something we should probably fix in the platform to ensure correct attribution in future Android releases. If it's the second of the two possibilities then this is definitely something we should look at. If you could help us to determine which of these cases is actually true, for example by running some experiments, that would be very helpful.

Note that it's how you build your apk that affects whether it's a release build or not, not how you depend on ExoPlayer in your build.gradle. See some of the comments from @andrewlewis further up this thread.

[Orbyt]

This is quite a long thread. Can I get a summation of whether or not this functionality has been implemented, and what API levels are supported?

[PaulWoitaschek]

It's supported on all api levels through ExoPlayer.setPlaybackParameters.