DRM Session acquisition in Loader

[kgrevehagen]

This is a follow-up on comments made in #7613, regarding seamless codec transition and especially what I see as your solution to it.
@tonihei says:

We have a pending improvement plan to move the entire DRM session acquisition to the Loader (that runs on its own thread), so that this does not occur anymore.

I have a couple of questions regarding that:

First of all, I just wonder if you have made any progress or plans for this in the near future?

But most importantly, maybe I was not clear enough in my description of the issue(in #7613), but I have a hard time seeing this as a solution. Let's say you implement the mentioned improvements of moving the entire DRM session acquisition to the Loader, how does this solve the problem? The problem as I see it is not about DRM session acquisition being on a different thread(this can easily be fixed), the problem is configuring the codec with the mediaCrypto(happens in e.g. MediaCodecAudioRenderers configureCodec() method).

As stated in #7390, DRM session acquisition can take long time on some devices. The problem arises when it is time to set the mediaCrypto on the codec. First of all, we can't just set a mediaCrypto on a codec that is playing. I guess this is the reason for preparing the drm before playback, so we use the secure decoder from the get-go?

What happens in this scenario is when configuring the codec(even though everything up to this point happens on a background thread), I can't put the configuring of the codec on a different thread, so when doing this it takes so much time that the internal buffer on the codec runs out before I'm finished with configuring the codec with crypto information, hence the glitch.

My proposed fix in #7613 is twofold: First I want to delay the drm preparation and acquisition until after playback has started to speed up startup time. This means we can't start playback with the secure decoder, because the mediaCrypto would need to be set when configuring the codec(before using that codec for playback). This again means we would need a 2nd codec(the secure codec) to switch to after drm acquisition is done.

Sorry for a bit back and forth, but this is a complex issue and somewhat hard to explain in as few words as possible. It all boils down to me wanting to speed up startup time, so we HAVE to start with a decoder without mediaCrypto. Since it is not possible to reconfigure a codec midstream to add crypto information, we need to switch to a secure decoder before clear lead finishes. I'm guessing the solution for the Loader fixes the glitch part of this issue, but not startup time, because we would still need to prepare and acquire drm session so that we can start playback with a secure decoder. Does this sound right?

Am I misunderstanding something or does it make sense to you?

I'm happy to provide any concrete examples of any of this if that would be of more help.

Again, thanks in advance!

[icbaker]

First of all, I just wonder if you have made any progress or plans for this in the near future?

This is done - it's not actually in the Loader, but it happens on the loading thread. The relevant commits are at the bottom of #4133 and it will be included in the 2.14.0 release.

---

The problem as I see it is not about DRM session acquisition being on a different thread(this can easily be fixed), the problem is configuring the codec with the mediaCrypto(happens in e.g. MediaCodecAudioRenderers configureCodec() method).

I'm not aware of configuring the codec being slow/a bottleneck. The issues you've linked from your post mention slowness in MediaDrm#provideKeyResponse, but this is part of what now happens on the loading thread (well, actually a different thread again, but it starts happening at 'buffering' time and so should generally be complete by the time the playback position reaches the DRM transition).

---

I'm guessing the solution for the Loader fixes the glitch part of this issue, but not startup time, because we would still need to prepare and acquire drm session so that we can start playback with a secure decoder. Does this sound right?

I'm not convinced this is always right, no. You don't need a 'full' DRM session in order to configure a secure decoder. ExoPlayer allows you to request a 'placeholder' DrmSession by passing a format.idrmInitData == null to DrmSessionManager#acquireSession.

Acquiring a placeholder session should be fast, and it doesn't interact with the TEE or make any network calls.

ExoPlayer will only request placeholder DRM sessions, and use a secure decoder for clear content, if requested via MediaItem#setDrmSessionForClearPeriods.

For content with a clear lead, if you want playback to start before the keys for the encrypted content is available, you might like to take a look at MediaItem#setDrmPlayClearContentWithoutKey.

The 'real' session can then be requested in the background while the clear lead is playing out - and swapped in at the transition point. This swap should be fast (because the session is fully open, with keys, and ready to be used).

---

Quoting from #7613:

Link to test content

Any DASH with Widevine(and most likely any drm-ed content with clear lead)

It might sound silly, but it would be helpful if you could actually provide the content that you're using for your experiments, so that I can be sure I'm looking at exactly similar content and therefore the same behaviour of the library + platform. The DRM space is confusing enough already, so it seems best to nail down as many unknowns as possible.

[kgrevehagen]

Thanks, I will have a look at the fixes in #4133, but this is not a big issue for me since I have found a way around it anyways, and it is no longer an issue for me. The real issue I'm facing now is no matter how I want to make the switch, I'm getting a glitch.

1. If I do your solution with the placeholder session and swap it at transition point, how exactly would I do that? I could use the updateDrmSessionV23 in MediaCodecRenderer.java, but that requires that I have set a mediaCrypto on the codec, which again requires that I open the session, which I really want to avoid, because mediaDrm.openSession() takes 2-300 ms on my Pixel 3, and that is quite a lot. Is there another way around this approach? If I could fake a sessionId to put on the mediaCrypto that I'm not going to use(because it will be replaced with a proper one when time is right)?

2. If the above is not possible, my next idea is to just call drainAndReinitializeCodec() in MediaCodecRenderer.java. This will then drain the non-secure codec, create a new secure codec and start feeding with that. This seemed to work, but this is where I get a tiny glitch when the switch is happening.

3. Since the above didn't work for me, I wanted to understand what was happening, and why it wasn't working. So I decided to create a second codec in the background, and when all is set up, start feeding that right away, while still draining from the non-secure codec. When the non-secure codec is done draining, it will go directly to the secure codec which already has fed some input buffers, and start draining that. The difference in real life from this and the solution above is that we don't need to drain the non-secure before start feeding the secure, so I hoped I could get rid of the glitch in hope that it was because it was actually an underrun. But, the same tiny glitch appears here as well.

I have logged everything I could think of, and according to the logs, it receives all buffers and drains them correctly, not skipping anything. This is the same exact glitch that is happening in the scenarios where I use drainAndReinitializeCodec(), so it seems to be redundant to do all of this fancy stuff when I could just use drainAndReinitializeCodec(), except it doesn't work as expected.

Do you have any idea of what is happening here? I have so many theories, but it is really hard to figure out what is actually happening. One thought is that the EOS is outputting a "no-sound" buffer, hence giving the glitch. Could that be the reason? If yes, is there any way to work around this? And, since I need to send an EOS signal when switching, do I need to send something special in the beginning? Could it be that I need to make the switch at the correct place, in regards to keyframes or something?

Any help is really appreciated, thanks in advance!

I'm sending a link to test content on mail. To reproduce the issue, just make sure that you start the playback with a codec without mediaCrypto set, do the drm setup in background and then call drainAndReinitializeCodec() at some point in time after this(I just do this hardcoded after 3 seconds for testing). The test content is a 23 second sine wave clip(so you can hear the glitch really good) with about 20 seconds of clear lead. You don't need to fetch the keys in order to reproduce.

[icbaker]

because mediaDrm.openSession() takes 2-300 ms on my Pixel 3

That does seem slow - but I did some testing and I see similar speeds (generally closer to 200ms) on a Pixel 3a XL. All my replies so far have been assuming MediaDrm#openSession was sufficiently fast to not be a concern - but I see that isn't necessarily the case, so apologies for that.

While testing I found that only the first session opened on a given MediaDrm was that slow. Subsequent ones were more like 15ms - 20ms. You can see this effect by trying the "Widevine DASH (MP4, H264) > Secure -> Clear -> Secure" content in the demo app (I did more testing to ensure the 'clear' session wasn't fast because it was a placeholder - it was also fast when opening subsequent 'real' sessions - and also the control flow is the same for both placeholder and 'real' sessions at this point so there couldn't really be a difference). Not sure if that's helpful, but I found it interesting...

---

If I could fake a sessionId to put on the mediaCrypto that I'm not going to use(because it will be replaced with a proper one when time is right)?

I doubt this will work - I suspect the sessionId you pass to MediaCrypto needs to correspond to a real, open session - even if it doesn't have any keys.

---

I'm sending a link to test content on mail. To reproduce the issue, just make sure that you start the playback with a codec without mediaCrypto set, do the drm setup in background and then call drainAndReinitializeCodec() at some point in time after this(I just do this hardcoded after 3 seconds for testing).

The changes you've described to Media(Audio)CodecRenderer here are quite involved - can you provide a fork of the demo app with the changes applied so I can be sure I'm using exactly the same code? WIthout any changes the link provided fails to start playback at all because no license URL is provided.

[kgrevehagen]

You are right about it only being the first one that takes a significant amount of time, so this would be an okay solution for us, but when using this solution, we are experiencing the glitch I have mentioned before(e.g. #6751).

This takes me back to the other solution I'm trying to implement(#7613) where I want to reinitialize the codec after playing on a non-secure codec to play on a secure codec.

I have been digging a bit here and it only happens on AAC, but not on FLAC. On FLAC it works as expected, no glitches, nothing. My gut tells me that we need to look into how the AAC codec works to figure out what is happening, and after talking to my team, two of the main differences on AAC and FLAC is:

1. AAC Priming frames. These are silent frames added by the encoder at the beginning of the track, so we should not be affected by this since this is mid-stream.
2. Windowing and overlap-add. The decoder needs to know about the previous 1-2 frames in order to correctly decode the current frame. This sounds very interesting and on the point to the problem, since the new codec I just switched to obviously has no knowledge about these frames.

Do you know of an easy way to test this theory? Is it as simple as just copy the last frames from the 1st decoder and reapplying them to the 2nd decoder without writing them to the AudioTrack? Any special flags that should be sent or a presentationTime? Or do you have any other suggestions, as to me, this sounds like the root issue of the problem.

For the url sent in the email to work, you just have to add the following block to media.exolist.json:

{
  "name": "AAC test content",
  "uri": "<url-sent-in-email>",
  "drm_scheme": "widevine",
  "drm_license_uri": "https://proxy.uat.widevine.com/proxy?provider=widevine_test"
}

Keys don't need to be fetched in order to play the 20 first seconds of the stream and the license url can just be what you already use. I tested this on 2.14.0 release-v2 branch and it played the first 20 seconds, which is enough for what we need.

In addition, to reproduce the glitch, all you have to do is sometime during playback call drainAndReinitializeCodec() in MediaCodecRenderer.java. I did it by adding a field private boolean reinitializedCodec = false; and the following code block in render() method(still in MediaCodecRenderer.java) right before drainOutputBuffer() is called:

if (positionUs > 1500000 && !reinitializedCodec){
  drainAndReinitializeCodec();
  reinitializedCodec = true;
}

This will drain the current codec, create a new one and start feeding and draining that. AAC will give a glitch here, but FLAC will work perfectly. Let me know if you need similar FLAC content to verify that it is actually working for FLAC.

I want to add that if I skip the first frame of the secure decoder in drainOutputBuffer() by doing this when on the first frame of the new decoder, it will play without glitch:

  outputBuffer.position(0);
  outputBuffer.limit(0);

This only works for the sine wave I provided(since all frames are basically the same), it will not work on proper tracks, then it will just jump over it. But this tells me that there is something wrong with that frame, it is "damaged" or "broken" somehow, and we need to repair it so the whole thing is playable.

Thanks again for looking into this, and hope my steps will aid you in the right direction as to help me in fixing this issue!

[kgrevehagen]

Update: I actually got this to work now by feeding the data of the last buffer of the first codec first thing on the new codec, and then just skipping it when draining. Like this:

In MediaCodecRenderer.feedInputBuffer() before reading from source:

if (<first feed loop from new codec>) {
  byte[] data = <lastFrameFromFirstCodec>;
  long presentationTimeUs = <buffer.timeUs from last frame from first codec>;

  buffer.data.put(data);
  codec.queueInputBuffer(inputIndex, 0, data.length, presentationTimeUs, 0);
  resetInputBuffer();
  return true;
}

and in MediaCodecRenderer.drainOutputBuffer() in the else if of if(shouldSkipAdaptationWorkaroundOutputBuffer):

long presentationTimeUs = <buffer.timeUs from last frame from first codec>;

else if (outputBufferInfo.presentationTimeUs == presentationTimeUs) {
  codec.releaseOutputBuffer(outputIndex, false);
  return true;
}

As you can see, the draining part is exactly the same as shouldSkipAdaptationWorkaroundOutputBuffer, but with an added part in the feed logic. So it seems you are kind of supporting half of my solution already.

Would this be the proper way of doing it? Of course only doing it for codecs that we know behaves like this.

Note: This is tightly coupled to #7390, #7613 and #6751 in order for clear lead(playClearSamplesWithoutKeys) to work as expected on AAC(or other similar) codec with no loss in startup time and it works for all devices. The other part of the job is to put ALL mediaDrm calls from DefaultDrmSession into background thread so they don't occupy the thread the codec works on.

Would this be something you might be implementing support for? I'm happy to help, even create a PR if you would like that.

[icbaker]

Following your repro steps I'm able to hear the glitch in the audio in the AAC track you provided. I tried some other AAC tracks in the demo app (e.g. Misc Audio -> Big buck bunny (AAC)) and didn't hear the same glitch, but it's possibly less obvious without a sine wave.

The AAC behaviour you're seeing seems to be the 'decoder delay' described here: https://developer.apple.com/library/archive/documentation/QuickTime/QTFF/QTFFAppenG/QTFFAppenG.html

Based on the wording there it seems it might be fluke that exactly one extra frame is enough? (because AAC frames are also 1024 audio samples i think)

---

Note: This is tightly coupled to #7390, #7613 and #6751 in order for clear lead(playClearSamplesWithoutKeys) to work as expected on AAC(or other similar) codec with no loss in startup time and it works for all devices. The other part of the job is to put ALL mediaDrm calls from DefaultDrmSession into background thread so they don't occupy the thread the codec works on.

Would this be something you might be implementing support for? I'm happy to help, even create a PR if you would like that.

This sounds like a pretty major change - because some of the existing control flow assumes that at least MediaDrm#openSession is called synchronously from the DefaultDrmSession constructor. I'm afraid we likely don't have the capacity to either make this change or review a PR on this in the short term.

For now, if you're sure you need to move all MediaDrm calls off the playback thread, I'm afraid the best I can suggest is that you fork DefaultDrmSession(Manager) and make the changes you need there.

[icbaker]

Closing because I think the question was answered and there's no immediate action planned here.