Playback may pause on key rotation if license server returns 307 redirect.

[TheJohnBowers]

Issue description

Our license servers will almost always return a 307 redirect when a wv license is requested. This seems to be handled appropriately at least some of the time or I suspect playback would not start at all, but after going through a license changes / key rotation or two I see the following exception thrown:

04-10 12:22:04.813 15318 15318 E EventLogger: internalError [427.07, drmSessionManagerError]
04-10 12:22:04.813 15318 15318 E EventLogger: com.google.android.exoplayer2.upstream.HttpDataSource$InvalidResponseCodeException: Response code: 307

Playback pauses at this point, but if I press play it seems to continue playing without any issues. I would expect playback to continue without pausing.

Reproduction steps

Play the sample content below through several period transitions. It doesn't usually happen until the 3rd or 4rth period transition / key change. The sample content below is 20ish minutes long, and I have yet to see it play past 5 minutes without duplicating the issue. I have set "drm_multi_session" to true as you can see below to allow for key rotation.

Link to test content

  {
    "name" : "Multi Period - 8 Byte IV",
    "uri": "https://content.uplynk.com/054225d59be2454fabdca3e96912d847.mpd?ad=dash",
    "drm_scheme": "widevine",
    "drm_license_url": "https://content.uplynk.com/wv",
    "drm_multi_session": true
  },

Version of ExoPlayer being used

demo app built from release-v2 branch, updated as of April 10th 2018 at 12:45pm:

$ git branch

• release-v2
  $ git rev-parse HEAD
  4b531dc

Device(s) and version(s) of Android being used

I suspect this isn't something that will be specific to any particular android version, but I ran the test and reproduced the error on 2 devices:

1 - Moto Z Droid running android 7.0 - Security Patch Level May 1, 2017
2 - LG V10 - Running android 6.0

A full bug report captured from the device

Sent to requested email address.

[ojw28]

Please see this FAQ. Your license servers appear to be specifying cross-protocol redirects, which aren't followed by default. With allowCrossProtocolRedirects set to true I was able to play through the whole test content.

[TheJohnBowers]

That is certainly not happening. We are redirecting from an HTTPS URL to an HTTPS URL. I was almost 100% certain of that, but to make sure I wasn't wrong I put a proxy between my android device and the world at large and captured all the widevine license requests. I am attaching the capture from my proxy. All the requests were from https://content.uplynk.com/wv to https://content-auso2.uplynk.com/wv. Both HTTPS urls, no scheme change.

[TheJohnBowers]

AllWVRequests.har.zip

[TheJohnBowers]

And the way we do redirects doesn't change ever. We are always redirecting the same way. Sometimes it works, and sometimes I get the com.google.android.exoplayer2.upstream.HttpDataSource$InvalidResponseCodeException: Response code: 307 error. There is definitely something going on here.

[ojw28]

It doesn't look like the redirect ever works, when there is one. Not every request is redirected though. I think I just got lucky in my previous test.

You're right that it's not a cross protocol redirect issue; sorry about that. It looks like the redirect isn't followed due to using 307 for a POST request. Some network stacks opt not to redirect in this case. For example in OkHttp3 here.

I was able to modify DefaultHttpDataSource to re-post to the new URL manually to get things working, so I'm pretty confident this is the correct root cause. It's unclear whether we should add a flag for this though, or whether the setup is just... unusual.

[TheJohnBowers]

So I am slightly confused..... Is returning a 302 required and handled differently for post? According to rfc 2616 " If the 302 status code is received in response to a request other
than GET or HEAD, the user agent MUST NOT automatically redirect the
request unless it can be confirmed by the user, since this might
change the conditions under which the request was issued." -- So on face value a 302 should not be handled differently on POST? If we were returning a 302 this would work properly though?

[TheJohnBowers]

Testing locally changing our redirect to a 302 doesn't fix ExoPlayer, but it does break every other client I have tested. The 302 redirect changes the operation from a POST to a GET on the wv license request. This happens with ExoPlayer and shaka player. The verb cannot be changed obviously or it won't work. Testing with shaka a 307 doesn't change the verb and we get a good re-direct. This stack overflow suggests that this behavior of changing verbs is the difference between a 302 and a 307 (https://stackoverflow.com/questions/2068418/whats-the-difference-between-a-302-and-a-307-redirect) -- After looking at it a bit, I am convinced that the 307 is the correct redirect in this instance and I think ExoPlayer needs a way to handle it.

[ojw28]

So I am slightly confused ... So on face value a 302 should not be handled differently on POST?

In theory yes. However in practice most implementations treat 302 as 303, which doesn't have the "MUST NOT automatically redirect" clause.

After looking at it a bit, I am convinced that the 307 is the correct redirect in this instance and I think ExoPlayer needs a way to handle it.

It's probably correct that a network stack doesn't automatically redirect the request, as per the RFC. However our code probably counts as "the user", and so we should probably confirm the redirect and follow it. So I think you're right, and am marking this issue as a bug accordingly.

I'm still curious as to why this hasn't ever come up before. Using 307s to redirect license requests must be unusual, else someone else would have hit this already. I wonder if most setups:

1. Don't ever redirect at all.
2. Don't ever redirect the first request during a playback session. Redirect subsequent requests by specifying a new license server URL in the license response. If I remember correctly this can be done directly in a Widevine response, the specified URL is then retrieved by calling MediaDrm.KeyRequest.getDefaultUrl() on future KeyRequests generated by the session, and ExoPlayer will use this URL for subsequent requests, if it's set.
3. Somehow work with 302 redirects (e.g. by encoding the post body directly in the redirect URL).

@wvpaf @jefftinker - Any idea what the "normal setup" is for redirecting license requests? In general, would you expect setups to use 307 redirects, and for players to automatically follow them?

[TheJohnBowers]

I wouldn't be surprised if many implementations didn't use a redirect on an initial license request. Our architecture requires use to be able to redirect to one of many load balancing zones however, and the best way for us to do this is to return a redirect the zone where our internal session data was created when the manifest requested. We have tried a few different ways to handle this, but a redirect is probably the most standard way this can be accomplished. It has also proved to be successfully handled by several other players (including the dash.js reference player). As for #2, are you are talking about the renewal server url that can be specified in the license server response? For that we can of course specify the exact location we need without requiring a re-direct, but without license acquisition url in the manifest that all players will universally honor the best way we can handle the initial license request is via re-direct.

[ojw28]

As for #2, are you are talking about the renewal server url that can be specified in the license server response?

Yes.

The good news is that I think it's easy for you to work around this in app code, until an official fix lands. I think you're probably instantiating an HttpMediaDrmCallback instance in app code, which is used to make the post requests. Replace that with an instance of your own MediaDrmCallback class that follows the redirects, and that should solve your issue. To create such a class you can fork HttpMediaDrmCallback, and replace L141 onward with something like (disclaimer: not well tested):

int maxRedirects = 10;
int redirectCount = 0;
while (true) {
  DataSpec dataSpec = new DataSpec(Uri.parse(url), data, 0, 0, C.LENGTH_UNSET, null,
          DataSpec.FLAG_ALLOW_GZIP);
  DataSourceInputStream inputStream = new DataSourceInputStream(dataSource, dataSpec);
  try {
    return Util.toByteArray(inputStream);
  } catch (HttpDataSource.InvalidResponseCodeException e) {
    if (e.responseCode == 307 && redirectCount++ < maxRedirects) {
      url = e.headerFields.get("Location").get(0);
    } else {
      throw e;
    }
  } finally {
    Util.closeQuietly(inputStream);
  }
}

[ojw28]

This is fixed in dev-v2. You can also work around it using existing versions, as described above.