Prepare progressive streams in MediaSource.prepare

[igorgur]

I am using the functionality of ConcatenatingMediaSource together with setShowMultiWindowTimeBar(true).
My goal is to play multiple videos on the same window with one long seekbar, one period, and multi-window time bar of all videos in total.
When I concatenate 2 videos it works as expected - calculates the total length and enables seeking over the whole concatenated video, but I have noticed that in case of more
than 2 videos it will only concatenate the, but won't play them on the same window with all related support, so once video A ends B starts immediately and so on, which means that concatenating works without multiWindowSupport in this case.

To reproduce that quickly in the demo app , I added the following code in PlayerActivity

MediaSource[] mediaSources = new MediaSource[uris.length];
  for (int i = 0; i < uris.length; i++) {
      mediaSources[i] = buildMediaSource(uris[i], extensions[i], mainHandler, eventLogger);
  }
  ConcatenatingMediaSource mediaSource = new ConcatenatingMediaSource(true, mediaSources);
  simpleExoPlayerView.controller.setShowMultiWindowTimeBar(true);

media.exolist.json

 []( {
    "name": "Cats -> Dogs",
    "playlist": [
      {
        "uri": "https://html5demos.com/assets/dizzy.mp4"
      },
      {
        "uri": "https://storage.googleapis.com/exoplayer-test-media-1/mkv/android-screens-lavf-56.36.100-aac-avc-main-1280x720.mkv"
      },
      {
        "uri": "https://html5demos.com/assets/dizzy.mp4"
      }
    ]
  },)

Go to menu and choose PlayLists >> Cats ->Dogs.
When using only two videos (dizzy.mp4 ,main-1280x720.mkv ) it works as expected, but in case of 3 videos as in my example, setShowMultiWindowTimeBar(true) does not work.
In this example the combination above fails, but changing the third video to a different source passes.
Why is this happening? Am I missing any restrictions on the total length of the videos?

This happens while concatenating 5+ different length private unique videos in my app, while any random pairs of 2,3,4 videos work perfect until comes the last video and disables MultiWindowTimeBar. Added example above with videos from demo app for easy testing. All videos are not corrupted mp4 since I'm able to use each one of them in a pair with other.
I am wondering what causes that?

Using version 2.8.4
Running on Android 6,7,8 , Galaxy S7,S8

Did the next workaround in demo app to test, all 3 videos came out as expected with correct time bar and a singular window.

PlaybackControlView

[725] // multiWindowTimeBar = showMultiWindowTimeBar && canShowMultiWindowTimeBar(player.getCurrentTimeline(), window);
multiWindowTimeBar = true;

[751]     if (window.durationUs == C.TIME_UNSET) {
           // Assertions.checkState(!multiWindowTimeBar);
            break;
          }

[igorgur]

@ojw28, @tonihei Thank you for picking up this ticket. Could you elaborate a little on that, do you plan to address that in the next release? if so what would be the time frame for that fix?

[tonihei]

Looking at the "workaround" you proposed above, it looks like you already figured out the reason behind this behaviour.

To show a multi-window time bar, the player has to know all durations of all items in the playlist. For traditional progressive streams (i.e. not DASH, HLS, or SmoothStreaming), the duration only becomes known after we actually started loading the stream. That means, we won't know all durations until the player at least started prebuffering all items at least once. You'll notice that the multi-window timebar works correctly after reaching the last item in the playlist. And also continues to work correctly after that.

To fix/workaround the issue, you can do one of the following:

1. Use DASH, HLS, or SmoothStreaming if available.
2. Before attaching the view to the player, start loading each item until the duration becomes known. You can do something like: seekTo(window=0, position=0), listen to onTimelineChanged until player.getDuration() is not C.TIME_UNSET, seekTo(window=1, position=0), listen until player.getDuration() is not C.TIME_UNSET, etc., until all duration of all items are known, and only then attach the view with showMultiWindowTimeBar enabled. (and seek back to window=0, position=0 to start from the beginning).
3. In case you know the duration of each item exactly by other means (e.g. they are stored in a database somewhere), you can wrap each item in a ClippingMediaSource with the known duration: mediaSourceWithDuration = new ClippingMediaSource(mediaSource, 0, knowDurationInUs).

Finally, it's actually our intention to support immediate preparation of all playlist items until the duration is known to prevent the issue you describe here (and other issues). I mark this as an enhancement to track adding this feature.

[igorgur]

Yes, you are absolutely right about the lack of duration for that video, and that is the case for the third video, though I would argue whether it's an enhancement or a 100% bug on the player side.

1. When enabling multi-window you would expect the SDK to handle all the logic you described in 'Support wider range of DASH manifests #2' and fire an event once all items were prebuffered , atom's headers were collected from each one of the items and duration for all items is known. That's what I understand by enabling that feature, if I need to handle videos preparation ,seeking, etc., that should be listed somewhere in the documentation or elsewhere, it's not that clear that the developer is responsible doing that in specific unknown scenarios.

2. I am using 3 videos in the example, notice that the third video is the same as the first (dizzy.mp4). The first was buffered and got the duration, why it isn't happening for the third video which is the same one - same structure? It's hard to point out on something, but replacing the third video with a different one would work )), odd situation, while keep doing that will eventually break the logic due to unknown duration of one of the videos, so how is that third item different from other videos?

3. I think that issue boils down to the way Extractors together with ExoPlayerImpl handle the buffering results and hand them to the player layer. I would think that using multi-window should trigger continuous buffering of all items in the list, which is not happening until you actually play all videos - reaching the last item in the playlist, so first try is one experience for the user and on the second try it's a different experience - a real multi window video )) , doesn't seem to be an enhancement.

4. unfortunately I don't know the duration of those videos in advance, neither HLS, DASH are available at this point for my use case.

[tonihei]

It's an "enhancement" because it's something we'd like to implement but haven't done so far.

1. Yes, I agree this should be implemented in the player and we have been planning to do this for a while but didn't get round to it. The seeking pattern I described above is more of a workaround if you need to make this work immediately. As you are the first one to explicitly ask for this feature we also didn't mention it in the docs so far.

2. There is nothing special about the third video. Most likely you are seeing the first two durations quite fast because the first video is playing and second one is already prebuffering (and thus its duration is known). Aside: If you use exactly the same MediaSource object for the first and the third video, we'll know the duration for both right away.

3. The actual implementation will utilize the MediaSource.prepare methods as currently done for DASH and HLS. But yeah, it boils down to start the Extractors right away until the point where we know the media structure.

4. Sorry for that. Have you tried the workaround (with the seeks) I proposed above?

[igorgur]

@tonihei Sorry for the delayed response. Yes, the workaround by wrapping each item in a ClippingMediaSource with the known duration worked, but I wouldn't say it's a convenient solution since not always the length could be accessible in advance.

[AndroidKiran]

@tonihei me too facing the same issue. would be really helpful if this enhancement ticket is taken on priority or if there any good workaround then it will be much appreciated.

[buntupana]

The ClippingMediaSource workaround is working for me, I use FFprobe to get video duration since I have the videos in the device and is easy to access to them. But it would be nice for this feature work out of the box as expected. Thanks to the development team for this great player.

[tonihei]

Updated the title to better reflect what this issue is tracking. Once progressive streams support full preparation they can be used with a fully atomic ConcatenatingMediaSource as tracked by #4868 instead of the current multiWindowTimeBar flag.