Hls.js v1.0.0 update and roadmap

[robwalch]

With the v0.14.0 release out of the way, here is an update on what we plan to release in version 1.0:

• Apple Low-Latency HLS support ("LHLS" support has been deprecated)
• Re-architecture of critical paths, including playlist and segment loading, transmuxing, and buffering
• 100% conversion to TypeScript
• Basic IMSC1 support
• EME improvements

Projects for tracking upcoming work

• Apple Low-Latency HLS
• Typescript Integration
• Release Planning

"LHLS" support deprecation

The LHLS spec has been abandoned in favor of Apple's Low-Latency HLS. I encourage anyone interested in chunked-transfer HLS streaming to review Apple's #EXT-X-PREFETCH-HINT guidelines. Progressive streaming using fetch will be supported (via progressive option), but not enabled by default moving forward. Support for manifest tags in the deprecated LHLS spec will be removed. For those interested in the code used for streaming legacy LHLS chunked-transfer streams, the check out the feature/chunked-transfer-lhls branch.

Current branch status (updated 2020/12/07)

• master
  • previous release: v0.15.0-alpha.2 (test release for new GitHub Actions CI)
  • next release: v0.15.0-beta.1 blocked by open issues
• patch/v0.14.x
  • previous release: v0.14.16
  • next release: no planned v0.14.17 patch (check milestones)
• feature/v1.0.0
  • previous release: v1.0.0-alpha.3
  • next release: v1.0.0-beta.1 (This will be close/similar to alpha.3. Determining if Group-id #2972 can be fixed prior to beta v1 release).

NPM release channels

The latest stable release remains under @latest. Nightlies (master branch CI builds) are back under @canary. The next couple of releases will include a new v1 alpha and a v0.15 beta. Once v1 goes beta, v0.15 will either be stable or go to rc.

Updated 2020/12/07:

$ npm dist-tags hls.js

alpha: 1.0.0-alpha.3
beta: 0.14.0-beta.2
canary: 0.15.0-alpha.2.0.canary.6179
latest: 0.14.16
rc: 0.14.0-rc.2

What is being worked on

My focus is on implementing Apple Low-Latency HLS support, planning some of the playlist and fragment loading refactor required for that, and fixing issues in the feature/v1.0.0 branch. A big part of this will also be determining when we can our first alpha release of version one, and how npm release channels can be used to track the latest builds and release in v1 and v0.

The focus of v0.15 is bug fixes and improved handling group based audio and subtitle tracks and fallback.

Additional work we need feedback and help with

There is a lot of work, and we need your help. These are just some of the ideas and issues that concern me that I have only limited capacity to take part in and contribute to.

• Performance benchmarks and enhancements
• Playlist and fragment loading in the worker (if supported by the browser)
• ABR Improvements
• Additional codec support (passthrough provided MSE can buffer the codec, and SCORE attribute on audio variants, h265...)
• Enhanced DRM support
• Enhanced IMSC1 support
• IDR frame tracking (expose the location of key-frames in fragments
• Support for HLS features related to a variety of tags like #EXT-X-INDEPENDENT-SEGMENTS, #EXT-X-GAP etc...
• Demo page improvements and tooling around issue triage
• v0.x maintenance, issue triage, bug template improvements and verification
• Improved TypeScript type definitions checking

Again please check Release Planning for the list of issues we're tracking. And, feel free to submit new issues for anything you don't see there by filling out one the issue templates.

[codener]

Thanks for publishing this roadmap. Can you give a rough estimate on when 1.0.0 will be released?

[robwalch]

@codener I hope by the time iOS 14 is GA but I can't make any promises.

[codener]

No promise needed :) thanks!

[alangdm]

Hi, it's amazing to hear this library is finally getting a major version, you all are doing an amazing job 👍

I have been using this in a couple of projects and I just noticed something, the current build only creates UMD files

Seeing as the library is in TS now, I think it would make sense to create more modern es-modules builds, what do you think?

[robwalch]

Hi @alangdm,

I think it would make sense to create more modern es-modules builds, what do you think?

hls.js still supports IE11. That's not changing for a little while. That doesn't mean we couldn't offer other output besides UMD.

The best place for this feedback would in a new Feature Request. Please submit one.

And, if you've forked the project and updated the webpack config to output the type of module that suites you, please consider sharing those changes as a pull request. Even a draft PR that outlines why the changes are useful would help.

Thanks! 😄

[softworkz]

Hi,

I'd have a few questions regarding low-latency implementation if you don't mind.

1. Which branch to test?

I've seen there a are several branches regarding LL which are recently updated, which one should I use for testing?

2. Best Low-Latency Strategy for local (LAN) or nearby streaming?

Assuming a rather local or near-local streaming situation (without CDN distribution) what do you think would be the best combination of LL-HLS features to achieve minimal latency - considering that I have full control of the server behavior?

3. Independent or non-independent segments and parts?

Assuming the server would need to connect to an upstream link (e.g. tv broadcast) itself upon client request, what would be preferable (with hlsjs as the client side):

• let the server generate parts and segments immediately (non-independent)
• let the server wait and discard stream data until a new GOP begins before creating parts and segments (independent segments)

Thanks

[chbro]

can i use code on this branch in production now?( for progressive feature )
if i set progressive to true, will video start play on receiving first I frame from ts file?

[robwalch]

1. Which branch to test?

feature/v1.0.0 #3072 and the PRs off of it #3112 3160

The latest release at this time from these is v1.0.0-alpha.1

2. Best Low-Latency Strategy for local (LAN) or nearby streaming?

Why use HLS in this scenario?

3. Independent or non-independent segments and parts?

independent. The start of each segment should always be independent. parts don't all have to be, but you should have at least two per segment. Use event key frame intervals / target duration / part targets.

[softworkz]

Hi Rob,

thanks a lot for your reply!

Just to give some context: We are re-streaming broadcast TV streams which are already mpegts. We filter for program and elementary streams and write them as segments without transcoding.

Why use HLS in this scenario?

Couple of reasons:

• Writing the mpegts stream to a single file and have clients access via range requests is not a great option:
  • after filtering out a channel, it's no longer CBR, which can cause seeking problems
  • it's not possible to handle discontinuities (e.g. intermittent bad reception due to bad weather conditions)
  • you can't remove something from the beginning of the file (for implementing a sliding timeshift buffer)
  • there are more..
• HLS is our primary delivery method in almost all client apps already
• In many cases, clients are not in the LAN

So, my question was about how to minimize latency in case of LAN clients. Maybe that wasn't the right question and I should have rather asked about the best low-latency strategy when segments are delivered directly rather than via a CDN (never in our case). Which of the LL-HLS features make sense to use in that case?

• Playlist Delta Updates
  That's clearly useful, because playlists can become pretty large (3s segment duration and a 90 minute time-shift buffer makes 1800 segment entries)
  That leaves the question about how to remove old segments from the start of the list in combination with delta-updates?
  As far as I've understood from the spec, the server should do this by increasing the EXT-X-MEDIASEQUENCE value and leaving the SKIPPED-SEGMENTS value (EXT-X-SKIP tag) constant, is that correct?
• Blocking Playlist Reload
  When using parts, this might make some sense, but I see the huge benefit in the delta-update feature, rather than in blocking reload
• Preload Hints and Parts
  This is the biggest questionmark at my side. I wonder whether it wouldn't be easier and better to always include the next but not-yet-existing segment in the playlist instead and then do something like "blocking segment delivery", which means to hold the http request for such segment, but directly start sending data to the client as soon as the server gets it.
  What do you think about that?

---

The start of each segment should always be independent. parts don't all have to be, but you should have at least two per segment. Use event key frame intervals / target duration / part targets.

In broadcast tv streams, audio packets are often having a negative offset (up to 1s) compared to video packets at the same position in the stream. That makes me wonder how to interpret the "segments SHOULD start with a key-frame" clause as I'm wondering, where and how to cut exactly?

• Cut exactly at the TS packet that represents the start of a video PES packet containing an IDR (key-) frame
• Or can a segment start with non-video packets and it's just that the first video stream in a segment should be a key-frame?
• It might also be possible to determine the PTS of a video keyframe and then cut by that PTS value
  This would lead to a situation, where segments start with a range with video packets only and have a range of only audio packets at the end

What would be the best way to do this?

Thanks

[robwalch]

Hi @softworkz,

We are re-streaming broadcast TV streams which are already mpegts.

Ah that makes sense. Thanks for clarifying.

Which of the LL-HLS features make sense to use in that case?

Blocking Playlist Reload

So I actually think this is the most important / first thing you want to do. Without it, your client is not sure to be in sync with playlist updates adding up to 1 target duration of latency without even knowing it.

Playlist Delta Updates
...how to remove old segments from the start of the list in combination with delta-updates?
As far as I've understood from the spec, the server should do this by increasing the EXT-X-MEDIASEQUENCE value and leaving the SKIPPED-SEGMENTS value (EXT-X-SKIP tag) constant, is that correct?

I don't think EXT-X-MEDIASEQUENCE changes - it should still be the msn of the first segment that was skipped. Refer to the Apple's guidelines and take a look at their sample stream (I shortened the playlist results and marks some key points with ### comments):

Without blocking/delta directives
https://ll-hls-test.apple.com/media1/lowLatencyHLS.php

#EXTM3U
#EXT-X-TARGETDURATION:4
#EXT-X-VERSION:3
#EXT-X-PART-INF:PART-TARGET=1.004000
#EXT-X-MEDIA-SEQUENCE:214452
#EXTINF:4.00000,	
fileSequence214452.ts
###  I removed fileSequence214452 - fileSequence214466 to keep this short  ###
#EXTINF:4.00000,	
fileSequence214467.ts

With blocking/delta directives
https://ll-hls-test.apple.com/media1/lowLatencyHLS.php?_HLS_msn=214468&_HLS_part=1&_HLS_skip=YES

#EXTM3U
#EXT-X-TARGETDURATION:4
#EXT-X-VERSION:9
#EXT-X-PART-INF:PART-TARGET=1.004000
#EXT-X-MEDIA-SEQUENCE:214452
#EXT-X-SKIP:SKIPPED-SEGMENTS=9  
#EXTINF:4.00000,	
fileSequence214461.ts  ### MEDIA-SEQUENCE 214452 + SKIPPED-SEGMENTS 9 = 214461
###  I removed fileSequence214462-fileSequence214466 to keep this short  ###
#EXTINF:4.00000,	
fileSequence214467.ts
###  214468 parts ###

Preload Hints and Parts

It's up to you if you need parts. What are the keyframe intervals like in your source streams? I wouldn't make an hls stream with segments shorter than 2 seconds, so if you want even smaller parts and more frequent playlist updates that add parts or try smaller segments - maybe it will be ok given the proximity of your clients. This is the most complex part of LL-HLS, and many clients still don't support parts so it's up to you to find the right balance.

[softworkz]

I don't think EXT-X-MEDIASEQUENCE changes - it should still be the msn of the first segment that was skipped.

Thank you very much for your examples. I'm not really sure whether that's the intended way of the spec, because in case of your examples, it wouldn't be possible for the server to indicate which segments are no longer available. From my understanding, the SKIP tag is meant to indicate that the skipped segments are still available just left out for the delta update.

What I had read was the following from section 6.2.2 (https://tools.ietf.org/html/draft-pantos-hls-rfc8216bis-07#section-6.2.2):

The server MAY limit the availability of Media Segments by removing
Media Segments from the Playlist file (Section 6.2.1).  If Media
Segments are to be removed, the Playlist file MUST contain an EXT-X-
MEDIA-SEQUENCE tag.  Its value MUST be incremented by 1 for every
Media Segment that is removed from the Playlist file

This is what made me think that this would be the way to indicate which segments would have become unavailable in a situation where delta update are being used - but I'm not sure..

[robwalch]

I'm not really sure whether that's the intended way of the spec, because in case of your examples, it wouldn't be possible for the server to indicate which segments are no longer available.

EXT-X-MEDIASEQUENCE is the first available segment. It does not change when the server returns a delta response.

From my understanding, the SKIP tag is meant to indicate that the skipped segments are still available just left out for the delta update.

That is correct.

I recommend joining video-dev.org and checking out the #hls slack channel for spec questions. This issue and project is not focused on server-side implementation of HLS streams.

[softworkz]

I recommend joining video-dev.org and checking out the #hls slack channel for spec questions. This issue and project is not focused on server-side implementation of HLS streams.

Thanks, I understand. But as we are using hlsjs in a number of clients, I need to be sure that it will work.

What would you suggest then for how to remove segments from the beginning of the list?

You are saying:

MEDIASEQUENCE does not change when the server returns a delta response.

Spec says:

MEDIASEQUENCE value MUST be incremented by 1 for every Media Segment that is removed from the Playlist file

Given, both are right, the conclusion for removing segments would be:

Serve a full list once for the removal and then delta updates again

Should I do it like this?
I

[robwalch]

@softworkz this is not a forum for how to do LL-HLS server side. The examples and documentation is all out there.

MEDIASEQUENCE does not change when the server returns a delta response.

Let me rephrase:
MEDIASEQUENCE should be the same in a delta or non-delta playlist response. If the request is made at the same time, with or without the delta request parameter _HLS_skip=YES, the MEDIASEQUENCE will be the same. The example above illustrates this.

[softworkz]

Then, how to remove a segment from a LIVE media playlist?

PS: Thanks for your patience. I will definitely join the Slack group you have suggested as I got more questions 😆

[robwalch]

Then, how to remove a segment from a LIVE media playlist?

Advance MEDIASEQUENCE like you always would.

As far as I've understood from the spec, the server should do this by increasing the EXT-X-MEDIASEQUENCE value and leaving the SKIPPED-SEGMENTS value (EXT-X-SKIP tag) constant, is that correct?

Maybe I misunderstood this phrase. There is no change to how MEDIASEQUENCE advances in live playlists whether or not it's a delta playlist. MEDIASEQUENCE advances in sliding window playlists and SKIP is the number of EXT-INF segments omitted. SKIP doesn't have to remain constant, but typically it would for a sliding window. If the playlist was like a DVR event that grows than MEDIASEQUENCE would not change and SKIP could increase to keep the delta playlist small.

[softworkz]

Thanks Rob, that was just what I meant.

I'd like to follow up on the keyframe and video/audio offset issues, but I'll give it a little time as we are currently testing with many different broadcast streams from all over the world and I'll get back when I got some more substantial facts together.

[robwalch]

I just released alpha 3 https://github.com/video-dev/hls.js/releases/tag/v1.0.0-alpha.3. We're pretty close to beta and pretty much feature complete with the exception of how we handle multiple alternative audio and subtitle groups #2972.

[kanongil]

One feature that seem to be missing from the Low Latency support is proper bandwidth estimation. The feature can't be used with automatic switching as it is now. It needs to only estimate bandwidth from the time when a request starts receiving data, and not from the time of the request.

When a fragment is 1171 bytes, a request timeline can look like this:

If measured from the start it has a bandwidth of ~320 Kbps. If measured from first byte, it has a more representative bandwidth of ~28Mbps.

[robwalch]

Thanks @kanongil,

Great idea, but we'd need to also measure TTFB and factor it into the ABRController level switching and emergency down-switch reasoning.

Please file a Feature Request so that we can get it onto the project backlog.

[kanongil]

Hey, so I have been experimenting with RC2, and LL-HLS content, and my takeaway is that this release is still very much beta quality. On the surface it can work, but it can quickly break apart.

Most critically, is the bandwidth estimation, which is completely broken for LL-HLS use cases (see #3563). Looking into the current logic, it is fundamentally flawed (transfer duration is used both as value and weight to EWMA) and will likely require breaking changes to fix in a meaningful manner. As such, I hope that you will prioritise fixing this before declaring 1.0.0 ready (or RC, but that ship has sailed).

There are other issues (eg. fetchLoader is somewhat broken) that prevent production usage, but these are more regular bugs.

[robwalch]

Please file issue(s) against fetchLoader if you've found them.