Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(mp4-tools): Fix two AV1 parsing issues #5774

Merged
merged 2 commits into from
Sep 2, 2023

Conversation

nyanmisaka
Copy link
Contributor

@nyanmisaka nyanmisaka commented Aug 28, 2023

This PR will...

Fix two AV1 parsing issues that make Firefox fails to play HLS stream

Why is this Pull Request needed?

  • fix(mp4-tools): Fix AV1 profile parsing

    Profile: unsigned right shift 5-bits for the 2nd av1CBox
    Level: bitwise AND the 2nd av1CBox with 0b11111(0x1f)

    Without this, Main av01.0.15M.08 is parsed as High av01.1.15M.08.
    In this case the High profile is not supprted in Firefox.

  • fix(mp4-tools): set default color descriptions to satisfy Firefox

    In Firefox either you omit the chroma values, or you have to complete the rest of the color descriptions.

    Otherwise even the most basic AV1 Main 4:2:0 8-bit video will be reported as unsupported in HTMLMediaElement.canPlayType and MediaSource.isTypeSupported. Chromium is not affected by this.

Are there any points in the code the reviewer needs to double check?

See also

Resolves issues:

  • av1 Main is mistakenly parsed as High profile and eventually fails in Firefox
    playlist:
    image
    log:
    image

  • Firefox is unhapply with the chroma values
    firefox

Checklist

  • changes have been done against master branch, and PR does not conflict
  • new unit / functional tests have been added (whenever applicable)
  • API or design changes are documented in API.md

@robwalch robwalch added this to the 1.5.0 milestone Aug 28, 2023
@robwalch
Copy link
Collaborator

Thank you for the contribution @nyanmisaka! I'll review soon with the hopes of including these fixes in the next release.

Comment on lines 368 to 375
const monochrome = (av1CBox[2] & 0x10) >> 4;
const chromaSubsamplingX = (av1CBox[2] & 0x08) >> 3;
const chromaSubsamplingY = (av1CBox[2] & 0x04) >> 2;
const chromaSamplePosition = av1CBox[2] & 0x03;
codec +=
'.' +
profile +
'.' +
addLeadingZero(level) +
tierFlag +
'.' +
addLeadingZero(bitDepth) +
'.' +
monochrome +
'.' +
chromaSubsamplingX +
chromaSubsamplingY +
chromaSamplePosition;
addLeadingZero(bitDepth);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am concerned that some user-agents may still depend these fields being present.

https://aomediacodec.github.io/av1-isobmff/#av1codecconfigurationbox-syntax

The parameters sample entry 4CC, profile, level, tier, and bitDepth are all mandatory fields. If any of these fields are empty, or not within their allowed range, the processing device SHOULD treat it as an error. All the other fields (including their leading '.') are optional, mutually inclusive (all or none) fields. If not specified then the values listed in the table below are assumed.

mono_chrome 0
chromaSubsampling 110 (4:2:0)
colorPrimaries 01 (ITU-R BT.709)
transferCharacteristics 01 (ITU-R BT.709)
matrixCoefficients 01 (ITU-R BT.709)
videoFullRangeFlag 0 (studio swing representation)

Would adding the default values defined above work for Firefox?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It does work. I also changed the PR to add these as default values.

1

Copy link
Collaborator

@robwalch robwalch Sep 1, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @nyanmisaka,

Eventually (not in this PR), we may need a workaround for incomplete AV1 codec strings in Multivariant Playlist CODECS sort of like the workaround for opus/flac strings:

hls.js/src/utils/codecs.ts

Lines 132 to 172 in 6227816

function getCodecCompatibleNameLower(
lowerCaseCodec: LowerCaseCodecType,
preferManagedMediaSource = true,
): string {
if (CODEC_COMPATIBLE_NAMES[lowerCaseCodec]) {
return CODEC_COMPATIBLE_NAMES[lowerCaseCodec]!;
}
// Idealy fLaC and Opus would be first (spec-compliant) but
// some browsers will report that fLaC is supported then fail.
// see: https://bugs.chromium.org/p/chromium/issues/detail?id=1422728
const codecsToCheck = {
flac: ['flac', 'fLaC', 'FLAC'],
opus: ['opus', 'Opus'],
}[lowerCaseCodec];
for (let i = 0; i < codecsToCheck.length; i++) {
if (
isCodecMediaSourceSupported(
codecsToCheck[i],
'audio',
preferManagedMediaSource,
)
) {
CODEC_COMPATIBLE_NAMES[lowerCaseCodec] = codecsToCheck[i];
return codecsToCheck[i];
}
}
return lowerCaseCodec;
}
const AUDIO_CODEC_REGEXP = /flac|opus/i;
export function getCodecCompatibleName(
codec: string,
preferManagedMediaSource = true,
): string {
return codec.replace(AUDIO_CODEC_REGEXP, (m) =>
getCodecCompatibleNameLower(
m.toLowerCase() as LowerCaseCodecType,
preferManagedMediaSource,

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed. Currently AV1 is not popular yet in HLS.
But as Apple provides native support for AV1, this problem will become apparent.

robwalch
robwalch previously approved these changes Sep 1, 2023
Comment on lines 368 to 375
const monochrome = (av1CBox[2] & 0x10) >> 4;
const chromaSubsamplingX = (av1CBox[2] & 0x08) >> 3;
const chromaSubsamplingY = (av1CBox[2] & 0x04) >> 2;
const chromaSamplePosition = av1CBox[2] & 0x03;
codec +=
'.' +
profile +
'.' +
addLeadingZero(level) +
tierFlag +
'.' +
addLeadingZero(bitDepth) +
'.' +
monochrome +
'.' +
chromaSubsamplingX +
chromaSubsamplingY +
chromaSamplePosition;
addLeadingZero(bitDepth);
Copy link
Collaborator

@robwalch robwalch Sep 1, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @nyanmisaka,

Eventually (not in this PR), we may need a workaround for incomplete AV1 codec strings in Multivariant Playlist CODECS sort of like the workaround for opus/flac strings:

hls.js/src/utils/codecs.ts

Lines 132 to 172 in 6227816

function getCodecCompatibleNameLower(
lowerCaseCodec: LowerCaseCodecType,
preferManagedMediaSource = true,
): string {
if (CODEC_COMPATIBLE_NAMES[lowerCaseCodec]) {
return CODEC_COMPATIBLE_NAMES[lowerCaseCodec]!;
}
// Idealy fLaC and Opus would be first (spec-compliant) but
// some browsers will report that fLaC is supported then fail.
// see: https://bugs.chromium.org/p/chromium/issues/detail?id=1422728
const codecsToCheck = {
flac: ['flac', 'fLaC', 'FLAC'],
opus: ['opus', 'Opus'],
}[lowerCaseCodec];
for (let i = 0; i < codecsToCheck.length; i++) {
if (
isCodecMediaSourceSupported(
codecsToCheck[i],
'audio',
preferManagedMediaSource,
)
) {
CODEC_COMPATIBLE_NAMES[lowerCaseCodec] = codecsToCheck[i];
return codecsToCheck[i];
}
}
return lowerCaseCodec;
}
const AUDIO_CODEC_REGEXP = /flac|opus/i;
export function getCodecCompatibleName(
codec: string,
preferManagedMediaSource = true,
): string {
return codec.replace(AUDIO_CODEC_REGEXP, (m) =>
getCodecCompatibleNameLower(
m.toLowerCase() as LowerCaseCodecType,
preferManagedMediaSource,

Profile: unsigned right shift 5-bits for the 2nd av1CBox
Level: bitwise AND the 2nd av1CBox with 0b11111(0x1f)

Without this, Main 'av01.0.15M.08' is parsed as High 'av01.1.15M.08'.
In this case the High profile is not supprted in Firefox.

Signed-off-by: nyanmisaka <nst799610810@gmail.com>
In Firefox either you omit the chroma values, or you have to complete the rest of the color descriptions.

Otherwise even the most basic AV1 Main 4:2:0 8-bit video will be reported as unsupported
in HTMLMediaElement.canPlayType and MediaSource.isTypeSupported. Chromium is not affected by this.

Signed-off-by: nyanmisaka <nst799610810@gmail.com>
@robwalch robwalch merged commit c9e9916 into video-dev:master Sep 2, 2023
@nyanmisaka nyanmisaka deleted the fix-av1-parseing branch September 2, 2023 19:34
github-actions bot pushed a commit to teiron-1604/hls.js that referenced this pull request Feb 26, 2025
# 1.0.0 (2025-02-26)

* [JW8-6113] Implement progressive AES decryption ([video-dev#219](https://github.com/teiron-1604/hls.js/issues/219)) ([1050e74](1050e74))

### Bug Fixes

* `setMediaKeys` return undefined in some Chrome versions ([video-dev#6669](https://github.com/teiron-1604/hls.js/issues/6669)) ([1d10714](1d10714))
* abrFactor greater than in docs ([1adc39e](1adc39e))
* afterEach callback should return a promise ([44aa769](44aa769))
* baseMediaDecodeTime in tfdt v0 should not be re-written to a negative value ([b884798](b884798))
* calling detatchMedia() followed by attachMedia() causes audio to not play ([161c66c](161c66c)), closes [video-dev#2099](https://github.com/teiron-1604/hls.js/issues/2099)
* catch exception ([715ca74](715ca74))
* change test streams hosting to mux ([video-dev#2307](https://github.com/teiron-1604/hls.js/issues/2307)) ([25f0bbe](25f0bbe))
* **ci:** invalid branch ([5c20968](5c20968))
* computeLivePosition minimum value of media.currentTime ([3fd87fe](3fd87fe))
* consider paused state  ([b58c595](b58c595)), closes [video-dev#2417](https://github.com/teiron-1604/hls.js/issues/2417)
* contiguous detection error ([2a9180d](2a9180d))
* Delete unused var ([9d3beb9](9d3beb9))
* **demo:** update http to https ([9327fc6](9327fc6))
* **deps:** update to esdoc@1.1.0 ([ad6a9e6](ad6a9e6))
* **docs:** fix links to src ([b82e921](b82e921))
* duplicate segmented vtt lines ([a25f0b1](a25f0b1))
* failed to switch back to main audio from additional audio ([video-dev#5170](https://github.com/teiron-1604/hls.js/issues/5170)) ([6e157ac](6e157ac))
* Fix indent ([6eb6eaf](6eb6eaf))
* fix live playlist not sync ([938606f](938606f))
* **func-test:** add "firefox" to black_list of angelOneShakaWidevine ([cb6e22e](cb6e22e))
* **func-test:** add `nudgeMaxRetry` option for slow seek env ([aca699c](aca699c))
* generate silent aac frame based on original codec ([video-dev#6123](https://github.com/teiron-1604/hls.js/issues/6123)) ([b721af3](b721af3))
* initial segment change without discontinuity ([ab9651e](ab9651e))
* **latency-controller:** only sync live stream ([d3845c2](d3845c2))
* level playlist parse error due to "rawProgramDateTime" of initial segment not reset ([a4b4f5f](a4b4f5f))
* lost program date time. ([7fef02c](7fef02c))
* move `types` condition to the front ([video-dev#5439](https://github.com/teiron-1604/hls.js/issues/5439)) ([9082342](9082342))
* **mp4-tools:** Fix two AV1 parsing issues ([video-dev#5774](https://github.com/teiron-1604/hls.js/issues/5774)) ([c9e9916](c9e9916))
* Not set initial segment for part and hint segments ([3331d6f](3331d6f))
* partial audiovideo fragments not being treated as partial ([video-dev#5460](https://github.com/teiron-1604/hls.js/issues/5460)) ([ef718d2](ef718d2))
* remove redundant code ([d8d684d](d8d684d))
* remove uneeded tests ([95f32a1](95f32a1))
* Remove unnecessary option ([video-dev#2273](https://github.com/teiron-1604/hls.js/issues/2273)) ([f723c06](f723c06))
* **Remuxer:** Safari segment overlap ensure PTS order ([video-dev#6132](https://github.com/teiron-1604/hls.js/issues/6132)) ([af42912](af42912))
* Replace Loader.getResponseHeader() with Loader.getCacheAge(). ([d520008](d520008))
* Replace tab to space ([6eec3df](6eec3df))
* set initial segment repeatedly for ll-hls stream ([78de428](78de428))
* show cc with low latency stream. ([6e23bd0](6e23bd0))
* stopLoad() after destroy() becomes no-op, startLoad() after destroy() throws nicer error message. ([0c32415](0c32415))
* StreamController#stop should call clearInterval ([1aa78bf](1aa78bf))
* **subtitle:** vtt fragment sn is consistent with ts, no need to request previous fragment sn ([6d1f3db](6d1f3db))
* the functional tests for "MPEG Audio Only demo" failed ([732c2c3](732c2c3))
* **timeline-controller:** reuse text tracks when not in same order ([video-dev#1869](https://github.com/teiron-1604/hls.js/issues/1869)) ([6db0112](6db0112))
* tizen 2017 widevine playback ([video-dev#6657](https://github.com/teiron-1604/hls.js/issues/6657)) ([f167465](f167465))
* type name error ([video-dev#5295](https://github.com/teiron-1604/hls.js/issues/5295)) ([91364bb](91364bb))
* **types:** Remove unused Loader.loader property. ([fb1ead0](fb1ead0))
* **type:** use extended generic type `LoaderContext` in type `Loader` ([video-dev#5487](https://github.com/teiron-1604/hls.js/issues/5487)) ([075e834](075e834))
* typo ([f769aac](f769aac))
* unit test fail, observer.emit is not a function ([6b85ffa](6b85ffa))
* use hasInterval instead of isNaN ([a9e6c45](a9e6c45))
* using implicit return ([4bf6ee9](4bf6ee9))
* vtt parser broke after master merge ([e96a2a1](e96a2a1))
* X-EXT-MAP in between duration and url ([d7e90c6](d7e90c6))
* X-EXT-MAP in between duration and url ([078fddc](078fddc))

### Code Refactoring

* Remove loader.getResponseHeader(). ([ca962e5](ca962e5))

### Features

* add hasFragment and removeAllFragments ([3ae3aa3](3ae3aa3))
* add support for CMCD nor ([video-dev#6091](https://github.com/teiron-1604/hls.js/issues/6091)) ([763c129](763c129)), closes [video-dev#6088](https://github.com/teiron-1604/hls.js/issues/6088)
* add support for decrypting init segments. ([41addd9](41addd9)), closes [video-dev#2259](https://github.com/teiron-1604/hls.js/issues/2259)
* Add supported M3U8 tags ([2ca6bb8](2ca6bb8))
* add type to discontinuities.js and fix @typescript-eslint/prefer-optional-chain error ([ad1bbef](ad1bbef))
* **ci:** add release workflow ([154409c](154409c))
* Format code by Prettier ([993ef82](993ef82))
* **level:** deliver level details when level switches ([55e4dbc](55e4dbc))
* using ts assert instead of runtime code changes ([0674273](0674273))

### Performance Improvements

* for .. of to simple for iteration ([409a1ab](409a1ab))
* inlining bufferedIncludesPosition ([ad5bfa1](ad5bfa1))
* pre-cache buffered positions ([03aa7ec](03aa7ec))
* use helper function to filter evicted fragments ([34190c0](34190c0))

### Reverts

* Revert "buffer-helper: correctly report bufferInfo when current time is located before buffered ranges, but within maxHoleDuration range" ([877f1c6](877f1c6)), closes [video-dev#420](https://github.com/teiron-1604/hls.js/issues/420)
* Revert "config defaults: set maxBufferHole to 15 second" ([8acc166](8acc166))
* Revert "Configure rennovate to `pinGitHubActionDigests`" ([2c993b1](2c993b1))
* Revert "Disable `AppleAdvancedHevcAvcHls` stream for now due to cors errors (…" ([video-dev#6187](https://github.com/teiron-1604/hls.js/issues/6187)) ([efb252c](efb252c))
* Revert "functional test: run against Safari 10.0" ([be5d7c5](be5d7c5))
* Revert "switch to npm 7 ([video-dev#3666](https://github.com/teiron-1604/hls.js/issues/3666))" ([video-dev#3671](https://github.com/teiron-1604/hls.js/issues/3671)) ([2e3f6f3](2e3f6f3))

### BREAKING CHANGES

* Remove loader.getResponseHeader() in favor of loader.getCacheAge().
* force software AES decrypter if config is true
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants