Alternate formats or bitrate for an audiobook

[HadrienGardeur]

Audio/video support for various formats can be a bit of a mess on the Web.

This is usually handled through fallbacks to different formats in the audio or video element.

In the case of an audiobook where audio resources are directly referenced from the reading order, the situation is more problematic because we currently don't have such fallbacks.

It's also not uncommon for audiobooks to provide variants with a lower/higher bitrate, depending on the available bandwidth and/or storage. This is not something that the audio element and the associated source element can deal with either.

[llemeurfr]

Media characteristics are useful when one needs 1/ to check that the UA can read the content (this is the case for the audio codec value) 2/ to select one piece of content among several alternatives.

Such checking or selection is usually made on:

audio

• codec or format (the distinction can be difficult; AAC is a codec, WMA is a format, MP3 is a codec and a format).
• channels (1, 2 or more)
• bitrate (in kbits per second)
• sample size or linear bit depth (number of bits per audio sample)
• sample rate (number of audio samples per second, expressed as a sampling frequency in KHz)

video

• codec or format
• aspect ratio (e.g. 16:9, 4:3)
• (average) bitrate (in kbits per second)
• frame rate (number of video frames per second, i.e. the rate at which the material should be shown in order to achieve the intended visual effect).

Adding a notion of alternative renditions at the level of each individual resource is impacting for our structure. Rather than adding this complexity inside the json manifest, let's study the notion of multiple renditions implemented as multiple json manifests. We would be able to express a choice between e.g. one rendition with all tracks in MP3 and a second rendition with all tracks in AAC.

[HadrienGardeur]

Adding a notion of alternative renditions at the level of each individual resource is impacting for our structure. Rather than adding this complexity inside the json manifest, let's study the notion of multiple renditions implemented as multiple json manifests. We would be able to express a choice between e.g. one rendition with all tracks in MP3 and a second rendition with all tracks in AAC.

I don't necessarily agree with that statement. I think it's worth exploring all options if we consider that this should be within the scope of our efforts.

On a given device, a user might want to downgrade to a lower bitrate for instance if they switch to a different network (or want to cache an audiobook but don't have enough space for the high def version) and it would be very cumbersome to juggle between multiple manifests in that case.

[wareid]

Summary from last taskforce meeting (for the benefit of the group):
The discussion on formats and bitrates is focused on whether or not the audiobooks requirements for WP should include a system for allowing for alternate formats/fallbacks for audio content. In the context of WP as a whole we wanted to raise the topic to the group as it has impacts on other file types like video or images.
There was also discussion on whether we should release a recommended list of file types/bitrates, and that is the topic of issue #323.

[iherman]

This issue was discussed in a meeting.

• No actions or resolutions

View the transcript

Wendy Reid: issue 308 bitrate, alt formats
Wendy Reid: no normative statement about which format MUST be used
… but we need a way to indicate what format is referenced (for streaming etc.) for reading systems / player implementations
… make decisions about what to download / cache
… use case for different qualities
… content creator to communicate bitrate, at the very least
Brady Duga: should this group handle this issue
… what about formats that have multiple bitrates etc
Ivan Herman: agree, should not discuss here.
Laurent Le Meur: alternative renditions of same content, DASH better because bundles multiple alternatives
… user agent to choose alt based on metadata expressed in manifest
Garth Conboy: what is the real-world use case
… in trade world
Wendy Reid: Audible does it, but not at distribution stage
Marisa DeMeglio: what’s special about audio books is that HTML mechanisms cannot be used in manifest
Hadrien Gardeur: only works for formats on audio and video (HTML)
… we have to address this issue if we want a solution that works everywhere
Brady Duga: if we allow this, we then need a fallback mechanism
… this is complicated
… create multiple version of the book instead of bundling / declaring the alternatives inside a single publication
George Kerscher: what about ingestion?
Wendy Reid: ingestion, we would be checking the files
Daniel Weck: Benjamin’: HTML audio video do not include bitrate, selection based on format
Wendy Reid: EPUB vs. WP audience, but I realized audiobook work predates all of this, venturing into solving problems that are solved at different level on web
Hadrien Gardeur: content negociation not likely to happen for audio files
… content management backend not smart enough
… some other group will have to deal with this, if this group / we don’t do it
Benjamin Young: https://www.w3.org/TR/html/semantics-embedded-content.html#dom-htmlsourceelement-media
Benjamin Young: not asking about content negociation, just the HTML media source spec
Romain Deltour: responsive image community group
… user agent to negociate, HTML picture element source media
… new HTML element + extended source element to allow for content negociation
… we could reuse constructs semantics , but would need to define processing model
Benjamin Young: this is the crux of the issue (processing model HTML vs. Web Pub manifest
Ivan Herman: -> https://www.w3.org/TR/html/semantics-embedded-content.html#the-picture-element Picture element
Wendy Reid: 308 need more thinking

[TzviyaSiegman]

I would argue that this is a shortcoming of the audio and video elements. I don't think we should try to solve this problem in this group. Perhaps we should look at the HTML spec's picture element [1] as precedent. This essentially allows fallbacks within the HTML.

I recommend writing up a proposal for a container element for audio/video that would enable similar functionality in HTML for alternate formats and submitting it to WICG.

[1] https://www.w3.org/TR/html53/semantics-embedded-content.html#the-picture-element

[plinss]

FWIW audio and video elements do support multiple formats by specifying multiple source elements within them. The codec can be included in the content type, though this is uncommon and may not be web-compat in all browsers. There still is a case for specifying various bitrates, sample rates, and number of channels which I agree could be done by augmenting the audio and video elements or possibly just th source element.

[llemeurfr]

@TzviyaSiegman and @plinss, the context of this issue is audiobooks, a situation where content is not html and therefore there is no support for html audio or video element. Therefore I don’t quite understand your proposal, sorry.

[HadrienGardeur]

FYI, this has been implemented in Readium's take on a Web Publication Manifest: https://readium.org/webpub-manifest/extensions/audiobook#3-alternate-audio-resources

As I've said before, this is a very common feature that I expect most apps to support if they have the ability to do it (Google Play Books being the latest adopter).

[wareid]

In advance of closing this issue, I'll be investigating https://schema.org/AudioObject and analyzing it as a solution for this and other issues.

[llemeurfr]

In schema.org the AudioObject is complex, as
1/ it belongs to an inheritance hierarchy : Thing > CreativeWork > MediaObject > AudioObject.
2/ as a MediaObject it can have a contentUrl, an encodingFormat, a duration, bitrate
3/ as a CreativeWork, it can have an array of associatedMedia i.e. "A media object that encodes this CreativeWork."

Therefore an AudioObject with a contentUrl can contain one or more associatedMedia AudioObjects. Are they the alternate formats @HadrienGardeur was proposing? schema.org doesn't say a thing about it.

[iherman]

This issue was discussed in a meeting.

• RESOLVED: Close #308 - Postpone inclusion of alternative formats and bitrates to a future version of audiobooks, look for feedback on implementation of v1

View the transcript

#308: Alternate formats or bitrate for an audiobook
Wendy Reid: #308
Wendy Reid: the question of alternate formats and bitrates
… we don’t want to mandate media types
… but we still have a question of bitrates
… the audioobject does have a bitrate value, as does mediaobject
… so audioobject could contain more than one bitrate
… do we want to allow this?
Laurent Le Meur: the structure allows one audioobject that contains other audioobjects
… but there is no rel value
… we could say in v1 of audiobooks we don’t allow this
… and we open it up later if the publishing industry needs this
… we have already done this in readium
Wendy Reid: I do like the idea of keeping v1 simple
… and I’m not wild about more @rel values
Brady Duga: I think we should put it off until the next version
… we shouldn’t add it unless someone says they really need it
Wendy Reid: I agree
George Kerscher: I agree, but I recognize that in places like developing countries, having an alternative bitrate that’s easier to stream is desirable
… but we need to get this off the ground first
Wendy Reid: you could do 2 audiobooks
Laurent Le Meur: if we postpone this, we know there is a schema.org solution that doesn’t wreck the structure.
… so it’s easy to defer until later.
Proposed resolution: Close #308 - Postpone inclusion of alternative modalities to a future version of audiobooks, look for feedback on implementation of v1 (Wendy Reid)
Dave Cramer: +1
Bill Kasdorf: +1
Luc Audrain: +1
Brady Duga: +1
Laurent Le Meur: +1
Garth Conboy: +1
George Kerscher: Is modality the right word?
Wendy Reid: it was a way to say alternative formats and bitrates
George Kerscher: +1
Resolution #2: Close #308 - Postpone inclusion of alternative formats and bitrates to a future version of audiobooks, look for feedback on implementation of v1