Mixed media in audiobook linear reading order #322
Labels
Comments
|
Possible use-case (?): free audio book sample/preview, first two "chapters" only. The 'reading order' is composed of three items:
I expect that the reading systems / specialised user agents which will implement support for the "audio profile" of web publications (packaged or not) will offer audio-centric user experiences (typical playback-oriented user interface). I suspect that some of these audiobook "players" may in fact not be capable of rendering web pages at all, either by design (e.g. mobile app with no built-in webview) or due to technical limitations (hardware device with physical buttons, basic visual display, or no screen at all). A mobile app should be able to launch an external web "activity" (e.g. system / installed web browser) in order to present the "buy me" page to the user, assuming the So why not just state: "user agents / reading systems that do not support non-audio resources listed in the reading order should ignore/skip these resources" |
|
I think you're making several very good points @danielweck but this is a very complex issue:
Based on all of these points, I think we should carefully consider an audiobook profile where:
Such a profile would be immediately (as in not in 2020+) useful for the industry (for the supply chain as well as end users), since there's a complete lack of standard format for audiobooks. |
|
Hadrien’s proposal seems really good. A UA which is able to switch between Audio resources and html resources will not stop when processing a « Book » publication, an audio-only UA will, as it will filter non « audiobook » publications. |
|
This issue was discussed in a meeting.
View the transcriptreading orderIvan Herman: link: Issue 322 Wendy Reid: the last issue is #322 … it came out of #320 on mixed media and the reading order … some audio books may have supplemental visual content … often added as PDF or EPUB, but no UAs support this … could a WP support this functionality? Should we have mixed media in reading order? Ivan Herman: from the WP point of view, more difficult to restrict it than to have it … we don’t have a restriction on the reading order–it can be any resource. PDF or drawing or whatever. Garth Conboy: your statement is correct … what happened with #320? Why is it ready to close? Wendy Reid: because 322 addresses the question better Garth Conboy: getting a PDF with a bunch of MP3 files is common … so we’re in a good position with what we have … the reading order would be a bunch of mp3s. … then the other stuff would be in the “resources” as it’s supplemental … . out of band auxiliary resources go in resources … not sure about having other stuff in the reading order Avneesh Singh: for 322, what garth said is right … they were discussing the practicalities–what if the reading system doesn’t support PDF? … what should reading system do if the media isn’t supported? … Hadrien mentioned a profile for audio books, where the reading order might be audio-only Brady Duga: avneesh hit on my issue … should we say what happens when these things happen in the reading order … I like allowing these things … but I don’t know how far we should go in the spec to require certain behaviors … I don’t want to prevent innovation Joshua Pyle: I agree with that … to avneesh’s point on what if a reading system doesn’t support PDF … a web publication should support anything the web supports … but what if Alexa comes across a PDF? What happens? … we can’t say what every implementation will do Ivan Herman: back to one comment by Avneesh about profiles … in a sense I could say we already have them, just not by name … we require the type of publication in the manifest … a reading system looking at the type can find out that it is an audiobook … we already say that certain schema terms only make sense in certain contexts … no need to make things more confusing Benjamin Young: I’d agree with ivan … I’d also like to mention atom and rss, which started as text-only but then podcasting happened, and now both formats represent both audio and text, and possibly intermixed … and you’d just listen to audio if your device has no screen … but on some other device you could consume both types of content … so we don’t need to overly constrain the world … we don’t want the types to constrain things. George Kerscher: I agree with the ideas of not limiting innovation … with AR, VR, etc … however, if a reading system ignores what it doesn’t understand, the audio book would be understandable if you just consumed the audio portions Wendy Reid: that’s the assumption that most user agents make with audiobooks … we just don’t offer the supplemental content, for example … we don’t want to stifle innovation. Supplemental content is important. |
|
The following is IMO a proper way to solve this issue: The SHOULD means that the schema will not constrain the resource type for an Audiobook. We don't constrain the behavior of the UA but we state expectations (non-expectation in this case). |
|
One of the most important thing is the ability to detect when the context should switch to "audio-only" or "mostly-audio". With our current specification, we can't look at the We could of course fetch every single resource to figure out what's returned, but this is not the most effective way of dealing with this issue. Inspecting the value of I don't know if we'll end up with a specific profile or just a best practice document (that's up to the Audiobook TF to decide cc @wareid), but this should be within the scope of such documents. |
I would be fine to restrict the current approach described in the current draft so that we would have a series of common suffixes that the processor would recognize to set the media type (after all, these suffixes are part of the media type registration) and the processor would remove all the other ones. I believe that if we do recognize html, svg, jpeg, png, gif, mp3 and maybe a few more common datatypes, we would keep the goal of simplicity for the usual cases and would make the life of implementations easier. Note, however, that in general, and unfortunately, the only secure way is to fetch the resource and see what is returned, because even if |
I'm not a super big fan of that approach either:
|
|
This issue was discussed in a meeting.
View the transcriptWendy Reid: #322Wendy Reid: issue 322 mixed media … audiobook readers don’t support supplemental content (sc) Laurent Le Meur: WP can help supporting sc Wendy Reid: the topic is important for the WG as a whole … publisher will probably mix content in WP even if this is not stated good practice. … an ad could be insterd in the middle of an audiobook Hadrien Gardeur: my concern is that we won’t be able to easily identifiy such supplemental content … the UA will discover it when it tries to fetch it. Brady Duga: maybe not too bad. Some annoying cases will arise. And there is no processing model defined. … A use case may be show a picture of a character + play the corresponding audio … but without processing model, we don’t know what to do. Juan: why don’t specify a duration with an image? George Kerscher: we could put the image in the ToC. … but it’s going in the sync media model Ivan Herman: I don’t understand. Each item in the reading order has a media-type. … the media-type may be wrong but specification wise, this is a fact. Hadrien Gardeur: therefore for an audiobook it”s a requirement to add a media-type. I’m ok. Marisa DeMeglio: this sounds like a case for sync-media. Pure audiobook and sync-media should have proper overlap so that audiobooks can get such bells and whistles. Garth Conboy: the generic case is a business book with tables, and a pdf. George Kerscher: we could do it with a ToC. Wendy Reid: it conrresponds to a use case in which the ToC has images. Ivan Herman: in fact in the current spec there is no default, I was wrong. Maybe we should say that a resource should be an html (by default), if no media type is given. In an audiobook each item would therefore require a media-type. Hadrien Gardeur: the problem is only when there is mixed content. It would be reasonable that if a WP is typed “audiobook”, it should contain only audio files. Proposed resolution: if a string appears in readingOrder/resources/links, the canonical manifest should include a Publication link for that URL signaled as text/html (Ivan Herman) Wendy Reid: or if all resources are audio files the UA sees it as an audiobook and treat it differently than a generic ebook. Avneesh Singh: Audio book may be a profile, which allows to say that it is recommended to have audio content in default reading order Proposed resolution: if a string, or a PublicaitonLink without encoding format, appears in readingOrder/resources/links, the canonical manifest should include a Publication link for that URL signaled as text/html (Ivan Herman) Brady Duga: you say that by default, even for an “audiobook”, the content is html. This would make a stupid format. Ivan Herman: how to be sure the content is an audio file only from its media type? Daniel Weck: we’re deviating from the web, where the media-type triggers behaviour in the UA Ivan Herman: previous proposal resjected ;-) George Kerscher: does epubcheck goes through an confirms the media type of content? Ivan Herman: if there is no explicit media type there is no checking. Daniel Weck: the same remarks we made about mixing content applies to comics also. Benjamin Young: this discussion is not great for web browsers Benjamin Young: to DanielWeck’s point: {"@context": "http://schema.org/", "@id": "http://w3.org/", "@type": "AudioBook"} is correct (but rather insane) JSON-LD Daniel Weck: where to I get the information of book vs audiobook? the audiobook type in the manifest just states the schema.org schema will validate some metadata. Laurent Le Meur: the meaning of “audiobook” should be stronger than that and indicate also that it contains audio Daniel Weck: for sync-media we use “audiobook” with html content. … I’ll open an issue. Hadrien Gardeur: hint should not be trusted. You should prepare for fallback when something goes wrong. |
|
This issue was discussed in a meeting.
View the transcript2. Mixed media in audiobook linear reading order (Issue 322)Wendy Reid: #322 Wendy Reid: Some of this is complicated by the TOC situation. We have 6 open issues that are tagged as audio issues. Wendy Reid: There was a discussion about the reading order - there is no restriction as to what can be in the reading order. We were wondering if in audiobooks, since it has a use case of being mainly audio, but sometimes it contains supplemental info (pictures, charts, etc)… … would that be considered a resource or part of the reading order? It’s innovative/wishful thinking. Based on the discussion. The likely best resolution is that it should be considered as not-part-of-the-reading-order Ivan Herman: I don’t fully understand as I was not part of the discussion. Why do we need to specify? Why isn’t it something that the author can supply/note? Wendy Reid: It’s a decision we have to make. I’m pro having it in the reading order, but the issue could come down to the reading order. The User-agent could end up not showing it. That is another option we have to consider. … It might be easier to say: “these are all resources” Benjamin Young: I don’t like sub-typing. I prefer to keep things consistent — so readers with varying skills can play and handle only what they know. If you have a screen, it can show you what you want, otherwise not. As a spec, we already accommodate it Laurent Le Meur: I would like to have authors coming back to developers — they say: “your system doesn’t work as it doesn’t show HTML when playing audiobooks” We have to set expectations. There is no expectation that a classic audiobook will render HTML. … I understand that some special reading agents would be able to read something else. Standard one would not. We should have expectation that it SHOULD be audio files, and it’s expected that a reading system can handle that, and everything else should be in resources. We can let the JSON schema be open to any resource… … but expectations should be clearly set. Garth Conboy: q Marisa DeMeglio: I don’t have a strong opinion if audiobooks should include other resources, but is this issue about having no playback model for non audio files? Wendy Reid: There is no model, at least not for now… Marisa DeMeglio: A resource in the middle of playing audio — you don’t know how long you should keep it up on the field… Garth Conboy: There is a class of audiobooks today — that come with a PDF or bunch of PDFs that are ancillary material. There’s not expectation for them to render along with the content, or that a reading system is to show it, but the content is to be bundled… … so a user’s action is to look at the supplemental content. We need to have the ability to have those types of audiobooks to be produced, but we should allow supplementary material Ivan Herman: I’m trying to combine those issues with others that are around — there may be other connections. If we say that an audiobook is defined that everything in the reading order must be an audio file. Which is fine — it’s a sensible definition for a well-existing market… … if we do that, nevertheless, there is a need for books that have a mixture of the various media for a bunch of other reasons. They are web publications, but they are not audiobooks. Audio still has items open about items in the manifest - readBy, etc.. All those properties should not be in the audiobook profile… … they renamed general properties that are relevant to audio. If we are strictly audiobooks, then everything related to describing the audio items, they need to be generally available, because they could describe items generally available. Wendy Reid: #351 Tzviya Siegman: Based on Garth and some email descriptions and from HC — I’m a bit confused about supplemental materials. If the method is a URL that the publisher provides, how does the user see it? Is it HTML? I don’t know what Google does, but I know it’s a huge problem. Wendy Reid: Publishers are providing a PDF file — I’ve yet to see anything but PDFs. Right now, if a customer asks for it, we can send it, but we don’t have a good way to deliver it. It’s a very awkward system if you’re using a reading system or an app. There’s no way to surface a URL. Tzviya Siegman: If we have a method of conveying HTML to a user — we have a solution for ToC Garth Conboy: I agree with what Wendy said, on our initial implementation — a user contacted us and we sent the PDF. Right now our listening system surfaces in the player “open supplemental” and we deliver the PDF behind the scenes so the client can click to open… … it works very well. What we need to do — capability wise — is that we need to allow the supplemental PDF to tag along with the content. If the reading system has way to deal with it… Publishers need to be able to package the material and it’s up to … the reading system to implement Nick Ruffilo: I agree with a lot of what’s been said. I think if we go with a web publication with audio and other stuff interspersed, versus just audio… If we give the user the option to download the supplements and letting them know… I’m failing to see why that would be a bad thing. Tzviya Siegman: It sounds like with the existing workflows — this isn’t something that exists today. Maybe we should hold off on spec’ing out. It’s a problem, but maybe it’s something we defer until we understand the scope of the issue. Wendy Reid: To clarify, this issue is about — every discussion the audiobook group talks about, it should include supplemental content. Knowing that not all reading systems will render it. So the question is if the supplemental content should be in the TOC or resource only Wendy Reid: #351 Garth Conboy: +1 Wendy Wendy Reid: so if we use the type specifically, the audiobook TYPE would only see audio files in the TOC> Benjamin Young: I’m not sure what the distinction. Why split the hair? If you have the ability to skip a file that you didn’t understand. If you stuck in a resource that it doesn’t understand. It either fails, or sees what is next and keeps going — possibly with mention that it had to skip. … RSS and ATOM feeds that distribute podcasts have a very similar data model. Those can mix and do often mix. You’ll have a text blog, media blog, etc. The podcast consumer will only process the audio ones. A richer RSS reader will process all… … We shouldn’t limit the resources, but include resources and information that gives implementors what to do with the files, but we should try not to limit ourselves and paint ourselves into a corner. Garth Conboy: I agree with what Wendy said before — we have to allow for supplemental materials. If they are in the reading order or other resources. I lean towards other resources, but it’s just an opinion. To harken back to early EPUB discussion, it can always just skip resources. Avneesh Singh: This has some TOC dependency. Reading order and TOC are different items - they may or may not have overlap. It’s possible the default reading order keeps just audiofiles and the HTML TOC has all George Kerscher: I get that novels are traditionally what is done today, but we should open to a wider range of materials. Magazines — yes there’s a reading order, but more likely I want to go to article 6 etc. A book of poetry — human narration of poetry is much better than machine read poetry… … I don’t want to read poetry book cover to cover, but to navigate to the poem, bookmark, etc. If we ignore this market of different types of publishing, then it gets relegated to someone figuring out how to put together a bunch of podcasts. Mateus Teixeira: +1 George… I’ve even seen requests for audio-textbooks George Kerscher: so we want to come up with something that does all these types of publications. Nick Ruffilo: +1 to mateus with the audio-textbooks |
|
PROPOSAL: Leave mixed media content out of the |
|
@wareid: is it a SHOULD or MUST? I mean is it
or is it SHOULD NOT? The behaviour of the user agent is very different. |
|
The Publishing Working Group just discussed The full IRC log of that discussion<bigbluehat> Topic: https://github.com//issues/322<dauwhe> Github: https://github.com//issues/322 <bigbluehat> wendyreid: the proposal is to leave mixed media content outside of the reading order <bigbluehat> ...but keep it in the resources list <bigbluehat> ...so it doesn't get shown unecessarily <bigbluehat> ...this would be a MUST <bigbluehat> ...so only audio content would be in the reading order <wendyreid> Resolution: Leave mixed media content out of the readingOrder and specify that supplemental content live in resources. This will allow it to be referenced in the TOC, but if a user agent cannot support it, it can be skipped. <bigbluehat> ...everything else would be in the extra resources <bigbluehat> q+ <wendyreid> ack bigbluehat <NickRuffilo> scribenick: nickruffilo <NickRuffilo> Benjamin: How does a reading system know to make use of any of these resources? <NickRuffilo> Wendy: Really good question - and not one solved currently by audiobooks - we say if the user opens them they open them <NickRuffilo> Benjamin: I'm wondering how they know they are there. Publications conceed dependencies across multiple resources. The cover, additional notes, tabs, whatever you might want to put - because of that, there's no binding or way to say where they go or how they should show up <tzviya> q+ <laurent_> q+ <NickRuffilo> ... so the reader or User Agent wouldn't know what to do with them. That seems like - not like what is actually wanted here. <wendyreid> ack tzviya <NickRuffilo> Wendy: This came up - when we discussed last time... <NickRuffilo> Scribenick: bigbluehat <bigbluehat> tzviya: it's basically just saying here's a link to your thing <bigbluehat> ...with no statements of how it should be displayed <bigbluehat> ...we don't explain how to process links <wendyreid> ack laurent_ <bigbluehat> ...and we don't need to get into the details of that <bigbluehat> laurent_: in the example of a cover <bigbluehat> ...there's a rel attribute that says this is a cover <bigbluehat> ...the user agent will know what to do with that <bigbluehat> ...it could be the same for a pdf containing something useful <bigbluehat> ...we could create a vocabulary of uses <George> q+ <wendyreid> ack George <bigbluehat> ...to instruct the UA what it should do with them <bigbluehat> ...without it being a requirement <bigbluehat> George: if there's a MUST here for these not to be in the reading order <bigbluehat> ...is there a MUST that they should be pointed to from a ToC? <bigbluehat> ...otherwise how would they be utilized? <bigbluehat> wendyreid: this is then related to the ToC conversation then <bigbluehat> ...laurent_ brings up a really good point about the rel attributes <bigbluehat> ...perhaps someone can make an issue for how to express supplemental content <bigbluehat> laurent_: ok. I can do that <bigbluehat> wendyreid: I think it deserves it's own discussion <laurent_> q+ <wendyreid> ack laurent_ <bigbluehat> laurent_: I just wanted to say that a vocabular of rel as a best practice is useful <bigbluehat> ...I think we shouldn't go toward excluding content <bigbluehat> ...I will first open the rel attributes for audiobooks <bigbluehat> wendyreid: I don't think this blocks the issue though from being closed <bigbluehat> q+ <bigbluehat> scribenick: NickRuffilo <Avneesh> Should we have another issue for George's recommendation <wendyreid> ack bigbluehat <NickRuffilo> scribenick: nickruffilo <tzviya> I agree with wendyreid and ask George to open a new issue <tzviya> q+ <NickRuffilo> Benjamin: I think pivoting on types is dangerous and moves us away from the work we've done - and coming up wiht a generic model. We're trending to n+1 possible types with possible processing instructions. If the RELs are valuable for cover, they can't be best practices, they have to be known to the reading systems... <NickRuffilo> ... It needs to be part of the data model if we expect it to be part of the data model - otherwise it is constructed however... I'm concerned with a lot of this being under specified and under-planned and not addressing accessibility and UA usage... <NickRuffilo> ... and what the expected behavior would be if a UA sees this. You'd have a reading order thing, a resources thing, some of which is valuable, some of which is not - and it's oddly constructed without clear intentions across different agents. <wendyreid> ack tzviya <bigbluehat> scribenick: bigbluehat <bigbluehat> tzviya: I want to be sure we don't backtrack <bigbluehat> ...I believe we came to a conclusion about rel values in resources <bigbluehat> ...I think we need to have that chat there, and not as part of this issue <bigbluehat> ...we could ask the question about how a UA knows to render anything <bigbluehat> ...we're focusing right now on whether the audio book profile <bigbluehat> ...and what to do with these extra resources <bigbluehat> ...and we should consider this in light of any web publication <bigbluehat> ...right now, there's no specification about how these things are handled <bigbluehat> ...my expectation is that these would simply open in a browser <bigbluehat> ...but we should discuss these in the context of that rel issue <bigbluehat> ...I think George should open that ToC issue, because that is a valid concern <bigbluehat> wendyreid: George could you open an issue for your question then? <bigbluehat> George: sure <wendyreid> https://github.com//issues/323 <tzviya> regrets+ Mateus |
|
Resolution: Leave mixed media content out of the readingOrder and specify that supplemental content live in resources. This will allow it to be referenced in the TOC, but if a user agent cannot support it, it can be skipped. There will be issues opened to address the TOC and rel questions. |
|
This issue was discussed in a meeting.
View the transcript1.2. Issue #322: Mixed media in audiobook linear reading orderWendy Reid: #322 Wendy Reid: the proposal is to leave mixed media content outside of the reading order … but keep it in the resources list … so it doesn’t get shown unnecessarily … this would be a MUST … so only audio content would be in the reading order Resolution #3: Leave mixed media content out of the readingOrder and specify that supplemental content live in resources. This will allow it to be referenced in the TOC, but if a user agent cannot support it, it can be skipped. Wendy Reid: everything else would be in the extra resources Benjamin Young: How does a reading system know to make use of any of these resources? Wendy Reid: Really good question — and not one solved currently by audiobooks — we say if the user opens them they open them Benjamin Young: I’m wondering how they know they are there. Publications concede dependencies across multiple resources. The cover, additional notes, tabs, whatever you might want to put — because of that, there’s no binding or way to say where they go or how they should show up … so the reader or User Agent wouldn’t know what to do with them. That seems like - not like what is actually wanted here. Wendy Reid: This came up — when we discussed last time… Tzviya Siegman: it’s basically just saying here’s a link to your thing … with no statements of how it should be displayed … we don’t explain how to process links … and we don’t need to get into the details of that Laurent Le Meur: in the example of a cover … there’s a rel attribute that says this is a cover … the user agent will know what to do with that … it could be the same for a pdf containing something useful … we could create a vocabulary of uses … to instruct the UA what it should do with them … without it being a requirement George Kerscher: if there’s a MUST here for these not to be in the reading order … is there a MUST that they should be pointed to from a ToC? … otherwise how would they be utilized? Wendy Reid: this is then related to the ToC conversation then … laurent_ brings up a really good point about the rel attributes … perhaps someone can make an issue for how to express supplemental content Laurent Le Meur: ok. I can do that Wendy Reid: I think it deserves it’s own discussion Laurent Le Meur: I just wanted to say that a vocabulary of rel as a best practice is useful … I think we shouldn’t go toward excluding content … I will first open the rel attributes for audiobooks Wendy Reid: I don’t think this blocks the issue though from being closed Avneesh Singh: Should we have another issue for George’s recommendation Tzviya Siegman: I agree with wendyreid and ask George to open a new issue Benjamin Young: I think pivoting on types is dangerous and moves us away from the work we’ve done - and coming up wiht a generic model. We’re trending to n+1 possible types with possible processing instructions. If the RELs are valuable for cover, they can’t be best practices, they have to be known to the reading systems… … It needs to be part of the data model if we expect it to be part of the data model - otherwise it is constructed however… I’m concerned with a lot of this being under specified and under-planned and not addressing accessibility and UA usage… … and what the expected behavior would be if a UA sees this. You’d have a reading order thing, a resources thing, some of which is valuable, some of which is not - and it’s oddly constructed without clear intentions across different agents. Tzviya Siegman: I want to be sure we don’t backtrack … I believe we came to a conclusion about rel values in resources … I think we need to have that chat there, and not as part of this issue … we could ask the question about how a UA knows to render anything … we’re focusing right now on whether the audio book profile … and what to do with these extra resources … and we should consider this in light of any web publication … right now, there’s no specification about how these things are handled … my expectation is that these would simply open in a browser … but we should discuss these in the context of that rel issue … I think George should open that ToC issue, because that is a valid concern Wendy Reid: George could you open an issue for your question then? George Kerscher: sure |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
During a discussion of issue #320 the question of non-audio content (for example, HTML) in the linear reading order of an audiobook came up. Specifically, is that allowed, and if it is what is the expected behavior? Potential answers are "yes" and "we are specifying a format not behaviors". Discuss!
The text was updated successfully, but these errors were encountered: