Relationships to the Web App Manifest specification.

[iherman]

In case the (concrete) manifest is expressed in JSON (see issue #7), should it be defined “on top” (i.e., as some form of an extension) of the Web Application Manifest specification, or should it be a fully separate specification?

[rdeltour]

It is of course be a bit early to answer definitively, but integrating with Web app manifests would make a lot of sense, as I said already in the mailing list (see thread "extending Web app manifest?").

I think we can already liaise with the Web Platform WG on this one.

[iherman]

(All these arguments are with the assumptions that we decide, by closing #7, for a JSON serialization of the manifest information items.)

I have been looking at the Web App manifest Editor's draft which made me realize something. Namely that only, roughly, 50% of the document is dealing with the definition of the various JSON fields ("members", as they are referred to in the document), their exact syntax, etc. The other half (and, at least as far as I am concerned, the much more complicated half, I cannot claim to have understood the technical details) deals with general issues on how a User Agent "processes" a manifest. Just look at the section headings: "Installation Events", "Navigation Scope", "Manifest life-cycle", etc. What these describe is how a Web Application is handled by a browser, how it "installs" the Web Applications (ie, makes it available to the end user as if it was on the screen of a mobile installed through, say, the Apple shop), how the application controls the display modes and what this means, how a manifest is located, processed, etc. If you look at the spec, these steps are all defined in excruciating details. Note also that if a browser "implements" a manifest, it would have to do all that (and the goal is to implement manifests of course with first implementation already coming…).

Well… we already said, I believe, that, from a browser point of view, there is indeed a strong similarity between a Web Application and a Web Publication. Having looked at those sections a bit I did not really find anything where I would have thought: this is really irrelevant for WP-s. Consider this the other way round: we would have to define all those features for WP-s ourselves unless we deliberately tried to build upon what has been done elsewhere. Which makes a convincing case, I believe, to say that it is the best option for us to build upon the Web App manifest, rather than define our own (even if there are certain aspects that we do not necessarily like).

This means, of course, some level of compromise, in the sense that the various "members" that we would define for JSON would either have to reuse (when applicable) the "members" as defined for the Web App Manifest, or be defined as additional "members" along the similar lines/styles. Maybe we will have to discuss with the Web App Manifest editors some aspects that may require changes (e.g., we may want to allow for the scope to define, possibly, several URL-s and not only one, just as an example). But that should be possible; the Web App Manifest is a living document, not a document cast in concrete (for now!). By, possibly, choosing a suboptimal syntax here and there we can, in exchange, save ourselves the (huge!) work of re-defining all the non-syntactic aspects that those guys have already done.

B.t.w., to make one thing clear: the document says, in the section on Processing the manifest that:

The following algorithm provides an extension point: other specifications that add new members to the manifest are encouraged to hook themselves into this specification at this point in the algorithm.

Ie, the question whether we can add our own "members" is a non-issue. We can.

[rdeltour]

Great points, @iherman. I fully agree (if that was not clear already 😉 ).

In order to move forward with the decision making, I believe we will need to:

• look at the Use Cases and Requirements for Installable Web Apps and see if there are some requirements that are not aligned, or contrary, to our own; and also see what additional requirements we could bring in the context of WP.
• see if the built-in extension points is enough to plug-in our own processing requirements (if any, TBD).

[HadrienGardeur]

I'm not comfortable with the Web App Manifest in its current state:

• it's missing a lot of things that we'll need and is very specific to Web Apps in general
• it also lacks a good extensibility mechanism (which we'll need since the current document doesn't cover our use case at all)
• Web Apps have a completely different philosophy regarding resources which will conflict with our goal (instead of listing resources, Web Apps rely on a Service Worker per app to cache resources and handle a network policy)
• browsers require the presence of a Service Worker and HTTPS to install a Web App, I don't think that we'll have the same requirements which means that they either won't support our use case or we'll drastically impact how this manifest is processed

IMO there are too many differences in our requirements and use cases to find a middle ground that would work for everyone.

[iherman]

@HadrienGardeur,

• it's missing a lot of things that we'll need and is very specific to Web Apps in general
• it also lacks a good extensibility mechanism (which we'll need since the current document doesn't cover our use case at all)

Yes, it is missing things we need. The "good extensibility mechanism", as I said in the previous comment, is there: we can add additional items ("members") to the JSON file at our heart's content, and the processing includes an explicit step for the extension specific processing. This is not an issue.

• Web Apps have a completely different philosophy regarding resources which will conflict with our goal (instead of listing resources, Web Apps rely on a Service Worker per app to cache resources and handle a network policy)
• browsers require the presence of a Service Worker and HTTPS to install a Web App, I don't think that we'll have the same requirements which means that they either won't support our use case or we'll drastically impact how this manifest is processed

Whether these things are indeed different or not is a question. One could consider a WP a special type of Web Application from the browser point of view after all. I realize it sounds a bit crazy but, thinking a bit more about it, it does make sense to me.

I do not think we can simply dismiss the Web App Manifest on the basis of such general statements. If we decide to go for a JSON serialization, we must (also for process reason) make a more thorough analysis of the Web App Manifest, and we also must (I believe) sit down with the people behind that document to hammer out the differences and see how we can reconcile these things. The Web App Manifest document is not final, it has changed already a lot and it is still evolving. We should discuss with them, we may want to get involved with the evolution of that document; I do not think we should just dismiss it.

There is also a very pragmatic reason. Web Manifest based implementations for Web Applications are coming to the fore, albeit in experimental format, in some browser engines (the document lists Gecko and Blink, ie, at some point, Firefox and Chrome…). It would be way easier to build on those internal implementations for a browser based WP handling rather than force a completely disjoint implementation. I think our chances to convince the browser engines to do such a different implementation would be extrememly thin. If we would have to accept some compromise to maximize our chances to have WP processing in browsers: so be it.

[BigBlueHat]

One thing we should keep in mind is that Web App Manifest's are additive authoritative metadata.

It is a very different animal than what we've defined as our intended use of "a manifest."

The Web App Manifest provides additive "install time" metadata not required for any part of the browser-based running of the Web App. At install time this metadata is considered to be authoritative in the context of "launcher" creation. It is also completely metadata--no resources declared just information and configuration settings for creating a launcher to run the actual Web App.

The Web App is still entirely written in HTML, CSS, JavaScript and the assets those utilize--none of which are referenced within the Web App Manifest.

Consequently, Web App Manifest should be completely usable alongside whatever we build in order to provide "installable" Web Publications.

[HadrienGardeur]

@iherman

The "good extensibility mechanism", as I said in the previous comment, is there: we can add additional items ("members") to the JSON file at our heart's content, and the processing includes an explicit step for the extension specific processing. This is not an issue.

You're describing the default behaviour for JSON, where the client simply ignores unknown elements.

That's not what I would call "good extensibility". The specification recently added a registry, which is a good first step towards good extensibility, but there's still a long way to go.

I do not think we can simply dismiss the Web App Manifest on the basis of such general statements. If we decide to go for a JSON serialization, we must (also for process reason) make a more thorough analysis of the Web App Manifest, and we also must (I believe) sit down with the people behind that document to hammer out the differences and see how we can reconcile these things. The Web App Manifest document is not final, it has changed already a lot and it is still evolving. We should discuss with them, we may want to get involved with the evolution of that document; I do not think we should just dismiss it.

My comment started with "I'm not comfortable with the Web App Manifest in its current state".

There is also a very pragmatic reason. Web Manifest based implementations for Web Applications are coming to the fore, albeit in experimental format, in some browser engines (the document lists Gecko and Blink, ie, at some point, Firefox and Chrome…). It would be way easier to build on those internal implementations for a browser based WP handling rather than force a completely disjoint implementation.

Sorry but I'm not buying that argument at all.

Parsing a basic JSON document is easy and is not the real barrier to entry.

The real problem is tied to use cases and what the browsers are supposed to do with that manifest.

For the Web App Manifest this is fairly well understood at this point, the primary purpose is to install a Web App and trigger behaviours related to that installation (such as displaying an install banner in Chrome).

For a Web Publication, we have yet to define these expectations but they will most likely be quite different:

• offline could be the responsability of the UA (vs simply letting a Service Worker do its job)
• navigation could also be partially at least the responsability of the UA (moving between primary resources, navigating from a primary resource to a TOC etc.)
• if we need a way to list publications, I doubt that installing them like apps (with requirements regarding icon and title that are not aligned with our own) is the best way to handle this

Convincing browsers that they should implement any of this (and whatever else we decide) will be the truly difficult part, and how we serialize our manifest has nothing to do with this.

If we really need a Web Publication to behave like a Web App (install banner & app icons on Android for example), we can always add a Web App Manifest to it, but it doesn't mean that we need to use the same syntax at all.

[llemeurfr]

Whether these things are indeed different or not is a question (@iherman)

This is indeed a very important question. We're designing a standard, i.e. expressing the intersection of the needs of all members of the publishing community.

Should every Web Publication, even the simplest, be defined and usable as a Web Application? Should every WP be installable? support an icon, application rating, screenshot and background color property for instance?

I believe that some publishers will be happy to deliver their publications as Web Applications, not all of them.
Therefore I believe that the Web Publication manifest should be a standalone structure, and the WG should study how, when needed, a Web Publication can be released as a Web Application.

This is not to state that we shouldn't be in close contact with the Web Application WG, which has made a great work studying both the structure and processing model of a Web Application: we have a lot to learn from them of the latter part.

[iherman]

My comment is more operational: I believe we should have a discussion with the Web App Manifest authors, sooner rather than later. We did have some discussion at the TPAC in Sapporo (or was it Lisbon?), and the person there was very open and cooperative (but lot has been changed since then). I believe we should try to find a convergence and avoid defining two different manifest formats, processing, etc, on the Web. (Let us not forget that the Web App Manifest document is still a draft.)

I do not claim to understand all the technical details of the Web App Manifest document, let alone the underlying but unwritten aspects of it, and I would think I am not alone with that. Until we have a better understanding I would propose we leave this issue open (we can still decide whether we use JSON and we can still define the major key/value pairs that we would need), and the chairs should try to set up a meeting as soon as possible. Hopefully this can happen before TPAC.

[mattgarrish]

I've added a wiki page that looks at issues with integration with Web App Manifest:
https://github.com/w3c/wpub/wiki/Options-for-Processing-a-Manifest

[HadrienGardeur]

I created a separate issue (#118) to explore how we could potentially extend the Web App Manifest.

My conclusion so far:

This feels almost like inventing a new syntax since there are very few elements that we can re-use from the main Web App Manifest specification.

[baldurbjarnason]

One question we have to ask ourselves is whether we want to allow truly lightweight web publications.

Hypothetical example: do you want to let blog engines turn every standalone post into a single web publication? It'd make sense for them to expose blog series as a unified publication but do we want to allow them to default to having each post be a publication until it's turned into a series?

Individual blog posts could benefit from the metadata and offlining features.

Or if a blog post is too lightweight, how about a single 10 000 word feature in the Atlantic magazine?

I ask because accepting the idea that each site could have hundreds of publications creates some tension with the concept of a Web App. The web app manifest is pretty focused on the idea of a substantial functional bundle of resources whereas web publications as a concept could scale down to individual HTML resources if we let it.

It isn't an impossible tension. We could just accept that basing the manifest on the web app manifest will result in hundreds of publications per site in some instances. There's no technical reason to avoid that scenario.

Has the outcome of the discussion with Web App Manifest authors and implementors been documented somewhere? It seems to me that the decision whether to use the Web App manifest isn't really a technical one but one of culture and UX.

What do browser vendors think about complicating web app manifests with the fairly wide-ranging and varying needs of web publications?

[HadrienGardeur]

Hypothetical example: do you want to let blog engines turn every standalone post into a single web publication? It'd make sense for them to expose blog series as a unified publication but do we want to allow them to default to having each post be a publication until it's turned into a series?

Individual blog posts could benefit from the metadata and offlining features.

Or if a blog post is too lightweight, how about a single 10 000 word feature in the Atlantic magazine?

I think these are all valid use cases for Web Publications.

Has the outcome of the discussion with Web App Manifest authors and implementors been documented somewhere? It seems to me that the decision whether to use the Web App manifest isn't really a technical one but one of culture and UX.

What do browser vendors think about complicating web app manifests with the fairly wide-ranging and varying needs of web publications?

I don't think we've really had this discussion on the UX side of things.

IMO, from a UX perspective Web Publications in browsers mostly make sense for reader modes (or Pocket in the case of Firefox). The WP manifest could enhance these reader modes by providing navigation between resources and offline access in addition to what they already do (user settings, reading modes).

In the case of Pocket and Firefox, there's also a concept of "bookshelf" (I think it's called "My List") as well where enhanced metadata could help.

Such lists (bookmarks would also fall in this category) are a more natural fit for publications than the UI typically dedicated to Web Apps.

[BigBlueHat]

@HadrienGardeur given your conclusion:

This feels almost like inventing a new syntax since there are very few elements that we can re-use from the main Web App Manifest specification.

Why try to put this information in the Web App Manifest?

The Web App Manifest format is not a manifest (as we've used the term), nor does it define a Web App--it simply points to one, limits what it can do, and provides some minimal metadata to facilitate making icons and store listings.

[iherman]

I have yet fully come "back" from my vacations, ie, I am not sure this is 100% relevant for us. That being said: the Web Payment WG has recently published a first public WD for a “Payment Method Manifest”. Their approach, if I understand it well (I only scanned through it) is to define a manifest that refers to a Web App Manifest (actually, several of those) as possible apps that can implement the web payments. There is a separate section on a processing model that relies/refers to the Web App manifest. I do not know whether this is a better approach for us.

It would certainly be possible to invite, e.g., the staff contact of that WG (Ian Jacobs) to talk about the implementation models that WG envisages and whether that would/could be relevant to us.

Again, I have not looked into all the details, and this may be, after all, not for us. But worth having a look.

[HadrienGardeur]

Why try to put this information in the Web App Manifest?

My personal preference is to use the Readium Web Publication Manifest rather than the Web App Manifest.

But exploring both options was necessary, as we have open issues and even specific comments in our draft about using the Web App Manifest for our syntax.

The conclusion of my experiment is that:

• we can barely use any element currently defined in the Web App Manifest
• the use case and UX are too different from what we're looking for

Their approach, if I understand it well (I only scanned through it) is to define a manifest that refers to a Web App Manifest (actually, several of those) as possible apps that can implement the web payments.

We already refer to an entry point into the publication using HTML. I don't think that we need this entry point to be specifically a Web App (or a Progressive Web App, which would require HTTPS and a Service Worker in addition to a Web App Manifest).

[triblondon]

Hi! I sit on the TAG and poked my head into some of the Publications group meetings at TPAC in Burlingame. Since then @TzviyaSiegman was kind enough to ping me recently asking for TAG feedback on this issue. We therefore discussed this at our ongoing F2F today in London and would like to contribute this guidance:

---

It seems to us on the TAG that the Readium manifest format is very unlikely to be considered for support by implementors of general purpose web browsers. Therefore, the question seems to be: is the goal of this group to make a new packaging format for specialist book reading software and devices, or is it to obtain first class support for missing book-related features in the web platform as a whole? If the latter, then it is an order of magnitude more likely to be achieved by building atop existing platform features - notably Web App Manifest and service worker, than creating a separate but similar concept.

We don't want to be discouraging of efforts or dismissive of concerns around conceptual semantic differences between 'books' and 'web apps', in fact I think your group's work makes us realise that there are clear missing parts of the platform that this work could address, and we'd like to see these things come to the web.

In terms of packaging, the WICG packaging standard is at an earlier stage, and now is a perfect time to ensure that the needs of books are addressed in that effort, before it ships. It seems like support is coalescing behind that solution, especially in light of Google's recent announcement that they will depend on WICG's web packaging standard as part of the long term evolution of the AMP project.

We'd be happy to chat as a group with you if that would be helpful - we meet by phone once a week and have upcoming face to face meetings in Tokyo in April and Seattle in July. If you are having any difficulties getting engagement from existing working groups, then the TAG would be glad to help with that.

[HadrienGardeur]

Thanks @triblondon for this feedback, but as detailed in #118 there are many open questions:

• the current draft for Web App Manifest only covers a very small fraction of the Web Publication infoset, if we extend the WAM it would still look like an entirely different manifest (this would barely qualify as building on top of existing platform features)
• our requirements for i18n (which seem to be aligned with what the i18n WG recommends) are also incompatible with how strings are currently expressed in the WAM
• while Service Workers are useful for potential implementations, this is also very different from the way offline access is typically handled in publications: rather than writing a script per publication, it makes a lot more sense for this community to list resources that should be cached
• we haven't received any interest or feedback from browser vendors so far (I'm glad that Microsoft recently joined this group and I hope that this will help), and it's not clear at all what their position will be (if heavily extending the WAM will be deemed as helpful or harmful for instance)

I think we can all agree that building on top of existing features is the best option, but in this case it feels more like building something completely different next to the WAM rather than on top of it, even if we adopt its syntax, processing and WebIDL.

[llemeurfr]

A side question being (for the WAM WG): would the WAM WG be ok to amend in a non backward-compatible way a draft spec already implemented in several browsers, to encompass i18n requirements, add a type ("application" vs "publication") that would trigger specific behaviors (create a bookmark for an app, create an entry in a bookshelf for a publication), split the spec in three parts (the part common to applications and publication, and two specific parts) ... and even change its name (as an publication is not an application, as the "type" illustrates)?

[triblondon]

Is your goal to extend the general web platform? It's certainly useful in this context to understand that.

WAM is implemented by all major general purpose web browsers, is very widely adopted by web authors, and it's absolutely designed as an extensible platform. If you have a proposal that adds the things you need to WAM, we'd love to take a look at that and help you get it adopted.

WAM does handle internationalisation.

rather than writing a script per publication, it makes a lot more sense for this community to list resources that should be cached

We tried that. It didn't work. For the general web, we needed something more flexible.

If you're looking for something simpler or more specialised than service worker, the industry could build something on top of serviceworker, as Google has done with Workbox. You'd build a serviceworker library, which expects to download a list of resources from a well known location, such as /awesomelib-webpub-resources.json, and caches them. Then ship that awesomelib-powered serviceworker with every web publication.

Regarding terminology, non-backwards compatible changes to WAM are a non-starter. But adding a type or at least some other WAM property that marks a web app as a publication to trigger some special purpose seems a not unreasonable extension to WAM! That said, depending on what you want to achieve with the bookshelf feature, registerProtocolHandler might be worth looking at as an alternative.

[HadrienGardeur]

Is your goal to extend the general web platform? It's certainly useful in this context to understand that.

Of course, but JSON-LD and schema.org are also part of the general Web platform, the Web App Manifest is not the only spec out there.

WAM is implemented by all major general purpose web browsers, is very widely adopted by web authors, and it's absolutely designed as an extensible platform. If you have a proposal that adds the things you need to WAM, we'd love to take a look at that and help you get it adopted.

This is covered entirely in #118.

If you're looking for something simpler or more specialised than service worker, the industry could build something on top of serviceworker, as Google has done with Workbox. You'd build a serviceworker library, which expects to download a list of resources from a well known location, such as /awesomelib-webpub-resources.json, and caches them. Then ship that awesomelib-powered serviceworker with every web publication.

If we must embed our own script in every Web Publication to handle offline reading, what else will we be required to handle ourselves?

Can we expect browsers to handle navigation between resources and user settings? Can we expect affordances to trigger a publication reading mode?

If we have to implement everything ourselves using JS, the benefits of using the WAM are inexistent (we're better off having a more specialized manifest that will be easier to use in a polyfill).

[HadrienGardeur]

WAM does handle internationalisation.

Not in a way that's consistent with the requirements expressed by the Japanese publishing industry, and it doesn't fully follow the i18n recommendations either.

WAM can only:

• support a default language and a dir for all members
• cannot include more than one language per member, and it can't specify a dir per member either

Our requirements are aligned with the i18n best practices, which means:

• the manifest can include a default language/direction
• but every member can override them and include multiple languages/directions

[TzviyaSiegman]

Thanks @triblondon for the detailed feedback. @HadrienGardeur, thanks for expressing your concerns. I think it would be a good idea to document individual items (such as specific requirements of Japanese publishers). This can be done in this repo or in WAM. Perhaps we can help make WAM more robust.

[iherman]

Closing per group resolution. See also #199