Add optional translations member

[FluorescentHallucinogen]

Sometimes it's difficult to find an app or game on a specific (especially, unpopular) language.

Conversely some apps (games) are available only in one language (e.g. Chinese).

Since we want PWAs to be presented in app stores, what about adding optional "translations"/"languages"/"supported_languages" array of strings member?

Something like:

"translations": ["en","fr","it"]

It should help to find (filter/exclude) apps by supported languages.

[marcoscaceres]

Sometimes it's difficult to find an app or game on a specific (especially, unpopular) language.

Ok, let's start here before talking about solutions. Generally, to find an application, the user goes to a search engine (or app store) and searches for that app. The search engine may know the user's language preferences or the search engine derives the language from geoip or whatever.

Which search engine or app store is failing to do this?

[CheloXL]

@marcoscaceres how would I describe in my manifest that my app supports multiple languages? Right now, there is only the lang entry (and that's for specifying the primary language). I'm ok with that (after all, the app HAS to display something if the user is requesting a language not supported by the app), but it would be nice if the manifest can also tell search engines/stores right upfront which languages the app supports.

[marcoscaceres]

@marcoscaceres how would I describe in my manifest that my app supports multiple languages? ... but it would be nice if the manifest can also tell search engines/stores right upfront which languages the app supports.

But as you say, it falls into the "nice to have" bucket. The problem is difficult: we made and explicit design choice when creating the manifest format that it shouldn't be a place to make claims, such as "this application supports French".

Rather, we want the applications themselves to support languages in their content. Search engines can then discover that content through various means, like following links (same as a normal user).

[marcoscaceres]

We have some i18n guidance here:
https://www.w3.org/TR/appmanifest/#internationalization

[marcoscaceres]

fixed a typo above, I was supposed to say "shouldn't be a place to make claims". Apologies for that.

[mgiuca]

I've got similar concerns about this (and just this week, we had a team having trouble localizing their manifest). I'm starting to think we could explicitly support translations of user-visible strings (like the name and description) in the manifest itself, which would implicitly give stores an indication of what languages the app supports.

The link above gives two localization suggestions:

1. Dynamically setting the language (using in-site UI to point at a different manifest URL for each language).
2. Using content-negotiation, or geotargeting, etc. on the server (serving a different manifest contents based on the user's chosen language).

Neither of these are great options. Option 1 interferes with the idea that "manifest == app". Depending on how the implementation keys apps off manifest URLs, changing the URL (even changing the query string) could cause the user agent to think of it as a different app. For example if you install manifest.php?lang=en on Chrome on Machine A, then install manifest.php?lang=fr on Chrome on Machine B, then Chrome Sync happens, we may consider these different apps and you'd end up with both versions on both machines. This is user-agent specific issue, but only because we haven't properly specified how apps are keyed. (#586)

Option 2 fixes that, but it means you have to have dynamic logic on the server side, which precludes any static hosting solutions like github.io. (Whereas the rest of the site UI can be localized in JavaScript, the manifest data cannot be.)

Finally, if the language solutions involve literally changing the bytes of the manifest, it means that when the user changes language, apps will remain in the originally installed language until the user agent re-pings the server with the new Accept-Language, redownloads the manifest, and updates the metadata. (On Chrome Desktop, we haven't even implemented updating yet, so you're stuck with your original language until you update.) Whereas if the manifest declaratively included all the translations, the user agent could automatically update all the UI with the correct strings as soon as the user changes language.

Therefore, I think it would be better if we actually let you put your localizations in the one manifest file itself. What are the downsides of this (besides potentially a much larger file)?

[CheloXL]

I'm on the same boat as @mgiuca right now. I'm using content-negotiation to provide a localized version of the manifest/app. This works fine for the user (people don't usually change the browser language on the fly... they don't change it at all, or they setup it once and be with it).
But that tells nothing to a search engine. Search engines will not hit my app with different languages just to see if I'm providing content-negotiation.
Why this can't be handled in the same vein as images? So, you have a "languages" entry (array of objects) where each entry can have a lang tag, dir, short_name, name and description (I'm not even sure if short_name/name are really necessary, as usually you don't translate the name of your app).
Of course the main/default ones are the ones outside the languages entry.
This means nothing for implementors (they don't have to do anything with this entry) but helps discoverability of the app, both in a store/by a search engine.

[marcoscaceres]

But that tells nothing to a search engine.

I’m still a bit confused as to which search engine or store are you referring to specifically?

[CheloXL]

Search engine? All of them. I'm not aware of any search engine that does language negotiation.

When a SE hits my app, my app tries to negotiate language based on accept-language header. Nothing is sent by the SE, so my app defaults to English.... and that's it. Even if the SE sends an accept-language header, it will send one. I'm quite sure that the SE will not try to see if my app changes what it returns by changing the headers.

The SE has no way to know my app supports multiple languages. There are no hints anywhere.
If the manifest had a place where I can define them, the SE can read it and use that meta-information for indexing purposes.

[marcoscaceres]

Search engine? All of them. I'm not aware of any search engine that does language negotiation.

Sorry, I wasn't very clear: Which search engine processes/supports web manifest format?

I ask because we are reluctant to add things to the web manifest format unless we have actual commitment implementation from a search engine or store.

Put differently, imagine we add this today without explicit support from a search engine: One year later, nothing changes because no actually implements the feature.

When a SE hits my app, my app tries to negotiate language based on accept-language header.

Ok, but do you also provide a manual way for the user to switch language? For instance, I speak multiple languages, I might prefer my work stuff in English, but my social apps in Spanish.

[marcoscaceres]

@mgiuca wrote:

Option 2 fixes that, but it means you have to have dynamic logic on the server side, which precludes any static hosting solutions like github.io. (Whereas the rest of the site UI can be localized in JavaScript, the manifest data cannot be.)

This is true. That assumption we originally made was that the user would need to re-authenticate after the application syncs. When they re-authenticate, they would pull down their app-specific profile data, including their language preferences.

[CheloXL]

Sorry, I wasn't very clear: Which search engine processes/supports web manifest format?

Ah... Right now? I don't know if any. But SE originally neither indexed flash and at some point they started to.

Put differently, imagine we add this today without explicit support from a search engine: One year later, nothing changes because no actually implements the feature.

I get that, but the same can apply to other features that were suggested to be added (screenshoots, masked icons, etc). Since this is a living standard, it is expected to have features that nobody had implemented.

Ok, but do you also provide a manual way for the user to switch language? For instance, I speak multiple languages, I might prefer my work stuff in English, but my social apps in Spanish.

Right now, no. Changing the language means not only to refresh the whole app, but also expires cached assets on SW, the SW itself, etc. In a future, probably, but in any case, the app is behind a login page, so even if we let the user change the language, that means nothing for search engines. Also, the only way for a SE to be able to index different languages of the app is to have different URLs for each language/page, and while for a website that works perfectly (and I have plenty of experience on that), for an app this only add a lot of complexity to the developer.

[marcoscaceres]

I get that, but the same can apply to other features that were suggested to be added (screenshoots, masked icons, etc). Since this is a living standard, it is expected to have features that nobody had implemented.

So, no, fortunately. For instance, the Microsoft Store folks are the ones that requested Screenshots. And masked icons was added because it's already implemented in Android, so there was fairly clear indication from people working on the Android side that they would be willing to implement that. The same for all other things we've added: they are all implementer backed.

However, that doesn't mean we can't specify a solution. It just means that we might then need to do some work to pitch it to either browser vendors and/or search engines.

Also, the only way for a SE to be able to index different languages of the app is to have different URLs for each language/page, and while for a website that works perfectly (and I have plenty of experience on that), for an app this only add a lot of complexity to the developer.

So, I think this is the problem we need to solve generally. The HTML spec suggests using rel="alternate":

<link rel=alternate href="/en/html" hreflang=en type=text/html title="English HTML">
<link rel=alternate href="/fr/html" hreflang=fr type=text/html title="French HTML">

The question is if the above insufficient to meet the use case, or if we do need to specify yet another mechanism.

[mgiuca]

Good question / possible solution. I think this solves a lot of the problems outlined above:

1. It means we have a canonical manifest URL across all languages.
2. It means developers can present all the languages on a static hosting provider (no server-side logic).

I think a problem with that approach is that user agents may wish to download all the languages up front, rather than just the user's preferred language.

The reason a UA would do this is to create a fully internationalized representation of the app in the native OS's preferred form, so that if the user switches OS language, the web app seamlessly changes language as a native app would. For example, on Linux we'd generate a .desktop file with all the localizations in it. On Android, we'd generate an Android Manifest with the same. Thus, switching language doesn't rely on the user agent to realise that the language has changed and fetch the language-alternative resources.

So in order to create, say, the localized .desktop file, we'd have to go and fetch every single rel=alternate manifest from the server, creating a lot of requests up front.

Also, would we allow the "alternate" manifests to have just the localized strings (augmenting the base manifest), or would they need to be a complete copy of the entire manifest? That could be a lot more total bytes to download than just having all the strings in one. [Edit: From the HTML spec, it says "referencing an alternate representation of the current document", so I don't think we could interpret it as "just provide the keys you want to override", it would have to be a complete copy.]

An approach we came up with is splitting it into two separate files: the manifest and a separate "translations" file with all the translations. That way, if you need to fetch the manifest for just the icons (or another non-string resource), you don't need to fetch the potentially much larger translations file.

[marcoscaceres]

For example, on Linux we'd generate a .desktop file with all the localizations in it. On Android, we'd generate an Android Manifest with the same. Thus, switching language doesn't rely on the user agent to realise that the language has changed and fetch the language-alternative resources.

I think this is interesting - and I think it's something we looked at really early on in the standardization process (back in 2014). We could see about salvaging parts of #211.

[mgiuca]

I would be in favour of bringing in something like #211 proposes, where any string can be replaced by an object with multiple languages in it.

I am concerned with the manifest becoming too large (since Chrome, at least, now blocks on fetching the manifest before showing a tab icon, b/c we get the icon from the manifest now). So we might consider putting language-specific overrides in another file.

[marcoscaceres]

It would be great if you can explore alternatives... might not be a big deal in a HTTP/2 or /3 world, but splitting the file might still actually be slower in a HTTP 1.1 world (specially if all the alternatives need to be downloaded). We might need to do some experimentation.

To keep backwards compatibility, we might need to do:

{
  "alternatives": {
     "en": "path/to/en.manifest",
     "fr": "path/to/fr.manifest",
     "it": "path/to/it.manifest"
   }
}

[marcoscaceres]

The above also retains compatibility with the advice in the spec, I think.

[marcoscaceres]

@mgiuca, just wanted to follow up on...

I am concerned with the manifest becoming too large (since Chrome, at least, now blocks on fetching the manifest before showing a tab icon, b/c we get the icon from the manifest now).

I'm also concerned about maintainability. The nested object get really hard to read/write - so I'd be in favor of separate resources.

[mgiuca]

I'm also concerned about maintainability. The nested object get really hard to read/write - so I'd be in favor of separate resources.

As in one-file-per-language, or a separate file for translations?

I will write up an analysis of the pros and cons of each method.

[marcoscaceres]

yes, "one-file-per-language". I think a single translation file with multiple languages might be difficult to maintain.

[mgiuca]

I wrote up a doc with the various options:

https://docs.google.com/document/d/1Vwi_u7RP-HrzCz389Qp4EPKrWhanetu8ivcJmGbhwFc/edit

Please comment.

[CheloXL]

I like option 3, either A or B. With B you don't even have to change anything on the manifest and I think it's better for SE, but again, both are good.

One thing to note (and I don't see that in the document) is that the localized manifests could be only a subset of the original manifest, with changes only on the keys you want to localize.

So, for example, the usual keys would be title, short_desc and desc. You would end up with a complete manifest in your default language, and N manifests for each translation, that only have 3 key entries.
But this opens the possibility to also have different in-app screenshoots, for example. That is key for app-stores, where you would probably want to display the screenshoots in the same language as the store itself.

I like it...

[nuxodin]

We have the "related_applications"-memeber
Would it make sense to add a lang attribute there?

"related_applications": [{
"platform": "Web",
"lang":'de-CH',
"url":"example.com/de/ch/manifest.json"
}, ...

[marcoscaceres]

Great summary, thanks @mgiuca for putting that doc together! I remain a fan of 3.

[mgiuca]

@CheloXL

One thing to note (and I don't see that in the document) is that the localized manifests could be only a subset of the original manifest, with changes only on the keys you want to localize.

That was the intention (which is why I showed "icons" in all the manifests) but I've added a paragraph explicitly stating this.

So, for example, the usual keys would be title, short_desc and desc. You would end up with a complete manifest in your default language, and N manifests for each translation, that only have 3 key entries.
But this opens the possibility to also have different in-app screenshoots, for example. That is key for app-stores, where you would probably want to display the screenshoots in the same language as the store itself.

I actually don't think we should allow this. If you allow it, you have to mandate it (since we can't have different user agents doing different things). If we're talking about all members being localizable, then that means, for example, we have to allow the possibility that English and French have different scope URLs, and that means we might need to update our link capturing logic when the user changes their OS language. That is a huge complexity spike for us.

We could allow it for a few additional members, like screenshots and maybe icons (though even icons invites a lot of difficulty in the user agent).

@nuxodin

We have the "related_applications"-memeber
Would it make sense to add a lang attribute there?

"related_applications": [{
"platform": "Web",
"lang":'de-CH',
"url":"example.com/de/ch/manifest.json"
}, ...

As an alternative to "translations" in my option (3a)? I don't really want to do this. That implies that each translated manifest is a separate application. For one thing, these language-specific manifest files are not supposed to be complete manifests, but supplemental. For another thing, the whole point of this feature request is so that we can start thinking about different languages as views of the same app, not separate apps.

A new member for a new purpose.

[kenchris]

I don't really want to do this. That implies that each translated manifest is a separate application.

Yes, that would be pretty bad

[CheloXL]

I actually don't think we should allow this. If you allow it, you have to mandate it (since we can't have different user agents doing different things). If we're talking about all members being localizable, then that means, for example, we have to allow the possibility that English and French have different scope URLs, and that means we might need to update our link capturing logic when the user changes their OS language. That is a huge complexity spike for us.

We could allow it for a few additional members, like screenshots and maybe icons (though even icons invites a lot of difficulty in the user agent).

@mgiuca
I understand, and in fact, while I didn't comment on the icons sections, I also think that icons should not be localized. Never though of having different scopes URLs. Thanks for pointing it out.

[marcoscaceres]

That is a huge complexity spike for us.

Ah yeah, forgot about that 😅 I think that’s why we explicitly called out the localizable members in the first place.

[FluorescentHallucinogen]

@CheloXL

I also think that icons should not be localized

Many companies use different logos for different markets. For example, VK (ВКонтакте), the most popular website in Russia and the 9th most visited website in the world, uses the different icons in their app for Russian-speaking and other countries:

And yes, they have PWA: https://pwa-directory.appspot.com/pwas/5739494557089792. 😊

[marcoscaceres]

Thanks @FluorescentHallucinogen. Using "content-negotiation" is indeed how we've always recommended doing this:
https://www.w3.org/TR/appmanifest/#internationalization

Pete also describes using a UI for language selection too, which is also what we recommend in the spec.

Having said that, control over server stuff is challenging (although, Firebase makes it pretty easy!) - so I'm sympathetic to us providing a declarative solution in the manifest itself.

[loubrett]

I've started an explainer for this here: https://github.com/WICG/manifest-incubations/blob/gh-pages/translations-explainer.md

This was based on the proposal by @aarongustafson above.

Can we close this issue and open issues on the manifest-incubations repo as needed instead?

[marcoscaceres]

Thanks for putting that together @loubrett.

Can we close this issue and open issues on the manifest-incubations repo as needed instead?

This has considerable interest and is a Candidate Recommendation "blocker" + it's not a huge feature that requires massive changes to user agents (just to processing)... it might be good to just bring it over as pull request so we can work out how the processing will actually work.

In particular, will we allow localizing everything? Are they just simple overrides?

[r12a]

I've started an explainer for this here: https://github.com/WICG/manifest-incubations/blob/gh-pages/translations-explainer.md

Is there a way of making all languages equal, rather than having one apparently main language and then all these others over there?

[tomayac]

Is there a way of making all languages equal, rather than having one apparently main language and then all these others over there?

A possible way of doing this would be to have all non-translatable properties on the root level, and everything else nested in translations. This would then mean that language on the root level would need to be not provided.

[aarongustafson]

@loubrett Thanks for putting that explainer together, I had not gotten around to it yet. Mind adding me as co-author? I’ve been working on this for a while in various capacities and plan to continue chipping away at the issue.

[loubrett]

In particular, will we allow localizing everything? Are they just simple overrides?

No we shouldn't allow localizing everything, @mgiuca commented earlier with a list of the fields that can be localized. Yes I think they should just be simple overrides.

Mind adding me as co-author?

Yes of course - added.

[mgiuca]

Is there a way of making all languages equal, rather than having one apparently main language and then all these others over there?

The idea of a "main language" is needed for two reasons:

• Backwards compatibility. A browser that doesn't know about the new translations feature needs to read the old data which is at the top level.
• Having a default. If the user's preferred language(s) are not supported, we can fall back to the "main language" of the manifest.

The concept of a "primary language" already exists in the Manifest, in the "lang" member. So "primary language" is something we need to continue supporting, but it doesn't necessarily mean English, and it doesn't necessarily mean the browser is unaware of what language it is.

[loubrett]

There's some discussion about aligning the translations and dark mode proposals over on #975.

[aphillips]

Playing a bit of catch-up here. I am not advocating for/against anything here, but re-raising this with the I18N WG. If I might suggest, this looks like an opportunity for a deeper discussion between our WGs (and maybe also ECMA-402 or Unicode's MessageFormat WG).

@mguica noted:

The idea of a "main language" is needed for two reasons:

I agree that there should be a default language. This often corresponds with the source language (the language that developers working on the code write into resources and which is later translated). In most systems, this also corresponds to the root locale resource file. That is, the fallback when searching for translations and the fallback in the locale system (that formats your dates and numbers and such) are on the same hierarchy. That is, the lang is more likely to be und or blank rather than being (e.g.) en (or some other specific language).

@marcoscaceres is correct that most platforms have the advantage of being able to bundle separate resource files without the penalty of having to separately download each. Web applications often bundle or group languages into blocks to avoid having multiple HTTP GETs in order to populate their resources.

I'll note that most localization processes are built around and have a strong preference for taking a "source file" and translating the contents to produce a "target language file". It is much more complicated for most localization teams to deal with multi-language files (and multilingual files are harder to work with--consider a file with Japanese, Arabic, Hindi, and French in it!). Even with good Unicode support, the chances for damage is higher when things are mixed together.

@FluorescentHallucinogen Thanks for pointing out the icon preferences example! Note that this might not be linked to locale, since a locale is not the same thing as a customer's physical location. Icon changes and other branding variations are often linked to country- or region-specific services or app versions instead (and other things, such as regulatory requirements or data retention policy is also often linked to business locality).

Cycling back to the very start of the issue, note that language/locale matching is important here. Examples of language tags like en, fr, and zh ignore that there are often regional variations (pt-BR for example) or script variations (sr-Cyrl, zh-Hans, etc.) in the translations--or that app authors want to support more locales (without having to list them) than they provide localization for.

[calidion]

String localization should be a native feature for all strings, so it is not proper to call it translation.
A string is can be as simple as a very primitive string like this, which only support one language:

 "Hello world!"

or it can be a multi language supported one like this:

{
   "en": " Hello World!",
   "en-us": " Hello World!",
   "zh": "你好世界!"
   "zh-cn": "你好世界!"
   // ... 
}

[christianliebel]

@calidion We are addressing this in PR #1101. You will be able to specify multiple languages, as you suggested. Here is an example of how it will work: https://pr-preview.s3.amazonaws.com/w3c/manifest/pull/1101.html#x_localized-members

[christianliebel]

#1101 would add localization support. It seems that this issue was originally intended to identify applications that support certain languages. After that, the discussion has moved on to translations/localization. The new *_localized members do not directly address the original concern of this issue, but users could derive this information from the localizations provided, e.g. for the name of the application.