Web Translation API

[domenic]

こんにちは TAG-さん!

I'm requesting a TAG review of Web Translation API.

Browsers are increasingly offering language translation to their users. Such translation capabilities can also be useful to web developers. This is especially the case when browser's built-in translation abilities cannot help, such as:

• translating user input or other interactive features;
• pages with complicated DOMs which trip up browser translation;
• providing in-page UI to start the translation; or
• translating content that is not in the DOM, e.g. spoken content.

To perform translation in such cases, web sites currently have to either call out to cloud APIs, or bring their own translation models and run them using technologies like WebAssembly and WebGPU. This proposal introduces a new JavaScript API for exposing a browser's existing language translation abilities to web pages, so that if present, they can serve as a simpler and less resource-intensive alternative.

• Explainer (minimally containing user needs and example code): https://github.com/WICG/translation-api
• User research: none
• Security and Privacy self-review: https://github.com/WICG/translation-api/blob/main/security-privacy-questionnaire.md
• GitHub repo (if you prefer feedback filed there): https://github.com/WICG/translation-api
• Primary contacts (and their relationship to the specification):
  • Domenic Denicola (@domenic), Google, spec-writer
• Organization/project driving the design: Google Chrome
• External status/issue trackers for this feature (publicly visible, e.g. Chrome Status): https://chromestatus.com/feature/5172811302961152

Further details:

• I have reviewed the TAG's Web Platform Design Principles
• The group where the incubation/design work on this is being done (or is intended to be done in the future): WICG
• The group where standardization of this work is intended to be done ("unknown" if not known): unknown. The Web Machine Learning W3C Working Group seems to explicitly rule out these sorts of high-level APIs in its charter. Maybe that can be amended though.
• Existing major pieces of multi-stakeholder review or discussion of this design:
  • Web Translation API WebKit/standards-positions#339
  • Web Translation API mozilla/standards-positions#1015
  • Web Translation API explainer 2024-05-07 > 2024-06-06 w3c/i18n-request#231
• Major unresolved issues with or opposition to this design:
  • It's unclear whether including translation from an unknown source language is worth including, or whether we should require web developers to do a two-step detect-language + translate language process. Explainer discussion, Should we include translation with an unknown source language? webmachinelearning/translation-api#1
  • The exact way in which language tags should be handled, especially to give good interoperability, is still unclear. Explainer discussion, Language tag handling webmachinelearning/translation-api#2
  • Naive implementations that give the site direct access to info about which language packs are downloaded can provide information that fingerprints the user. Various mitigations are possible but it's unclear which strike the best balance or are most feasible. Explainer discussion, Preventing fingerprinting via detecting the presence of downloaded language packs webmachinelearning/translation-api#3
  • We might want to expose some information about whether the translation is on-device or not, and about the model version. But this could also be harmful for interop. See explainer discussion at the bottom of the "Goals" section.
• This work is being funded by: Google

[shivaylamba]

I like this idea. And especially giving the capability to the user to choose between either using on-device or cloud based models. Given how performant and efficient on-device AI models have become using WASM/WebGPU, we can definitely vouch for the translations to take place while also minimizing risk of any data being sent to cloud.

[jasonmayes]

I think having an auto detect option could be really cool. Scenario: Imagine you are in a Google Meet video call and everyone is from a different country. This way, you could make a Chrome Extension that auto detects the language and is able to translate for each person speaking even if language changes between detection sessions. That would be super useful.

It would be good if you are thinking of hyrbid approach for the programmer to explicitly require offline on device model inference here in case there are privacy aspects they need to adhere to for their application. I think 3 options are useful:

1. client side (forces on device only - if cant run for some reason just fails with error to capture)
2. hybrid (browser chooses based on network conditions / device capabilities) and uses the right one in the moment
3. server side (forces server side only)

[shivaylamba]

I think having an auto detect option could be really cool. Scenario: Imagine you are in a Google Meet video call and everyone is from a different country. This way, you could make a Chrome Extension that auto detects the language and is able to translate for each person speaking even if language changes between detection sessions. That would be super useful.

It would be good if you are thinking of hyrbid approach for the programmer to explicitly require offline on device model inference here in case there are privacy aspects they need to adhere to for their application. I think 3 options are useful:

1. client side (forces on device only - if cant run for some reason just fails with error to capture)
2. hybrid (browser chooses based on network conditions / device capabilities) and uses the right one in the moment
3. server side (forces server side only)

I think we can also recommend the user based on their hardware, the type of model inference to be used (cloud vs on device). For instance, if the user's hardware doesn't have a GPU and has limited RAM, it might be better suited for cloud inference. Let me know what do you think @jasonmayes

[matatk]

We agree with and support the user need. Here are our thoughts...

1. It would be good to have a list of use cases. We could think of some from our own experience, but they may be different than the ones you had in mind. Having an explicit list of use cases ensures that everyone is on the same page.

2. Please continue chatting with the I18N WG folks about issue 11, and streaming input support.

3. We're concerned about the use of the network. Specifically, use of the network to download a model, or use of the network to actually perform the translation, could introduce both delay and privacy issues. Is it possible for the developer to specify: "only do this if network access is not required"? We feel that differentiation between fast-local, slow-local (i.e. with downlaod), and remote/cloud-based cases is important for MVP.

4. We loved the approach you propose to partitioning, and using a fake download, to mitigate fingerprinting!

5. We recommend a translation-specific namespace instead of ai.

6. Why is a separate namespace needed at all? We understand these objects are not constructible due to the asynchronicity, but since they are creating instances of the same class, making this obvious by adding the factory as a static method of this class seems more consistent with precedent. Same for the capabilities() method, we don't understand why this needs to live in a different namespace, and we think that the more objects this API is spread across, the harder it will be for authors to understand how the different parts fit together.

7. We think there should be a prominent note encouraging developers to make use of professional translation of pre-existing content rather than doing automatic translation wherever they can - for both accuracy and efficiency reasons.

8. It seems to make more sense, and help simplify the API and alleviate some privacy concerns if the UA renders the download progress bar.

9. We did wonder if it would make sense to have a single object for the detection and translation, since they are so related (and often detection is the first step to translation). Was this direction explored?

[domenic]

Thanks for the review!

1. It would be good to have a list of use cases. We could think of some from our own experience, but they may be different than the ones you had in mind. Having an explicit list of use cases ensures that everyone is on the same page.

I believe these are listed in the first paragraph of the explainer. https://github.com/WICG/translation-api/blob/main/README.md#explainer-for-the-web-translation-and-language-detection-apis

3. We're concerned about the use of the network. Specifically, use of the network to download a model, or use of the network to actually perform the translation, could introduce both delay and privacy issues. Is it possible for the developer to specify: "only do this if network access is not required"? We feel that differentiation between fast-local, slow-local (i.e. with downlaod), and remote/cloud-based cases is important for MVP.

It is possible for the developer to avoid downloading the model, if the browser intends to support on-device translation, by checking if capabilities.available is "readily" (as opposed to "after-download").

We haven't yet exposed whether the translation is done entirely on-device or through cloud services, because doing so could possibly cause developers to write code that excludes certain browsers. But, we understand this could be worthwhile. This is mentioned in https://github.com/WICG/translation-api/blob/main/README.md#goals . We'll closely monitor this space, to find out if there are developers who need this ability, and/or whether any browsers actually plan to implement using cloud services.

4. We loved the approach you propose to partitioning, and using a fake download, to mitigate fingerprinting!

Thanks for the kind words, although at least the fake downloads idea isn't looking too promising at the moment. webmachinelearning/translation-api#10

5. We recommend a translation-specific namespace instead of ai.

This is related to some ongoing work on other AI model-based APIs which are not yet at the stage of being ready for TAG review. We want them all to share a namespace and a set of common API patterns (e.g. sibling create() and capabilities() methods; "no"/"readily"/"after-download" availability signals; destroy() methods; specific AbortSignal integration patterns; etc.)

I understand it can be hard to judge this in the absence of other reviewable explainers, so we can revisit this later when we make more progress on those. Stay tuned!!

6. Why is a separate namespace needed at all? We understand these objects are not constructible due to the asynchronicity, but since they are creating instances of the same class, making this obvious by adding the factory as a static method of this class seems more consistent with precedent. Same for the capabilities() method, we don't understand why this needs to live in a different namespace, and we think that the more objects this API is spread across, the harder it will be for authors to understand how the different parts fit together.

I thought about this avenue as well.

First, to clarify, we do need a separate capabilities() method so that web developers can determine model capabilities without initiating a create operation. (Which can be expensive, both in bandwidth and in GPU memory.) So we cannot merge that into the translator object. And, this method needs to be asynchronous, as the source of truth for the capabilities information will not generally be in the same process as the WindowOrWorkerGlobalScope. (We could proactively load the capabilities information into every WindowOrWorkerGlobalScope, but that would cause all sites, including those not using these APIs, to pay the cost. Which is undesirable.)

So I think what you're suggesting ends up converting the API from something like

const capabilities = await ai.translator.capabilities();
const translator = await ai.translator.create();

to something like

const capabilities = await AITranslatorCapabilities.get();
const translator = await AITranslator.create();

I think this is a viable direction. A bit uglier in my opinion, but if the goal is to minimize the number of namespaces, then it does work. We can keep it as a possibility, and see which web developers prefer, or if other arguments appear on either side.

8. It seems to make more sense, and help simplify the API and alleviate some privacy concerns if the UA renders the download progress bar.

The exact UI signals for when these APIs are in use is definitely worth exploring. Browser UI teams are not always excited about adding "noise" to what the user sees, but if we end up needing a permission prompt or similar anyway for privacy reasons, maybe we could convince them to add in some progress measures.

9. We did wonder if it would make sense to have a single object for the detection and translation, since they are so related (and often detection is the first step to translation). Was this direction explored?

To some extent yes. Before webmachinelearning/translation-api@2cb6637 the APIs were more tighly coupled, both existing on a self.translation API. We still had separate detector and translator objects, though. This seems necessary, because a translator has a specific source/target language pair associated with it, and a detector does not.

We separated the APIs even more once we looked into the possible implementation strategies. It turns out that language detector models and translation models are generally quite different. And we wanted to allow browsers to take advantage of these differences, instead of forcing them to unify to a lowest-common-denominator, or expose strange inconsistencies to web developers.

For example, you can find small off-the-shelf language detector models supporting over 80 languages. (If I am reading this MDN page correctly correctly, both Firefox and Chrome use such a model for the Web Extensions i18n.detectLanguage() API.) But, for example, Firefox's language translation models support 10 languages. In our previous design, we had a single supportedLanguages() method, which doesn't make sense given such a setup.

A related question is discussed in https://github.com/WICG/translation-api/blob/main/README.md#allowing-unknown-source-languages-for-translation.

[jasonmayes]

Just sharing an example I was tagged in recently showing how JS users are using Web AI already to do real-time translation in browser entirely client side - may be useful to see use cases by real existing users that could help shape discussion here:
https://twitter.com/thorwebdev/status/1816745831225290909

[plinss]

So I think what you're suggesting ends up converting the API from something like

const capabilities = await ai.translator.capabilities();
const translator = await ai.translator.create();

to something like

const capabilities = await AITranslatorCapabilities.get();
const translator = await AITranslator.create();

Capabilities could simply be a static method on the translator, no?

const capabilities = await Translator.capabilities();

[domenic]

Capabilities could simply be a static method on the translator, no?

Sure, either way, although that's less symmetric than having each class vend its own instances, and takes us back toward kinda using classes as namespaces (just this time with static methods).

[martinthomson]

5. We recommend a translation-specific namespace instead of ai.

This is related to some ongoing work on other AI model-based APIs which are not yet at the stage of being ready for TAG review. We want them all to share a namespace and a set of common API patterns (e.g. sibling create() and capabilities() methods; "no"/"readily"/"after-download" availability signals; destroy() methods; specific AbortSignal integration patterns; etc.)

One wonders if, in the future, this will be as meaningless as an API called computer or software. Please reconsider.

[LeaVerou]

Capabilities could simply be a static method on the translator, no?

Sure, either way, although that's less symmetric than having each class vend its own instances, and takes us back toward kinda using classes as namespaces (just this time with static methods).

IMO It’s about what makes more sense in terms of entity-relationships. The human mental model is that we’re querying the capabilities of the translator; creating a TranslatorCapabilities object ¹ is not a natural language construct.

Which actually makes me wonder if we need this object at all. Why not simply an async getter and an async function on Translator?

Footnotes

1. as @martinthomson said, please reconsider using "AI" as part of the name for any of these. ↩

[plinss]

IMO It’s about what makes more sense in terms of entity-relationships. The human mental model is that we’re querying the capabilities of the translator; creating a TranslatorCapabilities object is not a natural language construct.

Agreed, if authors care about getting the capabilities of the translator, querying the translator directly makes more sense. Having each class vend its own instances is an example of putting theoretical purity of an arbitrary design pattern over user needs

[domenic]

Which actually makes me wonder if we need this object at all. Why not simply an async getter and an async function on Translator?

What type would that async getter/function return?

[torgo]

Hi Domenic - Our feedback that we sent above stands, particularly regarding the name space of this API. We don't think it belongs in a .ai namespace. We think that the name spaces should be funciton-led rather than technology-used-to-deliver-the-function-led. Your proposal under point 6 (AITranslatorCapabilities) is a step forward but we would prefer Translator.capabilities(). We also think your response to point 3 of our feedback is fine. We also would strongly encourage that the spec include a non-normative note encouraging the use of professional translation services where possible (point 7 of our original feedback). Please consider this. We're going to close as satisfied with concerns due to these concerns. Given the current state of the spec, and lack of venue, we understand this to be an early review. We expect that addressing the points above will be important for further standardization. ✨