Documenting implementation status

[anssiko]

As discussed at TPAC, implementation work for the WebNN API is reaching a point web developers can start to experiment with the feature in near future using a native implementation that is integrated into browsers starting with Chromium-based ones. And already today, experimentation can be done with the polyfill implementation that tracks the spec progress. Similarly, we learned at TPAC the Model Loader API implementation work is progressing in parallel.

As a learning from work on other Web APIs, I'm suggesting the group to start an Implementation Status document for the collective WG and CG specs to provide web developers a place to go to to answer their most common questions such as "can I use feature X of Y API in ...". We can also use such a page to document implementation quirks developers should be aware of that do not belong to the specs, as well as instructions how to enable the feature.

For discussion:

First, we should discuss whether this is a good idea. Assuming it is, we should discuss what is the MVP dataset to start with and how to make it easy to maintain by implementers as well as community members, to ensure this remains a useful resource. It is only useful if it is maintained.

One example that has been well received by web developers is the Web Bluetooth Implementation Status page. I think we can borrow ideas from it. We could host such a page at https://webmachinelearning.github.io/ possibly.

Later on, when the code starts to land behind a flag, we can complement this with caniuse.com entries that provide a high-level cross-browser support matrix quite popular among web developers, and link back to the more detailed Implementation Status page.

[dontcallmedom]

a suggestion I wanted to bring up: I think ultimately we will want to have the data integrated into MDN Browser Compat Data (which in turn gets used by e.g. caniuse).

If so, it might be useful to consider adopting the browser-compat-data conventions to document the implementation status. The schema includes structure to document flags/origin-trials as well as a way to document any limitation an implementation might have.

[anssiko]

@dontcallmedom, I think that is a great idea. Which spec we should look at for a good example of such an integration in action? Preferably a spec that is still evolving similarly to WebNN API to get an idea how the flow is for incremental updates.

We are interested in more web developer feedback for our work and I think the suggested MDN integration would help us meet web developers where they are.

[dontcallmedom]

at a high level, adding a spec / set of features in browser-compat-data requires the said feature to be implemented in at least one browser - once that's done, adding the data is a matter of constructing the right JSON file (following the schema described above) and submitting it as a PR. Data gets updated through pull requests as well.

To have the data showing up on MDN, there needs to be reference pages matching the various API entries.

[anssiko]

For Wasm, there's a feature-level compat table (data: features.json) as well as MDN compat table (data: https://github.com/mdn/browser-compat-data/tree/main/javascript/builtins/webassembly). These seem complementary.

For a feature that's early, I think we need both a compat table and a way to document browser- and OS-specific quirks, notes etc. similar to WebBT.

I believe our goal is the MDN reference pages replace the custom implementation status page as things get more stable and quirks ironed out, but such a page will be a helpful resource while the spec and implementations are in active development.

Note to self: see also MDN style guide and Compatibility tables for ideas.

[anssiko]

A few quick updates:

• Web Neural Network API initial Chromium patch landed
• Model Loader API Dev Trial in Chrome OS M102
  Rough ETA/roadmap? model-loader#36 (comment)

See also open and merged patches, #webml hashtag is used by both the APIs.

[anssiko]

Current status:

• The high-level WebNN API implementation status is tracked as part of Candidate Recommendation readiness tracker #240
• The Model Loader API implementation work is currently on pause.

This broader issue will soon become more topical for the WebNN API that is expected to enter Dev Trial in the near future. This issue provides a great opportunity for someone, possibly a new contributor, to help integrate the implementation status data into MDN Browser Compat Data as suggested in #223 (comment)

Please get in touch if you're interested.

Related, we've also made substantial progress with WebNN API web-platform-tests, also tracked in #240. This will help with this task at hand.

[anssiko]

Implementation Status of WebNN Operations:
https://webmachinelearning.github.io/webnn-status/

Future work is to interop with browser-compat-data.

[ibelem]

The initial PR was submitted @mdn/browser-compat-data#22569 @anssiko PTAL

[anssiko]

Thanks @ibelem!

@dontcallmedom is in the best position to review the BCD contribution, or delegate to someone who is.

[ibelem]

@dontcallmedom @anssiko The first version of WebNN BCD data has been merged in mdn/browser-compat-data#22569 and available in BCD release v5.5.16.

E.g.
https://caniuse.com/?search=conv2d
https://caniuse.com/?search=MLGraphBuilder

[anssiko]

Great work! I assume https://webmachinelearning.github.io/webnn-status/ will soon also make use of BCD data.

(Related: I suggested Fyrd/caniuse#6098 (comment) as an improvement idea for caniuse.com.)

[ibelem]

@anssiko

Thanks for the improvement idea for caniuse.com, indeed that's a great point!

When comparing the information that https://webmachinelearning.github.io/webnn-status/ needed and the BCD data structure, I am afraid I still need to maintain the original webnn_status.json file instead of using BCD data completely.

• BCD data is unable to support WebNN mapped ops of each native backends
• BCD data doesn't include WPT tests and test results
• BCD data is unable to cover JavaScript ML Frameworks integration status

Regardless of whether I use BCD directly or not, I need to maintain the additional json file, so I will keep both sides updated for the time being. WDYT?

[anssiko]

@ibelem thanks for reminding me of the important additional data we're tracking. It is reasonable to expect we need to maintain alongside the BCD data also our own webnn_status.json.

We should consider BCD to be the upstream and keep it up to date as a first priority. Only data that does not fit into the BCD schema we should maintain in our own repo. We should structure this in a way we know what's the canonical source for which data.

To enable broader contributions it'd be good to document the WebNN implementation status contribution mechanics. For inspiration, see https://github.com/Fyrd/caniuse/blob/main/README.md

Can you take the first stab on this? Thanks!

[ibelem]

We should consider BCD to be the upstream and keep it up to date as a first priority.

Definitely, this is our intention.

@anssiko Please review the webmachinelearning/webmachinelearning.github.io#67 for the WebNN implementation status contribution mechanics.

[anssiko]

Thanks to sustained major contributions by @ibelem we now have:

• a detailed implementation status page that covers both browser implementations and JS ML framework integration status for the WebNN API
• upstreamed a schema-compliant subset of this data to the Browser Compat Data repository used by MDN, caniuse.com (example), and more as a data source
• published contributing guidelines to help new contributors take part in this effort

Thank you @ibelem for your work on this! I want to also extend my thank you to people working on BCD @Elchi3 who helped review our initial contribution in a timely manner and @dontcallmedom for all the guidance on this journey.

I can't wait to see what web developers will build with this web technology now equipped with better developer documentation to aid them. 🚀

With this summary, I feel good closing this issue as completed. Maintenance and improvement work continues and further contributions are welcome.