[Discussion] WebNN backend

[huningxin]

WebNN (Web Neural Network) API is a new proposal that allows to access hardware acceleration (CPU/GPU/AI) for neural networks. WebNN is currently being incubated within W3C Machine Learning for the Web Community Group (WebML CG).

It is expected that the JavaScript ML frameworks are able to leverage WebNN API to offload computation to native (refer to framework-level use cases). So, as an investigation, @pinzhenx and me prototyped a WebNN backend for ONNX.js. Our prototype introduces WebNNBackend , WebNNSessionHandler, WebNNInferenceHandler, WebNNGraphNode and WebNNGraphOp. When loading a ONNX model, the WebNNSessionHandler.transformGraph will try to partition a sub-graph that can be offloaded to WebNN and rewrite the sub-graph with a custom WebNNGraphOp. When running the model, the WebNNGraphOp would first build a equivalent WebNN graph based on the nodes of the sub-graph, and then compile and execute the WebNN graph by native.

When testing on Chromium WebNN prototype, we observed good model inference performance speedup on both CPU and GPU comparing to existing WASM and WebGL backend. The following chart shows the performance results collected for different backends (WASM, WebGL, WebNN-CPU, WebNN-GPU) by the modified ONNX.js Squeezenet and Resnet50 examples.

NOTE: running the examples in vanilla browsers would just use the WebNN polyfill, so you may not observe the performance boost.

The specs of the machine that was used to collect the data are:

• OS: Ubuntu 16.04.5 LTS
• Browser: Chromium WebNN prototype (75.0.3739.0), WebNN-CPU is based on MKL-DNN, WebNN-GPU is based on clDNN.
• CPU: Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz, 6 Cores, 12 Logical Processor(s)
• GPU: Intel (R) UHD Graphics 630
• Memory: 32GB

WebNN is still in its early stage. However, we'd like to share the initial results and explore WebNN API design for JS ML frameworks optimization with the ONNX.js community.

Thoughts?