Define the set of operations and their specification

[huningxin]

As raised in CG meeting, the first foundation spec only lists 32 operation types without information about how to use them.

We need to define the set of operations and their specification/semantics.

The set of operations could be derived from the use cases and corresponding models. For WebNN POC, there is a spreadsheet that lists supported models and their required operations. It can be used as a starting point.

By following the spirit of WebML CG charter, the specification will be implementable on top of existing major platform APIs, such as Android NNAPI, Windows DirectML, and macOS/iOS MPS/BNNS. So when specifying operations, the platform APIs mapping/support need to be looked into. For WebNN POC, there is another spreadsheet the captures the native API mapping of supported operations. It can also be leveraged.

We can file individual issue for each operation specification and use this one as the meta issue.

[gregwhitworth]

Thanks for bringing this issue up @huningxin – this is something we’ve been pondering as well. We would like to propose that this CG not duplicate the efforts being done by the open ONNX community. This community has produced an MIT licensed operator specification based on years of investigations of the top ML models to standardize around. The community brings together platform, software, web and hardware companies to help provide a standardized approach which allows any team across the stack leverage the same set of operators and make development in ML more efficient for everyone involved. A few companies that are involved in this open community are Amazon, Facebook, Microsoft, Intel, IBM, Unity, Nvidia, Qualcomm, AMD and more.

We recommend that we follow the same pattern done by other specifications within the W3C & other standard organizations by referencing the work done by others rather than duplicating efforts and creating numerous standards. A few examples of this are:

• CSS Text references the Unicode specification
• Service Workers does not redefine what IETF has defined for the fetch API
• All specifications that reference floating points (probably including WebNN at some point) they point to IEEE

Additionally, this is an open community so if there are any changes we desire to this specification we can make them (and they also welcome them). Now, this doesn’t mean that this API needs to support ALL of the operators listed by default, we can create a list of operations from within that specification that must be supported by an UA in order to fully support the WebNN API, but we shouldn’t redefine the operators and their signatures, inputs & outputs.

[gregwhitworth]

PROPOSED RESOLUTION: The specification will reference the ONNX operations and if there are any improvements desired for ONNX the work should be there.

We will be resolving on this on the next telecon, let us know if you want any objections to this.

[anssiko]

As discussed on today's call, please provide your input in this issue for the proposed resolution documented below prior to our next call 8 August 2019. During that call we'll resolve this issue as proposed unless objections are recorded in this issue.

PROPOSED RESOLUTION: The specification will reference the ONNX operations and if there are any improvements desired for ONNX the work should be there.

(That's 6 weeks review time to provide feedback due to holiday season in some parts of the world.)

[nsthorat]

An important part of this specification will be ensuring this set of ops are compatible with the major ML JavaScript frameworks (e.g. conv2d paddings are compatible, memory layout of tensors are compatible, etc). It's not possible for us to move forward with this resolution without understanding compatibility.

Would somebody be willing to take on the work of detailing each op and whether it could be used by major frameworks in a doc that can be shared with the group?

Thanks!

[jbingham]

A related question: what's the plan for dealing with versioning? I'd expect ONNX to evolve and add ops over time. Also, the ops would be versioned, and details may change over time. Is the idea to reference an explicit list of ONNX ops with versions, and include them by reference in the web standard?

If yes, then the next step would be to agree on which specific ops + versions to reference, ensuring that they're compatible with the major ML JavaScript frameworks we want to support, as well as the backend providers. Maybe the answer is easy, and it's all of the ONNX ops, in their latest versions. I agree that we (the community group) need to do the homework to determine if that's the case.

[RafaelCintron]

@nsthorat and @jbingham, below are links to existing documents and code that should help answer your questions.

• Versioning. Up to us, as a community group, which version of the operator set we use for the web spec.

On the compatibility front, there already exists conversion tools to and from other frameworks and model types.

• ONNXMLTools. ONNXMLTools enables conversion of models to ONNX. Currently supports Keras, CoreML, LightGBM and Scikit-Learn
• TensorFlow to ONNX

[jbingham]

Thanks, @RafaelCintron ! We'll take a closer look.

I see the compatibility matrix. Great! It would be good for us to understand what compatibility really means here.

Some questions that you may already have great answers for:

Versioning:
I see that not all TF ops are convertible to a consistent ONNX version. Eg, most convert to 1, some convert to 1 or 6, others only convert to 7 or 10. And then there's the compatibility matrix for Keras, CoreML, LightGBM, and Scikit-Learn. How would we choose op versions that work for all of these libraries? Is it tractable?

Converting to ONNX from js:
Is the idea that, for each javascript ML framework (like TensorFlow.js), there would be some client-side js to convert from the framework's native graph representation (like a SavedModell) to a WebML graph with ONNX representations of all operators, and then the WebML API would operate on that?

Custom ops:
What if a desired operator isn't available? How are custom ops defined and included in the graph?

How many ops?
Do we need all of the operators in the spec? Or would we get 80% of the performance benefit with 2 or 3 ops, like matmul and conv2d? Does that simplify things at all, as a starting point to build consensus?

Thanks in advance for explaining!

[gramalingam]

@jbingham : re. your versioning question: ONNX has the notion of ops and opsets. An opset consists of a collection of ops with a given specification. Typically an op has the same specification in a number of consecutive opsets until it is updated. Thus, if we consider the Equal op, there is a version introduced in opset 1, which had the same spec in opsets 1 through 6, and was updated in opset 7 (to add broadcasting support).

I think the notation used in the compatibility matrix reflects this. It lists versions 1 and 7 for Equal op because opset versions 1 through 6 have the same spec for Equal, and version 7 onward have the same spec (until now, that is the current opset).

[walrusmcd]

To also add some more history and context for @jbingham questions:

Versioning:
some history: The ONNX journey was all about interchange from all frameworks. Our goal was to support training in one framework, and exporting to do inference in another framework. So our first opsets where optimized for inference and a set of canonical models + training frameworks. The first toolkit (here) focused on all the ones you mention (CoreML, LigthGBM, Keras, and more) . It was a solid opset. We learned a ton working with all the different behaviors of the frameworks, including key differences like how broadcasting works. We iterated rapidly, and got to production quality by opset7. This was June 2018. It is very tractable. We have deployed to client devices on a very large scale with a vibrant cross framework converter ecosystem. Microsoft even shipped an ONNX ML runtime in the box with a public developer API starting in Windows 1809 RTM :)

For tensorflow the journey has been a bit longer. Since ONNX adopted the LCD set of ops needed for inference, TF had a much larger set of ops (there are 500+ in the tensorflow::ops C++ namespace), contrib ops, experimentation variations, training, etc.. Just like TOCO went through its journey to convert from tf to tflite, we went through a similar journey with tf2onnx. LSTM is a great example of another issue we hit. Keras has LSTM primitives, where TF is more loosely structured (we solved the LSTM with our converters btw!).

Regarding the compatibility matrix link you posted , @gramalingam response is spot on. All of the opsets roll forward and include the previous versions. You target a single opset. Opset 7 is where we supported most tf models. And we just keep getting better everyday. We focus on the data science community and the tf operators they need most for their production deployed models. Examples are like in opset9 when we added NonMaxSuppression. If your model absolutely required NMS, you could use opset7 plus a custom operator, or roll forward to opset9 and the model would just work. Again very tractable.

Using ONNX in js:
The first idea would be to enable the browser platforms to include ML execution providers. Basically a provider knows how to run a kernel, and manage resources (memory). JavaScript ML frameworks could then build on top of these providers . It’s TBD what the provider IDL would look like. We have the other github issues tracking this one (graphs &| op kernels).

This particular github issue would track the idea that the op schema, namespaces, versioning, semantic definitions, could all be driven from ONNX. The idea being no matter which IDL we land with, we still need a common currency how to describe op kernels, schema, and behavior.

Custom ops:
I would imagine this we cover in the IDL conversation for the execution provider. From an ONNX point of view, these custom ops do not need to have governance by the standardized schema, right?

How many ops?
This is a great conversation !! Would love to dig in. We found a couple of things:

• Having too many was hard to get cross framework consensus on (ONNX landed on ~100 vs. the 500+ tf had).
• having too few means we often hit operators that we really needed in a model. Custom ops become the norm, vs. the escape valve.
• having just a couple is really BLAS/MLAS. Already covered, right? No need for an new ML operator set ;)
• Crossing resource boundaries is expensive. For our DirectML implementations, we strive to have 100% coverage inside our GPU execution provider to maximize perf.

I would imagine for WebML we would want the same. Execution providers that provide enough ops to run most models without leaving the provider resource boundary. Right? What training frameworks are you thinking about? With ONNX opset7 we were able to run almost all of the models we had targeted for (tf, pytorch, scikit-learn, LightGBM, CoreML). That was our goal, to be able to fully represent the models across ML frameworks.

I'm not sure what the right answer is , but I like your idea of starting with a straw man. What if we took opset10 (the latest opset) as a start to reason around that ?

@nsthorat totally agree, compatibility is a huge deal.

ONNX has 2 things it drives here:

1. This is the master operator schema. It is driven by def files (code) so the doc should not drift out of date. It defines things like Conv and exactly states what the shapes of tensors are, inputs , outputs, behavior defining attributes, and type constraints.
2. They also provide conformance tests. This is also code, so there can be little ambiguity. The code all lives here , and is based largely on numpy to provide a reference implementation. We then target a set of canonical models (alexnet, inception, squeezenet, vgg)

I image the community would totally jump in and help fill any testing and documentation gaps as we find them. the entire goal here is to enable framework developers to work with it. We have a couple of frameworks supporting ONNX natively (ORT, Caffe2, pytorch, CNTK) and would love to keep adding more !

[jbingham]

@gramalingam, thanks for clarifying. So if we wanted to cover a large number of common operators across many real-world models in multiple frameworks, we'd want to select a high enough op set version, such as opset10. That makes sense.

@walrusmcd Lots of great info there. We will likely want to go deeper on some of these topics, possibly in the other github issues you mention, or new ones.

Here's a summary of some of what you describe: imagine a future web platform that lets you train an ML model in any ML framework, and as long as you can convert it to a compatible graph representation, with a standardized set of ops, you can perform inference on any device, taking advantage of hardware acceleration, from any web app, running in any web browser.

While I'm digesting, I have another thought, which I'll riff on in a separate comment.

[jbingham]

Here's that other thought:

Given how complex this all is, and how many ops and ML frameworks, is it realistic to expect that there will ever be more than one code base that fully implements ONNX, much less keeps up with the evolution of opsets? From what I see on github, well over a hundred people -- and not just random people, but knowledgeable, highly technical people with deep domain knowledge -- have contributed code to get ONNX to where it is today. That would be difficult to reproduce, especially in a compatible way.

And if there is likely to only ever be one ONNX implementation, does that mean that to achieve standardization across web browsers, ML frameworks, and opsets, the only viable path, realistically, is for the browser vendors to agree to ship the exact same binary? This has been done before, so it's not unprecedented or impossible. But it would take some real effort to make happen, assuming the web should have a standard at this level of abstraction, as opposed to a few low level shaders, which would be less controversial.

Putting it all together would the decision to standardize on ONNX opsets mean that the web standards community would be effectively agreeing to adopt the first and, in practice, only implementation of ONNX, and the browser vendors would be agreeing to ship it?

If that's what's at stake, webnn issue 17 seems kind of important. We might want to put together a detailed case for why this is really where the community should go, and why the ML frameworks and browser makers should all be onboard.

It's also possible that I just took an accidental turn down a rabbit hole, in which case, please help :)

[huningxin]

Hi @jbingham ,

Custom ops:
What if a desired operator isn't available? How are custom ops defined and included in the graph?

The issue #6 is for custom ops discussion. So far, the proposal is that JS ML frameworks offload the supported sub-graph execution to WebNN and execute own (custom) WASM/WebGL/GPU kernels for ops that are not supported. It would require WebNN supports high performance data/tensor exchange between WebNN execution and WASM/WebGL/GPU kernel execution.

There are two early prototypes for TensorFlow.js/WebNN integration and ONNX.js/WebNN integration based on this proposal and WebNN POC.

How many ops?
Do we need all of the operators in the spec?

It would depend on what models WebNN would support and the models are according to use cases. (I'd like to provide some inputs in the next comment.)

Or would we get 80% of the performance benefit with 2 or 3 ops, like matmul and conv2d? Does that simplify things at all, as a starting point to build consensus?

As our investigation of custom ops, we have two key observations:

• Offloading expensive JS framework ops to WebNN can get good speedup
  In our experiment for MobileNet, offloading Convolution (Conv2D + DepthwiseConv2D) + Relu6 to WebNN (MKL-DNN backend) can get 30X speedup to WASM and 80% of native performance.
• Interop between JS framework ops and WebNN graph has overhead
  When offloading MobileNet to WebNN, per op execution is 3.5X slower than whole graph execution. So it may also require to offloading the less expensive ops (like Relu, concat, add etc.,) to avoid interop overhead.

[huningxin]

For the starting op set, there are some inputs from use case perspective. The idea is to derive the op set by models that support defined use cases.

This table maps the WebNN use cases to related networks.

Use case	Network
Image classification	MobileNet, SqueezeNet, ResNet, Inception
Object/Person detection	TinyYOLO, SSD
Semantic Segmentation	DeepLab
Skeleton Detection	PoseNet

There are pre-trained models in TFLite or ONNX for above networks. This table lists the required ops of each model.

ONNX Op	MobileNetV2 (ONNX)	MobileNetV2 (TFLite)	SqueezeNet1.1 (ONNX) [4]	SqueezeNet (TFLite)	ResNet50V2 (ONNX)	InceptionV4 (TFLite)	TinyYOLOV2 (ONNX) [6]	SSD MobileNetV1 (TFLite) [7]	PoseNet (TFLite)	DeepLabV3 (TFLite)
Add	✔️	✔️			✔️					✔️
AveragePool	✔️ [1]	✔️	✔️	✔️	✔️	✔️				✔️
BatchNormalization	✔️				✔️		✔️
Clip		✔️ [2]						✔️ [2]	✔️ [2]	✔️ [2]
Concat			✔️	✔️	✔️	✔️		✔️	✔️	✔️
Conv	✔️	✔️ [3]	✔️	✔️	✔️	✔️	✔️	✔️ [3]	✔️ [3]	✔️ [3]
Gemm					✔️	✔️ [5]
LeakyRelu							✔️
MaxPool			✔️	✔️	✔️	✔️	✔️
Relu	✔️		✔️	✔️	✔️	✔️
Reshape	✔️	✔️	✔️	✔️	✔️			✔️
Resize										✔️ [8]
Sigmoid								✔️
Softmax	✔️	✔️	✔️	✔️	✔️	✔️

Notes:

1. GlobalAveragePool op of ONNX
2. RELU6 op of TFLite
3. CONV_2D and DEPTHWISE_CONV_2D op of TFLite
4. Ignore Dropout op of ONNX
5. FULLY_CONNECTED op of TFLite
6. Ignore ImageScalar op of ONNX
7. Ignore TFLite_Detection_PostProcess op of TFLite
8. RESIZE_BILINEAR op of TFLite

The following table maps the TFLite op in above TFLite models to ONNX op.

TFLite Op	ONNX Op
ADD	Add
AVERAGE_POOL_2D	AveragePool
CONCATENATION	Concat
CONV_2D	Conv
DEPTHWISE_CONV_2D	Conv
FULLY_CONNECTED	Gemm
LEAKY_RELU	LeakyRelu
LOGISTIC	Sigmoid
MAX_POOL_2D	MaxPool
RELU	Relu
RELU6	Clip
RESHAPE	Reshape
RESIZE_BILINEAR	Resize
SOFTMAX	Softmax

[jbingham]

Great info, Ningxin. This is a ton of great work digging into these details.

Summarizing on a bunch of the comments above:

• We could start with just a couple of ops that give a big performance boost, and we would see real performance benefits
• BUT there are big costs to moving memory around and switching contexts, which argues for moving all ops in the graph to a single execution provider, ideally, even if many of the ops by themselves aren't accelerated
• There's separate discussion around how to make custom ops performant.