WebNN should support int8 quantized models

[wchao1115]

Supporting int8 quantized models is essential for mobile scenarios and in many NPU architectures. TensorFlow (Lite) and ONNX, for instances, have int8 quantization support built-in, and WebNN should to. Related #93

[anssiko]

@wchao1115 @huningxin do you think we should label this as "cr" for #240 purposes?

[huningxin]

I think this is important one and support to label as "cr".

[anssiko]

@wchao1115 this issue was on the agenda today, but we had to defer due to timing. Let us know your thoughts. I'm planning to bring this up for our next meeting for discussion.

[anssiko]

Per discussion at https://www.w3.org/2022/03/24-webmachinelearning-minutes.html#t06 we consider this to be in scope for CR.

[anssiko]

We've discussed this feature on our recent meetings:
https://www.w3.org/2022/09/22-webmachinelearning-minutes.html#t05
https://www.w3.org/2022/09/08-webmachinelearning-minutes.html#t05
https://www.w3.org/2022/08/25-webmachinelearning-minutes.html#t06

I will label this issue as "v2" due to required implementation experience for the initial CR inclusion. There's a mechanism for us to publish a Candidate Recommendation Draft subsequent to the initial CR that would give us adequate time to properly define, develop and test this feature.

Furthermore, we should soon start discussing WebNN "v2" plan as we look to extend our current charter and this feature could be one concrete feature to highlight. We can continue discuss this feature on our bi-weekly calls when there's new information and revise our position as appropriate.

[inexorabletash]

It looks like this was added to the spec in 0970115 and we may have some implementation experience at this point. Close, despite it being marked v2 ?

[huningxin]

The int8 quantized models may need some extra ops, for example DynamicQuantizeLinear, DequantizeLinear, ConvInteger and MatMulInteger, that are missed in current spec.

Transformer Models Analysis spread sheet has more details of ops required by int8 quantized model (see columns marked with (int8)).

@fdwr @Honry

[fdwr]

The int8 quantized models may need some extra ops, for example DynamicQuantizeLinear, DequantizeLinear, ConvInteger and MatMulInteger, that are missed in current spec.

Indeed, I have those 4 prototyped here (a minimal first four): https://github.com/fdwr/chromium-src-webnn-dml/pull/1/files#diff-e1b2517a6ae8f7c4494c75d17c8650b56e4f8d430f54f5e1f765475f00a5e1f3R427-R433

[wacky6]

Seems int4 quantization is also a thing (with negligible impact on output quality). int4 practically halfs the VRAM requirement of the model, and offers a speedup on devices that support them.

Example of a int4 quantization model: https://huggingface.co/01-ai/Yi-6B-Chat-4bits

Should this be considered for v2? Or is int4 too specific? (I'm not sure if 4bit is adequate for image or audio models)

// There's a more aggressive {-1,0,1} quantization. It's fairly new, and I believe it's application is limited to language models.

[inexorabletash]

The BitNet paper was really cool. https://arxiv.org/abs/2310.11453

[inexorabletash]

The int8 quantized models may need some extra ops, for example DynamicQuantizeLinear, DequantizeLinear, ConvInteger and MatMulInteger, that are missed in current spec.

Indeed, I have those 4 prototyped here (a minimal first four): https://github.com/fdwr/chromium-src-webnn-dml/pull/1/files#diff-e1b2517a6ae8f7c4494c75d17c8650b56e4f8d430f54f5e1f765475f00a5e1f3R427-R433

Hey @fdwr - how fresh is your prototype of these? And have you looked at how other backends (CoreML, TFLite) would implement these? Starting with the "minimum viable" quantization support as outlined in #623 is appealing!

[fdwr]

@inexorabletash

how fresh is your prototype

It's moldy bread by now (but snippets could be reused). The ORT WebNN EP implementation still exists (it was originally added during prototyping) and would light up again once the op is added into Chromium.

And have you looked at how other backends (CoreML, TFLite)

There are differences, but they should be expressible (🤞). For dequantization, most decompose to output = mul(sub(input, zeroPoint), scale) (except TF full, CoreML MIL's LUT mode, and CoreML's scale&bias form). They have differing broadcasting rules, which I'd like to iron out to be more consistent (consistent with unidirectional broadcasting of its decomposition and expand).

API Name	Equation	Types
TFLite DequantizeOp	real = (quantized - zeroPoint) * scale (link)	input: uint4, uint8, int8, int16, float16 zeroPoint: uint8 scale: float32 output: float32
TF tf.quantization.dequantize	output = minRange + (input * (maxRange - minRange) / dataTypeRange)	input: uint8 minRange: float32 maxRange: float32 dataTypeRange: int output: float32
CoreML MIL constexpr_affine_dequantize	real = (input - zeroPoint) * scale	input: uint8, int8 zeroPoint: uint8, int8, float32 scale: same as output output: float16, float32
CoreML MIL constexpr_lut_to_dense	real = lut[input]	input: uint1, uint2, uint4, uint6, uint8 output: uint8, int8, float16, float32
CoreML LinearQuantizationParams	? input * scale + bias ?	input: ? scale: float32 bias: float32 output: ?
ONNX DequantizeLinear	real = (input - zeroPoint) * scale	input: uint4, int4, uint8, int8, uint16, int16, int32, float8e4m3fn, float8e4m3fnuz, float8e5m2, float8e5m2fnuz zeroPoint: same as input scale: same as output output: bfloat16, float16, float32
DML DEQUANTIZE_LINEAR	real = (input - zeroPoint) * scale	input: uint4, int4, uint8, int8, uint16, int16, uint32, int32 zeroPoint: same as input scale: same as output output: float16, float32

[reillyeon]

Discussed at the TPAC 2024 F2F. Group consensus was to implement QDQ operators for int8 and int4. Deduplicating with #93.

[reillyeon]

Closing this issue. Discussion of quantization operators should continue on #93.