Zipformer Onnx FP16

[manickavela29]

Hi @csukuangfj , @yaozengwei
Exporting zipformer onnx model in FP16
Model is trained in mixed precision, therefore having fp16 shouldn't give any loss

Tested with data, and model accuracy is exactly same as fp32.
cc: k2-fsa/sherpa-onnx#41 , k2-fsa/sherpa-onnx#40

[csukuangfj]

Could you describe how you test it? Does it work on CPU.

Also, could you use fp16.onnx as the suffix when fp16 is used?

[manickavela29]

I have tested with A10 GPU and it is holding good in sherpa-onnx

I havn't tested this on CPU, but AVX512 has support for sure, I think AVX2 will also handle it.

Generally onnxrt will have fall back mechanism if fp16 inherently not there in fp32 conversion happen implicitly
but I have not tested it, and it is also subjected to Ops availability(should not be an issue with zipformer)

This is an optional export flag, and wanted to keep it simple with the existing model itself.
thats the reason didn't prefer having a new model with '-fp16' separately. It is also backward compatible 🙂

But if this is still required, I will modify the filename still.

FYI,
I have only tested with Zipformer encoder model, so raised this patch.
later if I can time to experiment with decoder/joiner too I will just extend the support

[csukuangfj]

Could you test the fp16 model either in icefall with onnx_pretrained.py or in sherpa-onnx with CPU?

thats the reason didn't prefer having a new model with '-fp16' separately. It is also backward compatible

Currently, we have .onnx for fp32, int8.onnx for int8. Using fp16.onnx makes it clearer to users the model they use is fp16.

[manickavela29]

I ran it once with sherpa, and it was up and running,
some minor warnings from onnxrt optimizaiton but it is stable

4/Exp'
2024-06-27 05:57:13.449610676 [W:onnxruntime:, constant_folding.cc:269 ApplyImpl] Could not find a CPU kernel and hence can't constant fold Exp node '/norm_15/Exp'
2024-06-27 05:57:13.449640846 [W:onnxruntime:, constant_folding.cc:269 ApplyImpl] Could not find a CPU kernel and hence can't constant fold Softmax node '/downsample_output/Softmax'

[csukuangfj]

Thanks! Is it ready to merge?

[manickavela29]

yes, completed from my end!

[csukuangfj]

Thank you for your contribution!