Add support for moonshine ASR models

[bil-ash]

Model description

Please add support for moonshine ASR models. The recent github commit adds support for onnx(python), so I guess porting to js won't take much effort. However, there is no mention about transformers usage.

This model is quite good for in-browser usage scenario since it is quite small and claims to use RAM proportional to length of audio.

Prerequisites

• The model is supported in Transformers (i.e., listed here)
• The model can be exported to ONNX with Optimum (i.e., listed here)

Additional information

No response

Your contribution

None

[evmaki]

Hi, Evan from Useful Sensors/Moonshine here. Just letting you know this is on our radar. We're working on Transformers support right now, and we (internally) have our current ONNX models running in the browser with onnxruntime-web. Shouldn't be too difficult to get support added to transformers.js from there.

[bil-ash]

Hi, Evan from Useful Sensors/Moonshine here. Just letting you know this is on our radar. We're working on Transformers support right now, and we (internally) have our current ONNX models running in the browser with onnxruntime-web. Shouldn't be too difficult to get support added to transformers.js from there.

@evmaki Not related to this issue, but I am also eagerly waiting for the ability to finetune for supporting a new language.

[xenova]

Hi, Evan from Useful Sensors/Moonshine here. Just letting you know this is on our radar. We're working on huggingface/transformers#34474 right now, and we (internally) have our current ONNX models running in the browser with onnxruntime-web. Shouldn't be too difficult to get support added to transformers.js from there.

@evmaki Great to hear! I have been following the ONNX support and it looks like a great start! One issue is that you currently export two versions of the decoder (w/ and w/o PKVs), leading to weight duplication (more of a problem when running in the browser since we load the decoder twice).

We were able to solve this in Optimum by adding an If node to the graph and then choosing which path to take based on whether the past key values are provided. See here for an example. And here is the code used to merge the two decoders.

I was experimenting with your codebase to pass zero-sized tensors as input, but I get gibberish output.

Either way, once we have transformers support, I expect this to be much easier to convert (since the input/output signatures should be similar to whisper).

[xenova]

I have some exciting news: we've got a working version of moonshine-tiny (See PR: #1099) which offers numerous benefits over the original/upstream ONNX implementation:

1. Huge reduction in model size (285MB → 109MB), with zero quality loss:
   Original = 285MB (7+30+120+128) at fp32
   
   Ours = 109MB (31 + 78) at fp32

   This is due to the following optimizations:

   • Deduplicated tied weights (the original model also suffers from this problem, which is why it displays as 46.5M params instead of the correct 27M params)
   • Merging the two decoders (w/ and w/o PKV inputs) into a single graph

2. Multiple quantization support: int8/uint8, q4, fp16, q4f16. We are facing an issue with fp16/q4f16 decoder (cc @guschmue), but all other quantizations work. At q4 quantization, which still works pretty well, the model size is further reduced to 55MB (44.7+10.4)

cc @evmaki - I think these benefits can also be upstreamed, and I'd be happy to make a PR if you'd like! 🤗

[evmaki]

@xenova That's fantastic! Please do open a PR – excited to take a look!

[fidoriel]

Has the merge conversion been changed in the transformers conversion script?

[fidoriel]

Has the merge conversion been changed in the transformers conversion script?

got it working with
python3 -m scripts.convert --quantize --model_id id --task automatic-speech-recognition-with-past

used latest version transformersjs 4362237

huggingface_hub
transformers[torch]==4.52.4
onnxruntime==1.20.1
git+https://github.com/huggingface/optimum.git@85376e337681b1db6aa9752cc4f592b56eedb85e
tqdm==4.67.1
onnxslim==0.1.56