Long-form audio transcription with Citrinets

[titu1994]

Long form audio transcription

Long-form audio transcription in the simplest sense is when an ASR model is evaluated on audio clips that are significantly longer in duration as compared to what the model was trained on. For example: Most models are trained on audio clips of length 15-20 seconds, but can possibly be evaluated on audio clips that are around 1 minute long.

There are many variations to long form audio transcription research - whether the model is used in streaming mode or not, the difference in duration between training and evaluation, whether the model was trained with specific losses to enable long form audio transcription etc.

This notebook demonstrates how to transcribe a relatively long audio file from a podcast in a single forward pass with a Citrinet model. The model operates in offline mode (is given entire audio sequence at once) and must transcribe it without streaming inference.

To make things more realistic, we chose a podcast where the discussion revolves around a technical topic, a domain that the model was never trained on. We preprocess the podcast to be in a format similar that can easily be compared to the model's transcription. Finally we also run some checks to see what's the longest audio segment that can be transcribed in a single forward pass.

---

Colab link to Notebook

[ShantanuNair]

Hey, @titu1994 really appreciate the notebook!! These notebooks are invaluable for those of us focusing more on long-form audio. I'm curious what considerations need to be taken when working with Citrinet vs the streaming conformer script added in v1.2.0 with respect to compromises in accuracy, given that Citrinet operates in offline vs the streaming conformer script.

[titu1994]

Streaming mode will generally reduce WER slightly, since only limited context is available for the model to see ahead. Citrinet doesn't need such script (unless you are transcribing a movie or two in one go) but Conformer requires it.

Thankfully Confirmer has self attention so what little context it does get it uses very well, so it's streaming wer with a large enough buffer size (say 5-10 seconds) will do just fine and get good results (around 0.5-1% worse WER with buffer sizes of 10 seconds).

[ShantanuNair]

@titu1994 Appreciate all the support from y'all :)

Streaming mode will generally reduce WER slightly,
Guessing you meant increase the WER here?

One more thing relating to long-form audio though — and this is for the Riva/Jarvis team really, but I don't have access to anyone from Riva so here it is - How do we do long form audio STT in production? I ask because it looks like Riva's gRPC server doesn't accept messages larger than 4MiB. Using the riva_asr_client with the citrinet offline model, I get this gRPC error:

Loading eval dataset...
filename: /work/wav/working_files/craig.wav
Done loading 1 files
RPC failed: Received message larger than max (150952067 vs. 4194304)

My file is 140MiB. The gRPC 4MiB limit is a real no-go for us and I have to work around it somehow. I guess I could chunk it into 4MiB chunks but I was worried that that would compromise Citrinet's ability to keep attention properly.

[titu1994]

For any Riva related question please refer to their dev forums - #2841

[ShantanuNair]

I understand, thanks. I went ahead and created a post on there. I've had lesser luck with the forum which is why I posted here.

For anyone else following, this is the thread I created.