Timestamps with Canary transcripts

[GreatHalf]

Hello,
it appears that all compute_timestamps options are ignored when using Canary, e.g. with examples/asr/speech_multitask/speech_to_text_aed_chunked_infer.py.

Furthermore, you cannot use preserve alignments, although these might be used to help Nemo Forced Aligner (as I stated here). The warning when setting this flag is [NeMo I 2024-04-12 15:46:02 multitask_beam_decoding:159] Preservation of alignments was requested but TransformerAEDBeamInfer does not implement it.

As far as I can tell, the only way to get Canary results with timestamps is:

1. transcribe with Canary to get the text
2. run NFA with the text (and recalculate fitting alignments)

Is this correct?
Is this due to the aed approach used by Canary? Can we expect timestamp support in the future? Whisper eventually supported them.

Cheers

[GreatHalf]

I found the answer here https://huggingface.co/nvidia/canary-1b/discussions/16. Everything appears to be as I suspected.

I would like to keep this open in case of new development that happened in the meantime.

[titu1994]

NFA combining Canary Transcripts with Parakeet CTC aligner is going to give much more accurate alignments than using greedy alignment algorithm that we have for CTC RNNT. We will add some documentation for it. Fyi @erastorgueva-nv

[GreatHalf]

Parakeet only has English models available, right?
So it's not an out-of-the-box option for the non-English transcripts of Canary.

[titu1994]

You can use any CTC or hybrid model for alignment, we have multiple models for each language that canary supports

[kunibald413]

@titu1994 Hi, thank you for your suggestion of using this aligner.

What is the difference between Parakeet CTC aligner vs these stt_en_fastconformer_hybrid_large_pc aligners? why would the parakeet aligner be better? (https://github.com/NVIDIA/NeMo/tree/main/tools/nemo_forced_aligner)

Thank you for your time!

[titu1994]

CTC is an alignment loss, it's task is to learn a mapping from speech to English, and it's a monotonic loss which also usually does better when learning this mapping.

Attention can learn it too, but not quite as well, and requires aids like Monotonic chunkwise attention (MoChA)