Question : Conformer RNNT / Decoder training

[amejri]

Hi again guys,

I have a question. For conformer rnnt model especially the decoder (transducer) part, based on the fact that decoder (predictor or prednet element) doesn't need the audio as input but just transcript text, is there an easy way to train it separately on huge text dataset, and after that charge it into conformer rnnt model.
In other words is there a way to train a transducer separately and charge it in conformer model.

With kind regards,

[titu1994]

I wouldn't call if an easy way, but there is research out there for supporting text only adaptation. We have not implemented it yet but we plan to for the future.

[eesungkim]

I don't think there is easy way to train the RNNT without the encoder part (conformer) only with text due to the nature of the transducer model. Instead, there is a recent study that proposed text adaptation using LM with prediction network of RNNT model.

As far as I know, Nemo does not support LM for conformer RNNT models yet. Hopefully, I also hope that it could be supported soon.

[titu1994]

Yes, we are looking into supporting RNNT adaptation via text only using that paper. It will require a little time before we get all the RNNT features out since it's a relatively more complicated model.

We do plan on that paper for our RNNT adaptation effort btw. Just need to allocate time to setup the code and experiments.

For external LM, @VahidooX was looking into it but there are some higher priority tasks right now and hence that will take a bit of time. Btw, we very much welcome external contributions so if you have the time you could contribute to it.

[VahidooX]

Please check it out here: #3412 (comment)

[zzomg]

@titu1994 @VahidooX Hi! Sorry for re-opening this discussion, I've been looking into domain adaptation of Conformer Transducer models. I noticed you've mentioned that you're planning on implementing what's proposed in this article in NeMo. I was wondering if it has been implemented or there are any other ways of training Conformer Transducer easier that one could use in NeMo at the moment?

[lee-onidas]

@titu1994 Also interested in text only adaption of the RNNT Decoder network. Has anyone managed to implement the method described the paper: https://arxiv.org/pdf/2104.11127.pdf

[ArtyomZemlyak]

@titu1994 Its very interesting approach! Good if this will be in NeMo