Finetuning Fairseq MMS models

[MedAymenF]

Is it possible to finetune MMS models? when I tried I got a bunch of errors like:

 > Restoring from G_100000.pth ...
 > Restoring Model...
 > Partial model initialization...
 | > Layer missing in the model definition: enc_p.emb.weight
 | > Layer missing in the model definition: enc_p.encoder.attn_layers.0.emb_rel_k
...
 | > 24 / 949 layers are restored.

followed by:

Traceback (most recent call last):
  File "/home/aymen/Tamazight-NLP/Speech/coqui_tts/train.py", line 109, in <module>
    trainer = Trainer(
  File "/home/aymen/miniconda3/envs/coqui_tts/lib/python3.10/site-packages/trainer/trainer.py", line 347, in __init__
    (self.model, self.optimizer, self.scaler, self.restore_step, self.restore_epoch) = self.restore_model(
  File "/home/aymen/miniconda3/envs/coqui_tts/lib/python3.10/site-packages/trainer/trainer.py", line 638, in restore_model
    logger.info(" > Model restored from step %i", checkpoint["step"])
KeyError: 'step'

Which might be because of this function. I'm using Torch 2.5.1 if it's relevant.

[eginhard]

It's probably possible in theory, because ultimate they are just Vits models. But yes, that function might only make the necessary changes for inference. I think we also only download the part of the model that is needed for inference, the others are available in a separate file: https://github.com/facebookresearch/fairseq/tree/main/examples/mms#tts

[MedAymenF]

Only the generator is loaded indeed, I didn't notice that.