Recommendations for fine-tuning for custom voice

[DesiKeki]

Hello TTS Community,

I am trying to fine tune tts_models--en--ljspeech--tacotron2-DDC_ph for my own voice. I see that documentation https://tts.readthedocs.io/en/latest/finetuning.html gives some instructions to get it working, like using a smaller learning rate and a minimum of 100 audio samples etc.

But I would like to know from the experience of the community what are the recommended do's and don't's for good results. For example, having answers to following questions can be really helpful for newbies like me:

1. Are 100 audio samples sufficient?
2. What should be the appropriate learning rate?
3. What is the recommended length (no of characters) of each audio sample?
4. Should we make any changes to batch size and number of epochs also? (Currently defaults in config file are 64 and 1000 respectively)
5. Any other changes which should be done in the config.json file?
6. Is tts_models--en--ljspeech--tacotron2-DDC_ph a good model to fine tune for custom voice?

These are the doubts before starting the training. And while the training is happening, it continues for a while. So what are the things that should be noted/monitored to ensure that everything is going fine.
Eg in Evaluation performance metrics,.. what is the acceptable average loss or acceptable average align error etc.

Thanks in advance!
Keki

[DesiKeki]

Hello Gurus & Friends,

An update (not a very good one) on my above post is that I tried fine tuning tts_models--en--ljspeech--tacotron2-DDC_ph using the suggested method:

CUDA_VISIBLE_DEVICES="0" python TTS/bin/train_tts.py \
    --config_path  /home/ubuntu/.local/share/tts/tts_models--en--ljspeech--glow-tts/config.json \
    --restore_path  /home/ubuntu/.local/share/tts/tts_models--en--ljspeech--glow-tts/model_file.pth.tar

But the resulting best model did not perform at all. It only generated noise for every text.
I used 100 audio samples as my training data.
Reduced the learning rate to 0.0001
Trained for default 1000 epochs

The training data and the config.json file can be accessed at https://cutt.ly/VOZ0wB5

I'll really appreciate if anyone can please tell what am I missing here?

Thanks
Keki

[DesiKeki]

Thanks!

[DesiKeki]

Thanks a lot @Edresson for helping here. I tried with the suggestion given by you and
The good news is that the fine tuned model resembles my voice now.
The bad news is that even after 77 hours of training its not the productize-able quality.
The speech generated by the model sounds electronic and gets totally distorted towards the end of the sentence.

Also, out of the multiple checkpoints created during the training, how do I know the best checkpoint model? Is there no "final model" created by the script?

All the model checkpoints, config file used and the training logs ("trainer_0_log.txt") are kept at here for your kind perusal.

Can you please have a look and tell me what am I missing now? How can I improve the quality of results here?

[Edresson]

Hi, What branch are you using? And Do you notice the quality improvement during the training (listen to the test set samples)?
After having a satisfactory result from your TTS model you should use a neural vocoder like HiFi-GAN (by default GlowTTS uses the Griffin-Lim vocoder which sounds robotic).

[DesiKeki]

>>What branch are you using?
Its 0.5.0. I am getting it from pip install TTS.

>>And Do you notice the quality improvement during the training (listen to the test set samples)?
I couldn't get this one. Can you please elaborate a bit? How do I listen to the test set samples?
Also, in the training log, I see that at the last epoch, the avg loss was 3.93372. I hope that the checkpoints getting saved are the ones with the best performance overall. I am using the last checkpoint saved.

>>After having a satisfactory result from your TTS model you should use a neural vocoder like HiFi-GAN
I tried this and the result was worse. I tried with following vocoder models:
ljspeech/multiband-melgan (Best performance, but again the voice getting distorted towards the end of the sentence)
ljspeech/univnet (better than hifigan, and close to melgan performance)
ljspeech/hifigan_v2 (noisy and distorted, worse out of the three)

Also in the training log, I see following prints:

�[1m > EVALUATION �[0m
 | > Synthesizing test sentences.
warning: audio amplitude out of range, auto clipped.
warning: audio amplitude out of range, auto clipped.

Can this cause any problem?
@Edresson I feel that I am close to achieving a good QOR, but I am just missing something basic. Can it be one of the following:
Training data
I am using 100 audio samples of average seq length 74.
Should the training data be more? Should each sample be longer?

Learning Rate
I kept the learning rate 1e-05. Should it be changed?

Anything else?

Thanks a lot for your help. Looking forward to making this work.

[Edresson]

@DesiKeki

About the dataset:
It depends on a lot of factors, but probably 100 samples will be not enough to adapt the traditional models to a new voice and keep the quality. When you are training the Tacotron model with few samples probably your model will lost vocabulary and it will turn the pronunciation of news words unnatural. To adapt to a new voice with not so much data I recommend you to use the YourTTS model given that this model can do zero-shot multi-speaker TTS and in the paper, we showed that it is possible to learn a new voice with less than 1 minute of Speech.

About the learning rate:
You are using the Rectified Adam (RAdam) optimizer, so like indicate in the paper the learning rate for this optimizer does not have so much relevance. In addition, 1e-05 looks like a 1e-05 looks like a reasonable value.

[harshvardhan-truefan]

@DesiKeki If you were able to fine-tune your model for a new voice, if possible, can we connect ? I'm facing issues in fine-tuning the model on my own voice, so looking for some guidance.

[DesiKeki]

Hi @harshvardhan-truefan , its long since I left this activity. I could get fine tuning done to a level, but could not pursue it further with more training data. Still I can try to help with whatever knowledge I may have. Please suggest how should we connect? Would you like to share your email id or whatsapp number here?

[charbelgereige]

Is this still the advised way to do it, as described above? or do we have new ways of fine tuning the models to custom voices? I was trying Valle-x, with 1 shot audio, but the output doesn't resemble the input. Any pointers?

[ADiEmme]

hi @DesiKeki, i am trying to perform the same but i am getting a similar outcome (only noise) and was curious to see the config.js you used to do your fine tuning with and the command used as well. You might have noticed i sent a request to get access to your google drive links.

Thanks in advance!

[harshitdeepsolv]

hi can you help me a little i am trying to start the training but getting errors!!