MB-iSTFT-VITS2

A... vits2_pytorch and MB-iSTFT-VITS hybrid... Gods, an abomination! Who created this atrocity?

This is an experimental build. Does not guarantee performance, therefore.

According to shigabeev's experiment, it can now dare claim the word SOTA for its performance (at least for Russian).

pre-requisites

1. Python >= 3.8

2. CUDA

3. Pytorch version 1.13.1 (+cu117)

4. Clone this repository

5. Install python requirements.

   pip install -r requirements.txt
   

   If you want to use the Triton version of Super Monotonic Align, be sure to install triton by

   pip install triton
   

   and set "model": {"monotonic_align": "sma_triton"} in your configuration file.

   1. You may need to install espeak first: apt-get install espeak

   If you want to proceed with those cleaned texts in filelists, you need to install espeak.

   apt-get install espeak
   

6. Prepare datasets & configuration

   1. ex) Download and extract the LJ Speech dataset, then rename or create a link to the dataset folder: ln -s /path/to/LJSpeech-1.1/wavs DUMMY1

   1. wav files (22050Hz Mono, PCM-16)

   2. Prepare text files. One for training^(ex) and one for validation^(ex). Split your dataset to each files. As shown in these examples, the datasets in validation file should be fewer than the training one, while being unique from those of training text.

      • Single speaker^(ex)

      wavfile_path|transcript
      

      • Multi speaker^(ex)

      wavfile_path|speaker_id|transcript
      

   3. Run preprocessing with a cleaner of your interest. You may change the symbols as well.

      • Single speaker

      python preprocess.py --text_index 1 --filelists PATH_TO_train.txt --text_cleaners CLEANER_NAME
      python preprocess.py --text_index 1 --filelists PATH_TO_val.txt --text_cleaners CLEANER_NAME
      

      • Multi speaker

      python preprocess.py --text_index 2 --filelists PATH_TO_train.txt --text_cleaners CLEANER_NAME
      python preprocess.py --text_index 2 --filelists PATH_TO_val.txt --text_cleaners CLEANER_NAME
      

      The resulting cleaned text would be like this(single). ^{ex - multi}

7. (OPTIONAL) Build Monotonic Alignment Search.

   This repo supports supertone-inc's super-monotonic-align for MAS. It removes Cython dependency(v1, v2, triton) and thus you do not have to build it anymore. However, you can still use the original version by the following

   # Cython-version Monotonoic Alignment Search
   cd monotonic_align
   mkdir monotonic_align
   python setup.py build_ext --inplace

   and set "model": {"monotonic_align": "ma"} in your configuration file.

8. Edit configurations based on files and cleaners you used.

Setting json file in configs

Model	How to set up json file in configs	Sample of json file configuration
iSTFT-VITS2	"istft_vits": true, "upsample_rates": [8,8],	istft_vits2_base.json
MB-iSTFT-VITS2	"subbands": 4, "mb_istft_vits": true, "upsample_rates": [4,4],	mb_istft_vits2_base.json
MS-iSTFT-VITS2	"subbands": 4, "ms_istft_vits": true, "upsample_rates": [4,4],	ms_istft_vits2_base.json
Mini-iSTFT-VITS2	"istft_vits": true, "upsample_rates": [8,8], "hidden_channels": 96, "n_layers": 3,	mini_istft_vits2_base.json
Mini-MB-iSTFT-VITS2	"subbands": 4, "mb_istft_vits": true, "upsample_rates": [4,4], "hidden_channels": 96, "n_layers": 3, "upsample_initial_channel": 256,	mini_mb_istft_vits2_base.json

Monotonic Align configuration

1. Super Monotonic Align (JIT_v1)

   "model": {"monotonic_align": "sma_v1"}
   

2. Super Monotonic Align (JIT_v2)

   "model": {"monotonic_align": "sma_v2"}
   

3. Super Monotonic Align (Triton)

   "model": {"monotonic_align": "sma_triton"}
   

4. Monotonic Align (original Cython version)

   "model": {"monotonic_align": "ma"}
   

   (It will still work as the original version of monotonic align if not specified.)

Training Example

# train_ms.py for multi speaker
python train.py -c configs/mb_istft_vits2_base.json -m models/test

Credits

• jaywalnut310/vits
• p0p4k/vits2_pytorch
• MasayaKawamura/MB-iSTFT-VITS
• ORI-Muchim/PolyLangVITS
• tonnetonne814/MB-iSTFT-VITS-44100-Ja
• misakiudon/MB-iSTFT-VITS-multilingual
• supertone-inc/super-monotonic-align