VoiceFilter realization problem

[va-volokhov]

Seungwon, hello.

My name is Vladimir. I am a researcher at Speech Technology Center, Russia, St. Petersburg. Your implementation of the VoiceFilter algorithm (https://github.com/mindslab-ai/voicefilter) is very interesting to me and my colleagues. Unfortunately, we could not get the SDR metric dynamics like yours, using your code with the standard settings in the default.yaml file. SDR converged to 4.5 dB after 200k iterations (see figure below), but not to 10 dB after 65k as in your results. Could you tell us your training settings, as well as the neural network architecture that you used to get your result?

Our python environment:

1. tqdm (ver. 4.32.1);
2. numpy (ver. 1.16.3);
3. torch (ver. 1.1.0);
4. pyyaml (ver. 5.1);
5. librosa (ver. 0.6.3);
6. mir_eval (ver. 0.5);
7. matplotlib (ver. 3.1.0);
8. tensorboardX (ver. 1.7);
9. ffmpeg (ver. 4.1.3);
10. ffmpeg_normalize (1.14.0);
11. python (ver. 3.6).

We use four Nvidia GeForce GTX 1080 Ti when training one VoiceFilter's model. Subsets train-clean-100, train-clean-360 and train-other-500 from LibriSpeech dataset are used to train VoiceFilter's model and dev-clean is used to test VoiceFilter's model. We use the pretrained d-vector model to encode the target speaker.

We used your default configuration file:

audio:
  n_fft: 1200
  num_freq: 601
  sample_rate: 16000
  hop_length: 160
  win_length: 400
  min_level_db: -100.0
  ref_level_db: 20.0
  preemphasis: 0.97
  power: 0.30

model:
  lstm_dim: 400
  fc1_dim: 600
  fc2_dim: 601

data:
  train_dir: 'path/to/train/data'
  test_dir: 'path/to/test/data'
  audio_len: 3.0

form:
  input: '*-norm.wav'
  dvec: '*-dvec.txt' 
  target:
    wav: '*-target.wav'
    mag: '*-target.pt'
  mixed:
    wav: '*-mixed.wav'
    mag: '*-mixed.pt'

train:
  batch_size: 8
  num_workers: 16
  optimizer: 'adam'
  adam: 0.001
  adabound:
    initial: 0.001
    final: 0.05
  summary_interval: 1
  checkpoint_interval: 1000

log:
  chkpt_dir: 'chkpt'
  log_dir: 'logs'

embedder:
  num_mels: 40
  n_fft: 512
  emb_dim: 256
  lstm_hidden: 768
  lstm_layers: 3
  window: 80
  stride: 40

The neural network architecture was standard and followed your implementation.

[seungwonpark]

Actually, the resulting SDR strongly depends on the test data that we use.
Try to run inference.py with audio samples shown in http://swpark.me/voicefilter/, and compare the resulting quality with mine by listening to them.

[seungwonpark]

I used batch_size=12, so this won't affect much.

[thejungwon]

Hi @va-volokhov , I also faced a similar situation.
The difference was made from train-other-500. If I used them, I also got a similar score as you did (SDR: 4~5), but if I did not include train-other-500 (train-clean-100, train-clean-360 and other smaller sets are included), I could get around 10 dB as well. But it doesn't mean that you shouldn't include them (more data will be the better result).

The chosen test dataset caused this difference. As you can see from @seungwonpark 's evaluation code, there is only one validation. Therefore if your test data were selected from the different folder, you would get a different result.

However, It will be still difficult to achieve a similar score from the paper. In the original paper, the median and mean SDR is already high even before the separation, which means when it mixes two audio files, interference audio is not fully interfering the all clean audio (shorter length maybe?). Therefore, some of the rear parts of mixed audio may be just clean audio.

Therefore, the easiest way to compare your performance is, as @seungwonpark said, comparing to the published sample. And the original paper's published sample as well (https://google.github.io/speaker-id/publications/VoiceFilter/).

By the way, thank you for the excellent implementation @seungwonpark, including all the preprocessing.

[va-volokhov]

Hi @thejungwon, thank you for your answer. Yes, after excluding the train-other-500 subset from training and testing on data from dev-clean, the SDR behavior becomes similar to that of @seungwonpark. Thank you for the help @thejungwon! Thank you for the excellent implementation @seungwonpark!

[seungwonpark]

Hi @thejungwon, thanks for pointing out that excluding train-other-500 is helpful for training! I will make a commit that excludes train-other-500 from generator.py.

I would also like to thank @va-volokhov for kindly sharing this issue here.

[hdubey]

@va-volokhov @seungwonpark Can you please suggest how to use 2 GPU to train using this code. Using single GPU is too slow. Is there any degradation in performance when we move from 1 to 2 GPU. @va-volokhov Seems you have used even more GPUs. Do you mind sharing the code snippets that can allow us to use this code on 2 GPUs? Thanks guys!

[bigcash]

Hi @thejungwon, thank you for your answer. Yes, after excluding the train-other-500 subset from training and testing on data from dev-clean, the SDR behavior becomes similar to that of @seungwonpark. Thank you for the help @thejungwon! Thank you for the excellent implementation @seungwonpark!

do you mean this?
training: train-clean-360; train-clean-100
testing: test-clean
skip files: train-other-500; dev-clean; dev-other; test-other
and i see author use train-clean-360 and train-clean-100 on training, dev-clean on testing, in the latest code. unfortunately, i trained a bad SDR
hope your reply!

[o74589055]

Hi @thejungwon, thank you for your answer. Yes, after excluding the train-other-500 subset from training and testing on data from dev-clean, the SDR behavior becomes similar to that of @seungwonpark. Thank you for the help @thejungwon! Thank you for the excellent implementation @seungwonpark!

do you mean this? training: train-clean-360; train-clean-100 testing: test-clean skip files: train-other-500; dev-clean; dev-other; test-other and i see author use train-clean-360 and train-clean-100 on training, dev-clean on testing, in the latest code. unfortunately, i trained a bad SDR hope your reply!

I have encounter the same problem, do you have a way to solve it?