Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Modified conformer with multi datasets #312

Merged
merged 23 commits into from
Apr 29, 2022
Merged

Modified conformer with multi datasets #312

merged 23 commits into from
Apr 29, 2022

Conversation

csukuangfj
Copy link
Collaborator

@csukuangfj csukuangfj commented Apr 12, 2022 •
edited
Loading

Use changes from

The training command is

export CUDA_VISIBLE_DEVICES="0,1,2,3,4,5,6,7"

./pruned_transducer_stateless3/train.py \
  --world-size 8 \
  --num-epochs 20 \
  --start-epoch 0 \
  --full-libri 1 \
  --exp-dir pruned_transducer_stateless3/exp \
  --max-duration 300 \
  --use-fp16 1 \
  --lr-epochs 4 \
  --giga-prob 0.5

Note that the GigaSpeech dataset is not perturbed by speed and it is using on-the-fly feature extraction during training.
The XL subset (10k hours) together with the full LibriSpeech are used during training, with a batch being sampled with prob 0.5 from each subset.

Also, note that it does not support resuming training from a checkpoint saved in a sub-epoch.

EDITED: The tensorboard log is https://tensorboard.dev/experiment/lVZFGwjKS9iMYHzrsAE2cw/

@danpovey
Copy link
Collaborator

I believe you were having a problem with this being slow due to on-the-fly features from Gigaspeech? Has that been resolved?

@danpovey
Copy link
Collaborator

I see that running this setup leads to commands like:

ffmpeg -threads 1 -ss 1803.05 -t 1.65 -i /ceph-fj/fangjun/open-source-2/icefall-pruned-multi-datasets/egs/librispeech/ASR/download/GigaSpeech/audio/podcast/P0032/POD0000003189.opus -ar 16000 -f f32le -threads 1

running, but due to the -ss and -t options to ffmpeg, it does have to decode the entire file. I see the GPU-util is not great. Sometimes, all but one machine is at 100% and one machine will be at 0% for 10 seconds or so (confusing)... sometimes they are all around 20%.

@danpovey danpovey mentioned this pull request Apr 12, 2022
Open
@pzelasko
Copy link
Collaborator

I don't believe that it has to decode the whole file when you specify offset and duration in ffmpeg; I was playing around with this in the past and got a great speedup by introducing these options.

@danpovey
Copy link
Collaborator

Yes, I tested the speed, it's fast.. sorry, my comment was a typo, I meant "doesn't" not "does".

csukuangfj
csukuangfj commented Apr 14, 2022
View reviewed changes
egs/librispeech/ASR/pruned_transducer_stateless3/train.py
train_dl = asr_datamodule.train_dataloaders(
train_cuts,
dynamic_bucketing=False,
on_the_fly_feats=False,
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: The librispeech dataset is using pre-computed features so torchaudio_load should not be invoked

@csukuangfj
Copy link
Collaborator Author

Here are the results for far for this PR:

greedy search

(py38) kuangfangjun:greedy_search$ grep -r -n --color "best for test-clean" log-* | sort -n -k2 | head
log-decode-epoch-17-avg-4-context-2-max-sym-per-frame-1-2022-04-17-22-46-50:15:greedy_search    2.47    best for test-clean
log-decode-epoch-17-avg-5-context-2-max-sym-per-frame-1-2022-04-17-22-47-45:15:greedy_search    2.47    best for test-clean
log-decode-epoch-17-avg-6-context-2-max-sym-per-frame-1-2022-04-17-22-48-42:15:greedy_search    2.48    best for test-clean
log-decode-epoch-17-avg-3-context-2-max-sym-per-frame-1-2022-04-17-22-45-56:15:greedy_search    2.49    best for test-clean
log-decode-epoch-16-avg-4-context-2-max-sym-per-frame-1-2022-04-17-21-43-34:15:greedy_search    2.5     best for test-clean
log-decode-epoch-16-avg-5-context-2-max-sym-per-frame-1-2022-04-17-21-44-33:15:greedy_search    2.5     best for test-clean
log-decode-epoch-17-avg-7-context-2-max-sym-per-frame-1-2022-04-17-22-49-41:15:greedy_search    2.51    best for test-clean
log-decode-epoch-16-avg-3-context-2-max-sym-per-frame-1-2022-04-17-21-42-41:15:greedy_search    2.52    best for test-clean
log-decode-epoch-16-avg-6-context-2-max-sym-per-frame-1-2022-04-17-21-45-34:15:greedy_search    2.53    best for test-clean
log-decode-epoch-17-avg-8-context-2-max-sym-per-frame-1-2022-04-17-22-50-40:15:greedy_search    2.53    best for test-clean
(py38) kuangfangjun:greedy_search$ grep -r -n --color "best for test-other" log-* | sort -n -k2 | head
log-decode-epoch-17-avg-7-context-2-max-sym-per-frame-1-2022-04-17-22-49-41:23:greedy_search    6.13    best for test-other
log-decode-epoch-17-avg-3-context-2-max-sym-per-frame-1-2022-04-17-22-45-56:23:greedy_search    6.15    best for test-other
log-decode-epoch-17-avg-5-context-2-max-sym-per-frame-1-2022-04-17-22-47-45:23:greedy_search    6.15    best for test-other
log-decode-epoch-17-avg-8-context-2-max-sym-per-frame-1-2022-04-17-22-50-40:23:greedy_search    6.15    best for test-other
log-decode-epoch-16-avg-4-context-2-max-sym-per-frame-1-2022-04-17-21-43-34:23:greedy_search    6.18    best for test-other
log-decode-epoch-17-avg-6-context-2-max-sym-per-frame-1-2022-04-17-22-48-42:23:greedy_search    6.18    best for test-other
log-decode-epoch-17-avg-4-context-2-max-sym-per-frame-1-2022-04-17-22-46-50:23:greedy_search    6.21    best for test-other
log-decode-epoch-17-avg-9-context-2-max-sym-per-frame-1-2022-04-17-22-51-40:23:greedy_search    6.21    best for test-other
log-decode-epoch-16-avg-5-context-2-max-sym-per-frame-1-2022-04-17-21-44-33:23:greedy_search    6.23    best for test-other
log-decode-epoch-16-avg-6-context-2-max-sym-per-frame-1-2022-04-17-21-45-34:23:greedy_search    6.23    best for test-other

modified beam search

(py38) kuangfangjun:modified_beam_search$ grep -r -n --color "best for test-clean" log-* | sort -n -k2 | head
log-decode-epoch-17-avg-5-beam-4-2022-04-17-23-12-03:34:beam_size_4     2.4     best for test-clean
log-decode-epoch-17-avg-4-beam-4-2022-04-17-23-07-17:34:beam_size_4     2.41    best for test-clean
log-decode-epoch-16-avg-5-beam-4-2022-04-17-22-06-19:34:beam_size_4     2.43    best for test-clean
log-decode-epoch-17-avg-3-beam-4-2022-04-17-23-02-54:34:beam_size_4     2.43    best for test-clean
log-decode-epoch-17-avg-6-beam-4-2022-04-17-23-16-42:34:beam_size_4     2.43    best for test-clean
log-decode-epoch-16-avg-3-beam-4-2022-04-17-21-58-29:34:beam_size_4     2.44    best for test-clean
log-decode-epoch-16-avg-4-beam-4-2022-04-17-22-02-24:34:beam_size_4     2.44    best for test-clean
log-decode-epoch-17-avg-7-beam-4-2022-04-17-23-32-49:34:beam_size_4     2.44    best for test-clean
log-decode-epoch-16-avg-6-beam-4-2022-04-17-22-10-17:34:beam_size_4     2.45    best for test-clean
log-decode-epoch-17-avg-8-beam-4-2022-04-18-00-08-25:34:beam_size_4     2.45    best for test-clean
(py38) kuangfangjun:modified_beam_search$ grep -r -n --color "best for test-other" log-* | sort -n -k2 | head
log-decode-epoch-17-avg-7-beam-4-2022-04-17-23-32-49:61:beam_size_4     5.98    best for test-other
log-decode-epoch-17-avg-3-beam-4-2022-04-17-23-02-54:61:beam_size_4     6.0     best for test-other
log-decode-epoch-17-avg-5-beam-4-2022-04-17-23-12-03:61:beam_size_4     6.01    best for test-other
log-decode-epoch-17-avg-6-beam-4-2022-04-17-23-16-42:61:beam_size_4     6.01    best for test-other
log-decode-epoch-17-avg-8-beam-4-2022-04-18-00-08-25:61:beam_size_4     6.03    best for test-other
log-decode-epoch-16-avg-5-beam-4-2022-04-17-22-06-19:61:beam_size_4     6.04    best for test-other
log-decode-epoch-16-avg-6-beam-4-2022-04-17-22-10-17:61:beam_size_4     6.05    best for test-other
log-decode-epoch-17-avg-4-beam-4-2022-04-17-23-07-17:61:beam_size_4     6.05    best for test-other
log-decode-epoch-16-avg-4-beam-4-2022-04-17-22-02-24:61:beam_size_4     6.06    best for test-other
log-decode-epoch-15-avg-5-beam-4-2022-04-17-16-06-20:61:beam_size_4     6.07    best for test-other

Note: The WER for test-clean is 2.4 at epoch-17, --avg=5. It is expected to be lower as the training continues.

I will also try to decode the gigaspeech test datasets with the trained model.

@csukuangfj csukuangfj mentioned this pull request Apr 20, 2022
Merged
@csukuangfj csukuangfj mentioned this pull request Apr 21, 2022
Open
@csukuangfj
Copy link
Collaborator Author

Here are the latest results for this PR.

Giga prob 0.5

Also, Gigaspeech is speed-perturbed with factors 0.9 and 1.1

Training time per epoch: about 2 hours

greedy search

(py38) kuangfangjun:greedy_search$ grep -r -n --color "best for test-clean" log-* | sort -n -k2 | head
log-decode-epoch-29-avg-14-context-2-max-sym-per-frame-1-2022-04-19-12-10-16:15:greedy_search   2.31    best for test-clean
log-decode-epoch-30-avg-15-context-2-max-sym-per-frame-1-2022-04-19-11-55-20:15:greedy_search   2.32    best for test-clean
log-decode-epoch-33-avg-16-context-2-max-sym-per-frame-1-2022-04-19-22-56-07:15:greedy_search   2.32    best for test-clean
log-decode-epoch-29-avg-11-context-2-max-sym-per-frame-1-2022-04-19-12-07-07:15:greedy_search   2.33    best for test-clean
log-decode-epoch-29-avg-13-context-2-max-sym-per-frame-1-2022-04-19-12-09-12:15:greedy_search   2.33    best for test-clean
log-decode-epoch-32-avg-15-context-2-max-sym-per-frame-1-2022-04-19-15-04-25:15:greedy_search   2.33    best for test-clean
log-decode-epoch-27-avg-10-context-2-max-sym-per-frame-1-2022-04-19-13-46-24:15:greedy_search   2.34    best for test-clean
log-decode-epoch-28-avg-11-context-2-max-sym-per-frame-1-2022-04-19-12-20-48:15:greedy_search   2.34    best for test-clean
log-decode-epoch-28-avg-12-context-2-max-sym-per-frame-1-2022-04-19-12-21-50:15:greedy_search   2.34    best for test-clean
log-decode-epoch-28-avg-13-context-2-max-sym-per-frame-1-2022-04-19-12-22-56:15:greedy_search   2.34    best for test-clean
(py38) kuangfangjun:greedy_search$ grep -r -n --color "best for test-other" log-* | sort -n -k2 | head
log-decode-epoch-33-avg-17-context-2-max-sym-per-frame-1-2022-04-19-22-57-22:23:greedy_search   5.6     best for test-other
log-decode-epoch-30-avg-11-context-2-max-sym-per-frame-1-2022-04-19-11-45-16:23:greedy_search   5.62    best for test-other
log-decode-epoch-31-avg-15-context-2-max-sym-per-frame-1-2022-04-19-15-17-24:23:greedy_search   5.63    best for test-other
log-decode-epoch-32-avg-16-context-2-max-sym-per-frame-1-2022-04-19-15-05-34:23:greedy_search   5.63    best for test-other
log-decode-epoch-31-avg-12-context-2-max-sym-per-frame-1-2022-04-19-15-14-41:23:greedy_search   5.64    best for test-other
log-decode-epoch-32-avg-13-context-2-max-sym-per-frame-1-2022-04-19-15-02-19:23:greedy_search   5.64    best for test-other
log-decode-epoch-33-avg-14-context-2-max-sym-per-frame-1-2022-04-19-22-53-32:23:greedy_search   5.64    best for test-other
log-decode-epoch-33-avg-18-context-2-max-sym-per-frame-1-2022-04-19-22-58-37:23:greedy_search   5.64    best for test-other
log-decode-epoch-35-avg-20-context-2-max-sym-per-frame-1-2022-04-20-07-52-11:23:greedy_search   5.64    best for test-other
log-decode-epoch-30-avg-14-context-2-max-sym-per-frame-1-2022-04-19-11-48-11:23:greedy_search   5.65    best for test-other

modified beam search

(py38) kuangfangjun:modified_beam_search$ grep -r -n --color "best for test-clean" log-* | sort -n -k2 | head
log-decode-epoch-29-avg-18-beam-4-2022-04-19-15-56-44:34:beam_size_4    2.29    best for test-clean
log-decode-epoch-28-avg-14-beam-4-2022-04-19-13-16-10:34:beam_size_4    2.3     best for test-clean
log-decode-epoch-28-avg-17-beam-4-2022-04-19-16-01-57:34:beam_size_4    2.3     best for test-clean
log-decode-epoch-29-avg-12-beam-4-2022-04-19-12-43-44:34:beam_size_4    2.3     best for test-clean
log-decode-epoch-29-avg-16-beam-4-2022-04-19-14-34-56:34:beam_size_4    2.3     best for test-clean
log-decode-epoch-29-avg-17-beam-4-2022-04-19-15-51-36:34:beam_size_4    2.3     best for test-clean
log-decode-epoch-30-avg-13-beam-4-2022-04-19-12-20-48:34:beam_size_4    2.3     best for test-clean
log-decode-epoch-32-avg-17-beam-4-2022-04-19-15-21-30:34:beam_size_4    2.3     best for test-clean
log-decode-epoch-28-avg-13-beam-4-2022-04-19-13-11-40:34:beam_size_4    2.31    best for test-clean
log-decode-epoch-28-avg-15-beam-4-2022-04-19-14-53-17:34:beam_size_4    2.31    best for test-clean
(py38) kuangfangjun:modified_beam_search$ grep -r -n --color "best for test-other" log-* | sort -n -k2 | head
log-decode-epoch-32-avg-18-beam-4-2022-04-19-15-26-18:61:beam_size_4    5.52    best for test-other
log-decode-epoch-30-avg-15-beam-4-2022-04-19-13-52-05:61:beam_size_4    5.53    best for test-other
log-decode-epoch-30-avg-14-beam-4-2022-04-19-12-29-45:61:beam_size_4    5.54    best for test-other
log-decode-epoch-31-avg-17-beam-4-2022-04-19-15-31-10:61:beam_size_4    5.54    best for test-other
log-decode-epoch-32-avg-17-beam-4-2022-04-19-15-21-30:61:beam_size_4    5.54    best for test-other
log-decode-epoch-28-avg-12-beam-4-2022-04-19-13-06-55:61:beam_size_4    5.55    best for test-other
log-decode-epoch-30-avg-16-beam-4-2022-04-19-14-02-53:61:beam_size_4    5.55    best for test-other
log-decode-epoch-28-avg-13-beam-4-2022-04-19-13-11-40:61:beam_size_4    5.56    best for test-other
log-decode-epoch-29-avg-13-beam-4-2022-04-19-12-48-18:61:beam_size_4    5.56    best for test-other
log-decode-epoch-29-avg-14-beam-4-2022-04-19-12-53-04:61:beam_size_4    5.56    best for test-other

Giga prob 0.8

Note: Gigapssech is not speed-perturbed.

Time per epoch: 5 hours 40 minutes

greedy search

(py38) kuangfangjun:greedy_search$ grep -r -n --color "best for test-clean" log-* | sort -n -k2 | head
log-decode-epoch-19-avg-4-context-2-max-sym-per-frame-1-2022-04-25-12-01-30:15:greedy_search    2.32    best for test-clean
log-decode-epoch-20-avg-3-context-2-max-sym-per-frame-1-2022-04-25-17-13-40:15:greedy_search    2.32    best for test-clean
log-decode-epoch-20-avg-5-context-2-max-sym-per-frame-1-2022-04-25-17-15-53:15:greedy_search    2.32    best for test-clean
log-decode-epoch-19-avg-6-context-2-max-sym-per-frame-1-2022-04-25-12-04-07:15:greedy_search    2.33    best for test-clean
log-decode-epoch-20-avg-4-context-2-max-sym-per-frame-1-2022-04-25-17-14-38:15:greedy_search    2.33    best for test-clean
log-decode-epoch-20-avg-7-context-2-max-sym-per-frame-1-2022-04-25-17-18-30:15:greedy_search    2.33    best for test-clean
log-decode-epoch-20-avg-8-context-2-max-sym-per-frame-1-2022-04-25-17-19-38:15:greedy_search    2.33    best for test-clean
log-decode-epoch-18-avg-6-context-2-max-sym-per-frame-1-2022-04-25-10-40-00:15:greedy_search    2.34    best for test-clean
log-decode-epoch-19-avg-3-context-2-max-sym-per-frame-1-2022-04-25-12-00-18:15:greedy_search    2.34    best for test-clean
log-decode-epoch-20-avg-2-context-2-max-sym-per-frame-1-2022-04-25-17-12-28:15:greedy_search    2.34    best for test-clean
(py38) kuangfangjun:greedy_search$ grep -r -n --color "best for test-other" log-* | sort -n -k2 | head
log-decode-epoch-20-avg-5-context-2-max-sym-per-frame-1-2022-04-25-17-15-53:23:greedy_search    5.2     best for test-other
log-decode-epoch-20-avg-4-context-2-max-sym-per-frame-1-2022-04-25-17-14-38:23:greedy_search    5.21    best for test-other
log-decode-epoch-20-avg-7-context-2-max-sym-per-frame-1-2022-04-25-17-18-30:23:greedy_search    5.22    best for test-other
log-decode-epoch-20-avg-6-context-2-max-sym-per-frame-1-2022-04-25-17-17-19:23:greedy_search    5.23    best for test-other
log-decode-epoch-20-avg-3-context-2-max-sym-per-frame-1-2022-04-25-17-13-40:23:greedy_search    5.24    best for test-other
log-decode-epoch-19-avg-6-context-2-max-sym-per-frame-1-2022-04-25-12-04-07:23:greedy_search    5.25    best for test-other
log-decode-epoch-20-avg-2-context-2-max-sym-per-frame-1-2022-04-25-17-12-28:23:greedy_search    5.25    best for test-other
log-decode-epoch-20-avg-9-context-2-max-sym-per-frame-1-2022-04-25-17-21-02:23:greedy_search    5.25    best for test-other
log-decode-epoch-20-avg-8-context-2-max-sym-per-frame-1-2022-04-25-17-19-38:23:greedy_search    5.26    best for test-other
log-decode-epoch-19-avg-4-context-2-max-sym-per-frame-1-2022-04-25-12-01-30:23:greedy_search    5.27    best for test-other

modified beam search

(py38) kuangfangjun:modified_beam_search$ grep -r -n --color "best for test-clean" log-* | sort -n -k2 | head
log-decode-epoch-20-avg-4-beam-8-2022-04-25-17-28-46:34:beam_size_8     2.28    best for test-clean
log-decode-epoch-20-avg-5-beam-8-2022-04-25-17-44-25:34:beam_size_8     2.28    best for test-clean
log-decode-epoch-19-avg-4-beam-4-2022-04-25-12-19-40:34:beam_size_4     2.3     best for test-clean
log-decode-epoch-18-avg-4-beam-4-2022-04-25-11-10-55:34:beam_size_4     2.31    best for test-clean
log-decode-epoch-18-avg-5-beam-4-2022-04-25-11-16-41:34:beam_size_4     2.31    best for test-clean
log-decode-epoch-18-avg-7-beam-4-2022-04-25-11-25-17:34:beam_size_4     2.31    best for test-clean
log-decode-epoch-19-avg-5-beam-4-2022-04-25-12-23-57:34:beam_size_4     2.31    best for test-clean
log-decode-epoch-19-avg-6-beam-4-2022-04-25-12-28-16:34:beam_size_4     2.31    best for test-clean
log-decode-epoch-18-avg-8-beam-4-2022-04-25-11-33-26:34:beam_size_4     2.32    best for test-clean
log-decode-epoch-19-avg-4-beam-8-2022-04-25-13-50-48:34:beam_size_8     2.32    best for test-clean
(py38) kuangfangjun:modified_beam_search$ grep -r -n --color "best for test-other" log-* | sort -n -k2 | head
log-decode-epoch-20-avg-5-beam-8-2022-04-25-17-44-25:61:beam_size_8     5.13    best for test-other
log-decode-epoch-20-avg-4-beam-8-2022-04-25-17-28-46:61:beam_size_8     5.15    best for test-other
log-decode-epoch-19-avg-6-beam-4-2022-04-25-12-28-16:61:beam_size_4     5.17    best for test-other
log-decode-epoch-19-avg-7-beam-4-2022-04-25-12-32-37:61:beam_size_4     5.2     best for test-other
log-decode-epoch-18-avg-3-beam-4-2022-04-25-10-55-21:61:beam_size_4     5.21    best for test-other
log-decode-epoch-19-avg-4-beam-8-2022-04-25-13-50-48:61:beam_size_8     5.21    best for test-other
log-decode-epoch-19-avg-5-beam-4-2022-04-25-12-23-57:61:beam_size_4     5.21    best for test-other
log-decode-epoch-19-avg-3-beam-4-2022-04-25-12-13-33:61:beam_size_4     5.22    best for test-other
log-decode-epoch-19-avg-4-beam-4-2022-04-25-12-19-40:61:beam_size_4     5.22    best for test-other
log-decode-epoch-19-avg-8-beam-4-2022-04-25-12-36-54:61:beam_size_4     5.22    best for test-other

You can see that using more data in training is helpful in reducing WER.

@danpovey
Copy link
Collaborator

Great! So higher giga-prob seems to help more on test-other (like most things, I suppose). These are good results.
We could probably do even better if we used all of gigaspeech.

@csukuangfj
Copy link
Collaborator Author

csukuangfj commented Apr 29, 2022 •
edited
Loading

Here are the final results for this PR
(See also https://github.com/csukuangfj/icefall/blob/modified-conformer-with-multi-datasets/egs/librispeech/ASR/RESULTS.md)

test-clean test-other comment
greedy search (max sym per frame 1) 2.21 5.09 --epoch 27 --avg 2 --max-duration 600
greedy search (max sym per frame 1) 2.25 5.02 --epoch 27 --avg 12 --max-duration 600
modified beam search 2.19 5.03 --epoch 25 --avg 6 --max-duration 600
modified beam search 2.23 4.94 --epoch 27 --avg 10 --max-duration 600
beam search 2.16 4.95 --epoch 25 --avg 7 --max-duration 600
fast beam search 2.21 4.96 --epoch 27 --avg 10 --max-duration 600
fast beam search 2.19 4.97 --epoch 27 --avg 12 --max-duration 600

The following table shows the
Nbest oracle WER
for fast beam search.

epoch avg num_paths nbest_scale test-clean test-other
27 10 50 0.5 0.91 2.74
27 10 50 0.8 0.94 2.82
27 10 50 1.0 1.06 2.88
27 10 100 0.5 0.82 2.58
27 10 100 0.8 0.92 2.65
27 10 100 1.0 0.95 2.77
27 10 200 0.5 0.81 2.50
27 10 200 0.8 0.85 2.56
27 10 200 1.0 0.91 2.64
27 10 400 0.5 N/A N/A
27 10 400 0.8 0.81 2.49
27 10 400 1.0 0.85 2.54

The Nbest oracle WER is computed using the following steps:

    1. Use fast_beam_search to produce a lattice.
    1. Extract N paths from the lattice using k2.random_path
    1. Unique paths so that each path
      has a distinct sequence of tokens
    1. Compute the edit distance of each path with the ground truth
    1. The path with the lowest edit distance is the final output and is used to
      compute the WER

The command to compute the Nbest oracle WER is:

for epoch in 27; do
  for avg in 10 ; do
    for num_paths in 50 100 200 400; do
      for nbest_scale in 0.5 0.8 1.0; do
        ./pruned_transducer_stateless3/decode.py \
            --epoch $epoch \
            --avg $avg \
            --exp-dir ./pruned_transducer_stateless3/exp \
            --max-duration 600 \
            --decoding-method fast_beam_search_nbest_oracle \
            --num-paths $num_paths \
            --max-states 32 \
            --beam 8 \
            --nbest-scale $nbest_scale
      done
    done
  done
done

Pretrained models, training logs, decoding logs, and decoding results
are available at
https://huggingface.co/csukuangfj/icefall-asr-librispeech-pruned-transducer-stateless3-2022-04-29

Note: The Nbest oracle WER shows that the WERs can be further reduced if some Nbest based LM rescoring models
are available.

I have tried to increase giga prob from 0.8 to 0.9. However, the training time per epoch is increased from 5 hours 40 minutes to 14 hours. It processes 12.5 k hours of training data for giga prob 0.8 and 30k hours of data for giga prob 0.9.

Note: For giga prob 0.8, a new epoch is started as soon as the GigaSpeech dataset is exhausted, which is 10k hours of data.
For giga prob 0.9, the GigaSpeech dataset is repeated and will never be exhausted; a new epoch is started when the librispeech dataset is exhausted.

@csukuangfj csukuangfj added ready and removed ready labels Apr 29, 2022
@csukuangfj
Copy link
Collaborator Author

I am merging it.

@csukuangfj csukuangfj merged commit ac84220 into k2-fsa:master Apr 29, 2022
@csukuangfj csukuangfj deleted the modified-conformer-with-multi-datasets branch April 29, 2022 07:40
@danpovey
Copy link
Collaborator

cool!

This was referenced May 4, 2022
Closed
Merged
yaozengwei added a commit that referenced this pull request May 15, 2022
@yaozengwei @danpovey
@pkufool @luomingshuang @csukuangfj @glynpu @wgb14 @whsqkaak @pehonnet
Streaming merge (#369)
a9dccdc 
* Remove ReLU in attention

* Adding diagnostics code...

* Refactor/simplify ConformerEncoder

* First version of rand-combine iterated-training-like idea.

* Improvements to diagnostics (RE those with 1 dim

* Add pelu to this good-performing setup..

* Small bug fixes/imports

* Add baseline for the PeLU expt, keeping only the small normalization-related changes.

* pelu_base->expscale, add 2xExpScale in subsampling, and in feedforward units.

* Double learning rate of exp-scale units

* Combine ExpScale and swish for memory reduction

* Add import

* Fix backprop bug

* Fix bug in diagnostics

* Increase scale on Scale from 4 to 20

* Increase scale from 20 to 50.

* Fix duplicate Swish; replace norm+swish with swish+exp-scale in convolution module

* Reduce scale from 50 to 20

* Add deriv-balancing code

* Double the threshold in brelu; slightly increase max_factor.

* Fix exp dir

* Convert swish nonlinearities to ReLU

* Replace relu with swish-squared.

* Restore ConvolutionModule to state before changes; change all Swish,Swish(Swish) to SwishOffset.

* Replace norm on input layer with scale of 0.1.

* Extensions to diagnostics code

* Update diagnostics

* Add BasicNorm module

* Replace most normalizations with scales (still have norm in conv)

* Change exp dir

* Replace norm in ConvolutionModule with a scaling factor.

* use nonzero threshold in DerivBalancer

* Add min-abs-value 0.2

* Fix dirname

* Change min-abs threshold from 0.2 to 0.5

* Scale up pos_bias_u and pos_bias_v before use.

* Reduce max_factor to 0.01

* Fix q*scaling logic

* Change max_factor in DerivBalancer from 0.025 to 0.01; fix scaling code.

* init 1st conv module to smaller variance

* Change how scales are applied; fix residual bug

* Reduce min_abs from 0.5 to 0.2

* Introduce in_scale=0.5 for SwishExpScale

* Fix scale from 0.5 to 2.0 as I really intended..

* Set scaling on SwishExpScale

* Add identity pre_norm_final for diagnostics.

* Add learnable post-scale for mha

* Fix self.post-scale-mha

* Another rework, use scales on linear/conv

* Change dir name

* Reduce initial scaling of modules

* Bug-fix RE bias

* Cosmetic change

* Reduce initial_scale.

* Replace ExpScaleRelu with DoubleSwish()

* DoubleSwish fix

* Use learnable scales for joiner and decoder

* Add max-abs-value constraint in DerivBalancer

* Add max-abs-value

* Change dir name

* Remove ExpScale in feedforward layes.

* Reduce max-abs limit from 1000 to 100; introduce 2 DerivBalancer modules in conv layer.

* Make DoubleSwish more memory efficient

* Reduce constraints from deriv-balancer in ConvModule.

* Add warmup mode

* Remove max-positive constraint in deriv-balancing; add second DerivBalancer in conv module.

* Add some extra info to diagnostics

* Add deriv-balancer at output of embedding.

* Add more stats.

* Make epsilon in BasicNorm learnable, optionally.

* Draft of 0mean changes..

* Rework of initialization

* Fix typo

* Remove dead code

* Modifying initialization from normal->uniform; add initial_scale when initializing

* bug fix re sqrt

* Remove xscale from pos_embedding

* Remove some dead code.

* Cosmetic changes/renaming things

* Start adding some files..

* Add more files..

* update decode.py file type

* Add remaining files in pruned_transducer_stateless2

* Fix diagnostics-getting code

* Scale down pruned loss in warmup mode

* Reduce warmup scale on pruned loss form 0.1 to 0.01.

* Remove scale_speed, make swish deriv more efficient.

* Cosmetic changes to swish

* Double warm_step

* Fix bug with import

* Change initial std from 0.05 to 0.025.

* Set also scale for embedding to 0.025.

* Remove logging code that broke with newer Lhotse; fix bug with pruned_loss

* Add norm+balancer to VggSubsampling

* Incorporate changes from master into pruned_transducer_stateless2.

* Add max-abs=6, debugged version

* Change 0.025,0.05 to 0.01 in initializations

* Fix balancer code

* Whitespace fix

* Reduce initial pruned_loss scale from 0.01 to 0.0

* Increase warm_step (and valid_interval)

* Change max-abs from 6 to 10

* Change how warmup works.

* Add changes from master to decode.py, train.py

* Simplify the warmup code; max_abs 10->6

* Make warmup work by scaling layer contributions; leave residual layer-drop

* Fix bug

* Fix test mode with random layer dropout

* Add random-number-setting function in dataloader

* Fix/patch how fix_random_seed() is imported.

* Reduce layer-drop prob

* Reduce layer-drop prob after warmup to 1 in 100

* Change power of lr-schedule from -0.5 to -0.333

* Increase model_warm_step to 4k

* Change max-keep-prob to 0.95

* Refactoring and simplifying conformer and frontend

* Rework conformer, remove some code.

* Reduce 1st conv channels from 64 to 32

* Add another convolutional layer

* Fix padding bug

* Remove dropout in output layer

* Reduce speed of some components

* Initial refactoring to remove unnecessary vocab_size

* Fix RE identity

* Bug-fix

* Add final dropout to conformer

* Remove some un-used code

* Replace nn.Linear with ScaledLinear in simple joiner

* Make 2 projections..

* Reduce initial_speed

* Use initial_speed=0.5

* Reduce initial_speed further from 0.5 to 0.25

* Reduce initial_speed from 0.5 to 0.25

* Change how warmup is applied.

* Bug fix to warmup_scale

* Fix test-mode

* Remove final dropout

* Make layer dropout rate 0.075, was 0.1.

* First draft of model rework

* Various bug fixes

* Change learning speed of simple_lm_proj

* Revert transducer_stateless/ to state in upstream/master

* Fix to joiner to allow different dims

* Some cleanups

* Make training more efficient, avoid redoing some projections.

* Change how warm-step is set

* First draft of new approach to learning rates + init

* Some fixes..

* Change initialization to 0.25

* Fix type of parameter

* Fix weight decay formula by adding 1/1-beta

* Fix weight decay formula by adding 1/1-beta

* Fix checkpoint-writing

* Fix to reading scheudler from optim

* Simplified optimizer, rework somet things..

* Reduce model_warm_step from 4k to 3k

* Fix bug in lambda

* Bug-fix RE sign of target_rms

* Changing initial_speed from 0.25 to 01

* Change some defaults in LR-setting rule.

* Remove initial_speed

* Set new scheduler

* Change exponential part of lrate to be epoch based

* Fix bug

* Set 2n rule..

* Implement 2o schedule

* Make lrate rule more symmetric

* Implement 2p version of learning rate schedule.

* Refactor how learning rate is set.

* Fix import

* Modify init (#301)

* update icefall/__init__.py to import more common functions.

* update icefall/__init__.py

* make imports style consistent.

* exclude black check for icefall/__init__.py in pyproject.toml.

* Minor fixes for logging (#296)

* Minor fixes for logging

* Minor fix

* Fix dir names

* Modify beam search to be efficient with current joienr

* Fix adding learning rate to tensorboard

* Fix docs in optim.py

* Support mix precision training on the reworked model (#305)

* Add mix precision support

* Minor fixes

* Minor fixes

* Minor fixes

* Tedlium3 pruned transducer stateless (#261)

* update tedlium3-pruned-transducer-stateless-codes

* update README.md

* update README.md

* add fast beam search for decoding

* do a change for RESULTS.md

* do a change for RESULTS.md

* do a fix

* do some changes for pruned RNN-T

* Add mix precision support

* Minor fixes

* Minor fixes

* Updating RESULTS.md; fix in beam_search.py

* Fix rebase

* Code style check for librispeech pruned transducer stateless2 (#308)

* Update results for tedlium3 pruned RNN-T (#307)

* Update README.md

* Fix CI errors. (#310)

* Add more results

* Fix tensorboard log location

* Add one more epoch of full expt

* fix comments

* Add results for mixed precision with max-duration 300

* Changes for pretrained.py (tedlium3 pruned RNN-T) (#311)

* GigaSpeech recipe (#120)

* initial commit

* support download, data prep, and fbank

* on-the-fly feature extraction by default

* support BPE based lang

* support HLG for BPE

* small fix

* small fix

* chunked feature extraction by default

* Compute features for GigaSpeech by splitting the manifest.

* Fixes after review.

* Split manifests into 2000 pieces.

* set audio duration mismatch tolerance to 0.01

* small fix

* add conformer training recipe

* Add conformer.py without pre-commit checking

* lazy loading and use SingleCutSampler

* DynamicBucketingSampler

* use KaldifeatFbank to compute fbank for musan

* use pretrained language model and lexicon

* use 3gram to decode, 4gram to rescore

* Add decode.py

* Update .flake8

* Delete compute_fbank_gigaspeech.py

* Use BucketingSampler for valid and test dataloader

* Update params in train.py

* Use bpe_500

* update params in decode.py

* Decrease num_paths while CUDA OOM

* Added README

* Update RESULTS

* black

* Decrease num_paths while CUDA OOM

* Decode with post-processing

* Update results

* Remove lazy_load option

* Use default `storage_type`

* Keep the original tolerance

* Use split-lazy

* black

* Update pretrained model

Co-authored-by: Fangjun Kuang <csukuangfj@gmail.com>

* Add LG decoding (#277)

* Add LG decoding

* Add log weight pushing

* Minor fixes

* Support computing RNN-T loss with torchaudio (#316)

* Update results for torchaudio RNN-T. (#322)

* Fix some typos. (#329)

* fix fp16 option in example usage (#332)

* Support averaging models with weight tying. (#333)

* Support specifying iteration number of checkpoints for decoding. (#336)

See also #289

* Modified conformer with multi datasets (#312)

* Copy files for editing.

* Use librispeech + gigaspeech with modified conformer.

* Support specifying number of workers for on-the-fly feature extraction.

* Feature extraction code for GigaSpeech.

* Combine XL splits lazily during training.

* Fix warnings in decoding.

* Add decoding code for GigaSpeech.

* Fix decoding the gigaspeech dataset.

We have to use the decoder/joiner networks for the GigaSpeech dataset.

* Disable speed perturbe for XL subset.

* Compute the Nbest oracle WER for RNN-T decoding.

* Minor fixes.

* Minor fixes.

* Add results.

* Update results.

* Update CI.

* Update results.

* Fix style issues.

* Update results.

* Fix style issues.

* Update results. (#340)

* Update results.

* Typo fixes.

* Validate generated manifest files. (#338)

* Validate generated manifest files. (#338)

* Save batch to disk on OOM. (#343)

* Save batch to disk on OOM.

* minor fixes

* Fixes after review.

* Fix style issues.

* Fix decoding for gigaspeech in the libri + giga setup. (#345)

* Model average (#344)

* First upload of model average codes.

* minor fix

* update decode file

* update .flake8

* rename pruned_transducer_stateless3 to pruned_transducer_stateless4

* change epoch number counter starting from 1 instead of 0

* minor fix of pruned_transducer_stateless4/train.py

* refactor the checkpoint.py

* minor fix, update docs, and modify the epoch number to count from 1 in the pruned_transducer_stateless4/decode.py

* update author info

* add docs of the scaling in function average_checkpoints_with_averaged_model

* Save batch to disk on exception. (#350)

* Bug fix (#352)

* Keep model_avg on cpu (#348)

* keep model_avg on cpu

* explicitly convert model_avg to cpu

* minor fix

* remove device convertion for model_avg

* modify usage of the model device in train.py

* change model.device to next(model.parameters()).device for decoding

* assert params.start_epoch>0

* assert params.start_epoch>0, params.start_epoch

* Do some changes for aishell/ASR/transducer stateless/export.py (#347)

* do some changes for aishell/ASR/transducer_stateless/export.py

* Support decoding with averaged model when using --iter (#353)

* support decoding with averaged model when using --iter

* minor fix

* monir fix of copyright date

* Stringify torch.__version__ before serializing it. (#354)

* Run decode.py in GitHub actions. (#356)

* Ignore padding frames during RNN-T decoding. (#358)

* Ignore padding frames during RNN-T decoding.

* Fix outdated decoding code.

* Minor fixes.

* Support --iter in export.py (#360)

* GigaSpeech RNN-T experiments (#318)

* Copy RNN-T recipe from librispeech

* flake8

* flake8

* Update params

* gigaspeech decode

* black

* Update results

* syntax highlight

* Update RESULTS.md

* typo

* Update decoding script for gigaspeech and remove duplicate files. (#361)

* Validate that there are no OOV tokens in BPE-based lexicons. (#359)

* Validate that there are no OOV tokens in BPE-based lexicons.

* Typo fixes.

* Decode gigaspeech in GitHub actions (#362)

* Add CI for gigaspeech.

* Update results for libri+giga multi dataset setup. (#363)

* Update results for libri+giga multi dataset setup.

* Update GigaSpeech reults (#364)

* Update decode.py

* Update export.py

* Update results

* Update README.md

* Fix GitHub CI for decoding GigaSpeech dev/test datasets (#366)

* modify .flake8

* minor fix

* minor fix

Co-authored-by: Daniel Povey <dpovey@gmail.com>
Co-authored-by: Wei Kang <wkang@pku.org.cn>
Co-authored-by: Mingshuang Luo <37799481+luomingshuang@users.noreply.github.com>
Co-authored-by: Fangjun Kuang <csukuangfj@gmail.com>
Co-authored-by: Guo Liyong <guonwpu@qq.com>
Co-authored-by: Wang, Guanbo <wgb14@outlook.com>
Co-authored-by: whsqkaak <whsqkaak@naver.com>
Co-authored-by: pehonnet <pe.honnet@gmail.com>
@Cescfangs Cescfangs mentioned this pull request Aug 8, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
ready
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@csukuangfj @danpovey @pzelasko