Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

uing shard data type for librispeech, I got errors " WARNING error to parse" #954

Closed
NathanJHLee opened this issue Feb 22, 2022 · 3 comments
Closed

uing shard data type for librispeech, I got errors " WARNING error to parse" #954

NathanJHLee opened this issue Feb 22, 2022 · 3 comments

Comments

@NathanJHLee
Copy link

NathanJHLee commented Feb 22, 2022 •
edited
Loading

Describe the bug

HI, I am trying to train using shard data type for librispeech data set. This project which is prepared refers from aishell example.
When Training stage at "4", I encountered a problem.
Please check this issue.
Thank you.

My Environments as below
Centos 7.5
torch1.7.1+cuda 10.1
GPU : Tesla P100(16G) x 2

#training
stage : 4
data_type=shard
num_utts_per_shard=1000

#Error message as below

12 ./run_test.sh: init method is file:///home/asr1/DB/wenet-main_2022_0214/examples/librispeech/s0/exp/sp_spec_aug/ddp_init
13 /usr/local/lib64/python3.6/site-packages/torchaudio/backend/utils.py:54: UserWarning: "sox" backend is being deprecated. The default backend will be changed to "sox_io" backend in 0.8.0 and "sox" backend will be removed in 0.9.0. Please migrate to "sox_io" backend. Please refer to pytorch/audio#903 for the detail.
14 '"sox" backend is being deprecated. '
15 2022-02-22 22:52:09,055 INFO training on multiple gpus, this gpu 0
16 /usr/local/lib64/python3.6/site-packages/torchaudio/backend/utils.py:54: UserWarning: "sox" backend is being deprecated. The default backend will be changed to "sox_io" backend in 0.8.0 and "sox" backend will be removed in 0.9.0. Please migrate to "sox_io" backend. Please refer to pytorch/audio#903 for the detail.
17 '"sox" backend is being deprecated. '
18 2022-02-22 22:52:09,095 INFO training on multiple gpus, this gpu 1
19 2022-02-22 22:52:16,403 INFO Epoch 0 TRAIN info lr 1.6e-07
20 2022-02-22 22:52:16,405 INFO using accumulate grad, new batch size is 1 times larger than before
21 2022-02-22 22:52:16,407 INFO Checkpoint: save to checkpoint /home/asr1/DB/wenet-main_2022_0214/examples/librispeech/s0/exp/sp_spec_aug/init.pt
22 ASRModel(
.
.
. #####model described
.
.
1307 the number of model params: 49941268
1308 2022-02-22 22:52:17,477 WARNING error to parse 1769-143485-0039.flac
1309 2022-02-22 22:52:17,478 WARNING error to parse 1769-143485-0040.flac
1310 2022-02-22 22:52:17,478 WARNING error to parse 1769-143485-0041.flac
1311 2022-02-22 22:52:17,479 WARNING error to parse 1769-143485-0042.flac
1312 2022-02-22 22:52:17,479 WARNING error to parse 8803-296085-0016.flac
1313 2022-02-22 22:52:17,480 WARNING error to parse 1769-143485-0043.flac
1314 2022-02-22 22:52:17,479 WARNING error to parse 8273-283130-0008.flac
1315 2022-02-22 22:52:17,480 WARNING error to parse 1343-139681-0016.flac
1316 2022-02-22 22:52:17,480 WARNING error to parse 2361-137615-0037.flac
.
.
.
.
40061 2022-02-22 22:52:20,412 WARNING error to parse 6286-69422-0017.flac
40062 2022-02-22 22:52:20,413 WARNING error to parse 5588-68192-0021.flac
40063 2022-02-22 22:52:20,413 WARNING error to parse 3990-31892-0001.flac
40064 2022-02-22 22:52:20,413 WARNING error to parse 6735-284431-0013.flac
40065 2022-02-22 22:52:20,413 WARNING error to parse 3328-168626-0053.flac
40066 2022-02-22 22:52:20,413 WARNING error to parse 2812-160187-0023.flac
40067 2022-02-22 22:52:20,413 WARNING error to parse 8180-274725-0009.flac
40068 2022-02-22 22:52:20,413 WARNING error to parse 3486-166446-0037.flac
40069 2022-02-22 22:52:20,413 WARNING error to parse 6286-69422-0018.flac
40070 Traceback (most recent call last):
40071 File "wenet/bin/train.py", line 278, in
40072 main()
40073 File "wenet/bin/train.py", line 252, in main
40074 writer, configs, scaler)
40075 File "/home/asr1/DB/wenet-main_2022_0214/wenet/utils/executor.py", line 41, in train
40076 for batch_idx, batch in enumerate(data_loader):
40077 File "/usr/local/lib64/python3.6/site-packages/torch/utils/data/dataloader.py", line 435, in next
40078 data = self._next_data()
40079 File "/usr/local/lib64/python3.6/site-packages/torch/utils/data/dataloader.py", line 1085, in _next_data
40080 return self._process_data(data)
40081 File "/usr/local/lib64/python3.6/site-packages/torch/utils/data/dataloader.py", line 1111, in _process_data
40082 data.reraise()
40083 File "/usr/local/lib64/python3.6/site-packages/torch/_utils.py", line 428, in reraise
40084 raise self.exc_type(msg)
40085 AssertionError: Caught AssertionError in DataLoader worker process 0.
40086 Original Traceback (most recent call last):
40087 File "/usr/local/lib64/python3.6/site-packages/torch/utils/data/_utils/worker.py", line 198, in _worker_loop
40088 data = fetcher.fetch(index)
40089 File "/usr/local/lib64/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 34, in fetch
40090 data = next(self.dataset_iter)
40091 File "/home/asr1/DB/wenet-main_2022_0214/wenet/dataset/processor.py", line 527, in padding
40092 for sample in data:
40093 File "/home/asr1/DB/wenet-main_2022_0214/wenet/dataset/processor.py", line 469, in static_batch
40094 for sample in data:
40095 File "/home/asr1/DB/wenet-main_2022_0214/wenet/dataset/processor.py", line 445, in sort
40096 for sample in data:
40097 File "/home/asr1/DB/wenet-main_2022_0214/wenet/dataset/processor.py", line 417, in shuffle
40098 for sample in data:
40099 File "/home/asr1/DB/wenet-main_2022_0214/wenet/dataset/processor.py", line 383, in spec_aug
40100 for sample in data:
40101 File "/home/asr1/DB/wenet-main_2022_0214/wenet/dataset/processor.py", line 266, in compute_fbank
40102 for sample in data:
40103 File "/home/asr1/DB/wenet-main_2022_0214/wenet/dataset/processor.py", line 238, in speed_perturb
40104 for sample in data:
40105 File "/home/asr1/DB/wenet-main_2022_0214/wenet/dataset/processor.py", line 213, in resample
40106 for sample in data:
40107 File "/home/asr1/DB/wenet-main_2022_0214/wenet/dataset/processor.py", line 181, in filter
40108 assert 'sample_rate' in sample
40109 AssertionError
40110
@robin1001
Copy link
Collaborator

what is your pytorch/torchaudio version?

@NathanJHLee
Copy link
Author

NathanJHLee commented Mar 10, 2022 •
edited
Loading

what is your pytorch/torchaudio version?

Sorry about late check.
At that time, my env is below
torch 1.7.1+cu101
torchaudio 0.7.2
and python 3.6

Actually I already tried at torch1.10.0+cu102 & torchaudio0.10.0+cu102 version environment.
It works fine for wav data set.(But it still has a problem for flac)
I upgraded only torchaudio version, but it's not allow to install because of torch version dependency.
Is there any solution for keeping torch 1.7.1 and work properly?

I think 'dataset/processor.py' cause problem, especially 'tarfile'.

Thank you.
Nathan Lee

@xingchensong
Copy link
Member

xingchensong commented Feb 21, 2023 •
edited
Loading

u can pre-process xx.flac to xx.wav before training recipe

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@robin1001 @xingchensong @NathanJHLee