wav2vec2 doesn't work with torch.distributed.launch & multi GPU

[qqpann]

Environment info

• transformers version: 4.6.0.dev0
• Platform: Linux-4.15.0-140-generic-x86_64-with-debian-buster-sid
• Python version: 3.7.9
• PyTorch version (GPU?): 1.8.1 (True)
• Tensorflow version (GPU?): not installed (NA)
• Using GPU in script?: Yes
• Using distributed or parallel set-up in script?: Yes

Who can help

@patil-suraj @elgeish @patrickvonplaten
I see the readme is written by @patil-suraj and @elgeish , so any help would be appreciated.

Information

Model I am using (Bert, XLNet ...): wav2vec2

The problem arises when using:

• the official example scripts: (give details below)

Although the fine-tuning week is over, the example is pretty useful.
I am working on a voice recognition problem and want to train using distributed learning.
I refer to huggingface's official example here:
https://github.com/huggingface/transformers/blob/master/examples/research_projects/wav2vec2/FINE_TUNE_XLSR_WAV2VEC2.md

To reproduce

Steps to reproduce the behavior:

1. On a clean environment, install requirements and git clone transformers repository.
2. Run multi GPU training code as written in the readme.
3. Bug reproduces.

The code is

git clone https://github.com/huggingface/transformers.git
cd transformers/examples/research_projects/wav2vec2/
mkdir outputs
python -m torch.distributed.launch \
        --nproc_per_node=4 run_common_voice.py \
        --model_name_or_path="facebook/wav2vec2-large-xlsr-53" \
        --dataset_config_name="tr" \
        --output_dir=./outputs \
        --overwrite_output_dir \
        --num_train_epochs="5" \
        --per_device_train_batch_size="16" \
        --learning_rate="3e-4" \
        --warmup_steps="500" \
        --evaluation_strategy="steps" \
        --save_steps="400" \
        --eval_steps="400" \
        --logging_steps="400" \
        --save_total_limit="3" \
        --freeze_feature_extractor \
        --feat_proj_dropout="0.0" \
        --layerdrop="0.1" \
        --gradient_checkpointing \
        --fp16 \
        --group_by_length \
        --do_train --do_eval

Error

The following error occurs.

  0%|                                                                                                                                                                                                                 | 0/275 [00:00<?, ?it/s]/home/aidealab/.conda/envs/hf/lib/python3.7/site-packages/torch/nn/modules/module.py:760: UserWarning: Using non-full backward hooks on a Module that does not return a single Tensor or a tuple of Tensors is deprecated and will be removed in future versions. This hook will be missing some of the grad_output. Please use register_full_backward_hook to get the documented behavior.
  warnings.warn("Using non-full backward hooks on a Module that does not return a "
/home/aidealab/.conda/envs/hf/lib/python3.7/site-packages/torch/nn/modules/module.py:795: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
  warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
Traceback (most recent call last):
  File "run_common_voice.py", line 512, in <module>
    main()
  File "run_common_voice.py", line 484, in main
    train_result = trainer.train(resume_from_checkpoint=checkpoint)
  File "/home/aidealab/.conda/envs/hf/lib/python3.7/site-packages/transformers/trainer.py", line 1118, in train
    tr_loss += self.training_step(model, inputs)
  File "run_common_voice.py", line 230, in training_step
    loss = self.compute_loss(model, inputs)
  File "/home/aidealab/.conda/envs/hf/lib/python3.7/site-packages/transformers/trainer.py", line 1548, in compute_loss
    outputs = model(**inputs)
  File "/home/aidealab/.conda/envs/hf/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/aidealab/.conda/envs/hf/lib/python3.7/site-packages/torch/nn/parallel/distributed.py", line 692, in forward
    if self.reducer._rebuild_buckets():
RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your module has parameters that were not used in producing loss. You can enable unused parameter detection by (1) passing the keyword argument `find_unused_parameters=True` to `torch.nn.parallel.DistributedDataParallel`; (2) making sure all `forward` function outputs participate in calculating loss. If you already have done the above two steps, then the distributed data parallel module wasn't able to locate the output tensors in the return value of your module's `forward` function. Please include the loss function and the structure of the return value of `forward` of your module when reporting this issue (e.g. list, dict, iterable).
Traceback (most recent call last):
  File "run_common_voice.py", line 512, in <module>
    main()
  File "run_common_voice.py", line 484, in main
    train_result = trainer.train(resume_from_checkpoint=checkpoint)
  File "/home/aidealab/.conda/envs/hf/lib/python3.7/site-packages/transformers/trainer.py", line 1118, in train
    tr_loss += self.training_step(model, inputs)
  File "run_common_voice.py", line 230, in training_step
    loss = self.compute_loss(model, inputs)
  File "/home/aidealab/.conda/envs/hf/lib/python3.7/site-packages/transformers/trainer.py", line 1548, in compute_loss
    outputs = model(**inputs)
  File "/home/aidealab/.conda/envs/hf/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/aidealab/.conda/envs/hf/lib/python3.7/site-packages/torch/nn/parallel/distributed.py", line 692, in forward
    if self.reducer._rebuild_buckets():
RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your module has parameters that were not used in producing loss. You can enable unused parameter detection by (1) passing the keyword argument `find_unused_parameters=True` to `torch.nn.parallel.DistributedDataParallel`; (2) making sure all `forward` function outputs participate in calculating loss. If you already have done the above two steps, then the distributed data parallel module wasn't able to locate the output tensors in the return value of your module's `forward` function. Please include the loss function and the structure of the return value of `forward` of your module when reporting this issue (e.g. list, dict, iterable).
Killing subprocess 25001
Killing subprocess 25002
Killing subprocess 25003
Killing subprocess 25004
Traceback (most recent call last):
  File "/home/aidealab/.conda/envs/hf/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/aidealab/.conda/envs/hf/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/aidealab/.conda/envs/hf/lib/python3.7/site-packages/torch/distributed/launch.py", line 340, in <module>
    main()
  File "/home/aidealab/.conda/envs/hf/lib/python3.7/site-packages/torch/distributed/launch.py", line 326, in main
    sigkill_handler(signal.SIGTERM, None)  # not coming back
  File "/home/aidealab/.conda/envs/hf/lib/python3.7/site-packages/torch/distributed/launch.py", line 301, in sigkill_handler
    raise subprocess.CalledProcessError(returncode=last_return_code, cmd=cmd)
subprocess.CalledProcessError: Command '['/home/aidealab/.conda/envs/hf/bin/python', '-u', 'run_common_voice.py', '--local_rank=3', '--model_name_or_path=facebook/wav2vec2-large-xlsr-53', '--dataset_config_name=tr', '--output_dir=/home/aidealab/workspace/transformers/examples/research_projects/wav2vec2/outputs', '--overwrite_output_dir', '--num_train_epochs=5', '--per_device_train_batch_size=16', '--learning_rate=3e-4', '--warmup_steps=500', '--evaluation_strategy=steps', '--save_steps=400', '--eval_steps=400', '--logging_steps=400', '--save_total_limit=3', '--freeze_feature_extractor', '--feat_proj_dropout=0.0', '--layerdrop=0.1', '--gradient_checkpointing', '--fp16', '--group_by_length', '--do_train', '--do_eval']' returned non-zero exit status 1.

Expected behavior

It is expected the script runs without error.

[qqpann]

Error messages in short

Warning

UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
  warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "

This warning may not seem to be the direct reason for the crush, but I encounter this warning in my own scripts as well and ends up the training freeze.

Unproceedable Error

RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your module has parameters that were not used in producing loss. You can enable unused parameter detection by (1) passing the keyword argument `find_unused_parameters=True` to `torch.nn.parallel.DistributedDataParallel`; (2) making sure all `forward` function outputs participate in calculating loss. If you already have done the above two steps, then the distributed data parallel module wasn't able to locate the output tensors in the return value of your module's `forward` function. Please include the loss function and the structure of the return value of `forward` of your module when reporting this issue (e.g. list, dict, iterable).

And I have no idea how to solve this error.

[qqpann]

Non-wav2vec2 case

I also looked up other examples.
https://github.com/huggingface/transformers/tree/master/examples/pytorch/question-answering
And it worked.
So it seems to me a wav2vec2 model's problem with multi GPU training.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[patrickvonplaten]

Easiest solution at the moment will be to follow @stas00 PR at #11638

[stas00]

This error most likely has to do with randomly skipping the layers in LayerDrop - so one gpu skips and another continues -and they get out of sync.

Try to see if the error goes away if you disable its skipping logic and let all layers run, You can see how I did it

transformers/src/transformers/models/wav2vec2/modeling_wav2vec2.py

Lines 643 to 667 in c8acf92

	skip_the_layer = True if self.training and (dropout_probability < self.config.layerdrop) else False
	if not skip_the_layer or deepspeed_zero3_is_enabled:
	# under deepspeed zero3 all gpus must run in sync
	# XXX: could optimize this like synced_gpus in generate_utils but not sure if it's worth the code complication
	if getattr(self.config, "gradient_checkpointing", False) and self.training:
	# create gradient checkpointing function
	def create_custom_forward(module):
	def custom_forward(*inputs):
	return module(*inputs, output_attentions)
	return custom_forward
	layer_outputs = torch.utils.checkpoint.checkpoint(
	create_custom_forward(layer),
	hidden_states,
	attention_mask,
	)
	else:
	layer_outputs = layer(
	hidden_states, attention_mask=attention_mask, output_attentions=output_attentions
	)
	hidden_states = layer_outputs[0]
	if skip_the_layer:
	layer_outputs = (None, None)

I don't think LayerDrop they way it's used can be used with more than one GPU.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[voidful]

FYI, I have the same issue when setting mask_time_prob to 0

[patrickvonplaten]

Hey @voidful - could you add a reproducible code snippet here? :-)

[avishaiElmakies]

Hi, I would love to understand if this issue is being fixed. I am having the same issue with wavlm