[Bug] RuntimeError: min(): Expected reduction dim to be specified for input.numel() == 0. Specify the reduction dim with the 'dim' argument.

[offside609]

Describe the bug

I am training a voice cloning model using VITS. My dataset is in LJSpeech Format. I am trying to train Indian English model straight from character with Phonemizer = False. The training runs for 35-40 epochs and then abruptly stops. Sometimes it runs for even longer, like 15k steps and then stops. I can share the notebook I am using for training. I have successfully completed my training with this notebook several times, but in recent times I am facing this error.

Also I am getting this warning at the beginning of the training.

/usr/local/lib/python3.10/dist-packages/torch/functional.py:641: UserWarning: stft with return_complex=False is deprecated. In a future pytorch release, stft will return complex tensors for all inputs, and return_complex=False will raise an error.
Note: you can still call torch.view_as_real on the complex output to recover the old return format. (Triggered internally at ../aten/src/ATen/native/SpectralOps.cpp:862.)
return _VF.stft(input, n_fft, hop_length, win_length, window, # type: ignore[attr-defined]

I am providing the screenshots of the error I encounter everytime.

To Reproduce

https://colab.research.google.com/drive/1k8Fk5kfU_aZ2lM7Esih3Ud1fYtNlujOQ?authuser=0#scrollTo=A49iDwajBtu_

I am using this colab notebook for training purpose. Every configuration regarding the training can be referred from here. Mind that training will go on for 35-40 epochs then it will stop.

Expected behavior

Training should continue.

Logs

No response

Environment

https://colab.research.google.com/drive/1k8Fk5kfU_aZ2lM7Esih3Ud1fYtNlujOQ?authuser=0#scrollTo=A49iDwajBtu_

Additional context

I have tried to resolve the warning and error both as I think both are related.
I tried following solutions to resolve the warning.
jaywalnut310/vits#15
and the following to solve the error.
#1949
Looks like Torch version == 1.8 is unstable and distribution is not available. I tried 1.9 too because github above prescribed it. Distribution not available.

[prakharpbuf]

checkout #1949 it might be useful

[offside609]

It was useful. But downgraded Torch version 1.8.0 is no longer stable and does not support GPU version. Please help.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. You might also look our discussion channels.

[mniiinm]

exactly same problem. any fix?

[mniiinm]

It seems that the problem is from mixed_precision, and if change it to False, the error will be solved

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. You might also look our discussion channels.

[aqez]

I'm also hitting this, anyone have any luck? I've tried no mixed precision but still get the issue.

Edit: I found my issue, it was actually my data.

[mniiinm]

I'm also hitting this, anyone have any luck? I've tried no mixed precision but still get the issue.

Edit: I found my issue, it was actually my data.

Can you tell about your data problem?

[mesut92]

I got same error. I am using python 3.10, torch 2.0 . What is the problem about data?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. You might also look our discussion channels.

[lucasjinreal]

Half precision issue, just add dim arg

[yijingshihenxiule]

Half precision issue, just add dim arg

@lucasjinreal Hello, could you please clarify? Need your help! Thank you!

[VafaKnm]

any update for this issue? I have same problem too