Using so-vits-svc-fork-4.0.ipynb: Unable to register factories. The TensorBoard runs some seconds and stops.

[Taizemn]

Describe the bug

As I reach the train step and I clicked onto the arrow button itself I get this from the tensor:

Reusing TensorBoard on port 6006 (pid 7397), started 0:07:57 ago. (Use '!kill 7397' to kill it.)
2023-11-10 16:22:14.699866: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:9342] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2023-11-10 16:22:14.699924: E tensorflow/compiler/xla/stream_executor/cuda/cuda_fft.cc:609] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2023-11-10 16:22:14.699952: E tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:1518] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2023-11-10 16:22:16.048258: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT

Even if I retry
%reload_ext tensorboard
%tensorboard --logdir drive/MyDrive/so-vits-svc-fork/logs/44k
!svc train --model-path drive/MyDrive/so-vits-svc-fork/logs/44k

or !kill 7397 and try again its not working.

I also payed for the Colab Pro so I'm kinda sad. Everything worked locally but here. :(

To Reproduce

Just make the steps: https://colab.research.google.com/github/voicepaw/so-vits-svc-fork/blob/main/notebooks/so-vits-svc-fork-4.0.ipynb

As I mentioned as I clicked to the Train arrow button I got:

2023-11-10 16:22:14.699866: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:9342] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2023-11-10 16:22:14.699924: E tensorflow/compiler/xla/stream_executor/cuda/cuda_fft.cc:609] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2023-11-10 16:22:14.699952: E tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:1518] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2023-11-10 16:22:16.048258: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
[16:22:16] INFO [16:22:16] NumExpr defaulting to 8 threads.

Additional context

The issue:

The drive has the necessary files:

As the virtual env:

Version

so-vits-svc-fork-4.0.ipynb

Platform

Windows 10

Code of Conduct

• I agree to follow this project's Code of Conduct.

No Duplicate

• I have checked existing issues to avoid duplicates.

[olaviinha]

I was experiencing similar behaviour a while ago with V100 GPU (training ran a few secs and stopped repeatedly), still after restarts etc. Only after I got A100 it started working again. Some days ago it worked fine with T4 too, so I'm wildly guessing perhaps V100+sovits does not compute. As said, wild guessing here, but maybe checking and switching GPU type is worth a shot if you are on a hosted runtime.