Fine-tuning on multiple GPUs

[spacemonqi]

Since fine-tuning the larger models is of course very resource intensive, I was wondering how to fine-tune Whisper using a setup with multiple GPU's? The fine-tuning method I intend to use is the one in #64. I assume that it should be easy enough since Whisper was of course trained originally on a multi GPU setup.

[Lilboat-Wohoo]

Current Pytorch version should automatically enable data parallel, it will use every one of your gpu on your machine and train with the batch size you defined in training arguments. So if your batch size in training arguments is 1 and you have 4 gpus, then your actual batch size is 4.

But if you are referinng to training with multiple gpus across different machines, pls read passage about dpp from pytorch and refer to following passage to achieve it in code level:https://towardsdatascience.com/how-to-scale-training-on-multiple-gpus-dae1041f49d2

Good luck! :)

[Brenttonst]

Hi, I'm currently trying to do the same. Can I know if you were able to set up fine-tuning to run on multiple GPUs? Thanks c:

[pkoli437]

I tried fine tuning whisper small model on 4 core T4 GPU . But it was using only one core . When I tried to use Data parallel to enable the model to use all available GPUs on system it didn't work . Why is it ? Is Fine tuning Whisper Small Model on Multi GPU possible??