RuntimeError: mat1 and mat2 shapes cannot be multiplied (1024x6 and 7x4096)

[Nikitala0014]

I have two NVIDIA A10 24GB. When I start the finetune.py script I recieve this error on batch size 24. Can you help me understand what is the real problem? Thank's.

[chiayisu]

I encountered this errror as well. However, in my case, after I added "os.environ["CUDA_VISIBLE_DEVICES"] = "1" to restrict to use only one gpu, the problem was resolved.

[Nikitala0014]

Thank you for your response. Indeed, this solved not only this problem, but for some reason only with this approach to launching on one gpu, 24GB of memory is enough for training.

[AngainorDev]

See this other - same - issue and answers

#8 (comment)

Training on multiple GPUs is possible with torchrun, you'll double batch size and half the training time.
Take care of veryfiying the math by hand so gradient accumulation steps is right.

[Nikitala0014]

See this other - same - issue and answers

#8 (comment)

Training on multiple GPUs is possible with torchrun, you'll double batch size and half the training time. Take care of veryfiying the math by hand so gradient accumulation steps is right.

Thank you. That's what I need ;)