GPU temp over 90 while running peft_bnb_whisper_large_v2_training code in pycharm

[sneha117]

I decided to train the model on custom dataset using this code peft_bnb_whisper_large_v2_training. while training I noticed that the GPU temp rises beyond 90. This the first time it happened. So, I checked the hardware side. Fan is working. GPU is nvidia geforce rtx 3060. I have used this to successfully train whisper medium on the same dataset and GPU temp was never over 70. then why now? I immediately restarted the pc and run a training code for whisper medium no issues there.

[BenjaminBossan]

I don't think this is a question that we can answer for you. There is nothing specific in PEFT that should affect GPU temperature compared to other training workflows not involving PEFT. If this behavior is reproducible, my only guess is that for your task, using PEFT leads to better GPU utilization (e.g. less bottleneck for i/o) which would result in higher temperatures. Maybe you can monitor that.

As to how to solve the issue, I would recommend to search for general GPU temperature issues, e.g. to improve airflow. I don't believe there is anything in PEFT that we can do.

[sneha117]

Thank you for the immediate reply. As per your recommendation, I'll look in to it. The GPU temperature issues have already been checked and confirmed that the cooling systems are all working fine. Let me look in to any other factors that may contribute to the temperature issue.