Computer crashing/freezing a few epochs in for 3d fullres + 3d fullres cascade

[thegooby]

Hello. I've been having generally good success when using nnUnet. However, I've run into an issue with 3d fullres and 3d fullres cascade on some folds of my five-fold split. I've had the issue both with the current v2 and previous v1 versions of the tool. One or two of the subsequent cascade folds ran from start to finish with no issues, 2 requires several restarts, and I couldn't get the final one to run. For the 3d fullres on the v2, I was able to do the folds=all without issue on the first try but I cannot get any of the individual folds to work.

So, what happens is on one of the first few epochs, the computer freezes without any error popping up (I view it through AnyDesk instead of having it physically connected to a monitor and use a dummy HDMI insert to keep it running fine), and then the cooling fans start blowing REALLY loudly. I had the same behavior from the fans when my computer didn't recognize my GPU (I had to reinstall CUDA).

I am using a 4090. Additionally, I've attached screenshots from a screen recording I took of attempting to run a 3d fullres fold. The screenshots are from after the computer has frozen. I do not have anything else noteworthy running on the device.

I was not able to find anyone else running into a similar issue. I find it odd that it is not a consistent issue (such as for the cascade attempts) and that the folds=all worked fine while I cannot get a normal fold to run for 3d fullres in v2.

Additionally, the 3d lowres is taking about 20% more time to run than 3d fullres or 3d fullres cascade, both on v1 and v2.

Any suggestions to troubleshoot this would be very welcome. Thank you!

edit: it also just crashed when running for 3d lowres on v2.

edit2: i've tried having the monitor directly connected. What happens is the screen goes black a second or two before the fan gets really loud, and then the "no input device detected" blue screen shows up on my monitor.

edit3: I don't think it's ram, either. Here is the last frame before it froze again on fold 1 of this training, but it was working fine on fold 3 when I checked to see how far that would get.

[FabianIsensee]

Yeah so that's what defective hardware looks like. Not sure which component this is, but something is broken on your machine.
Could be unstable CPU overclock, thermal issues (CPU/GPU) or something else

[thegooby]

That's unfortunate. I've looked into others with a 4090 that have run into similar issues and one person fixed it by going from 4x16GB RAM to 2x32. Before I try that, I'll also look into the CPU issue you mentioned. Thank you!

[FabianIsensee]

RAM could be fault so changing RAM can fix the problem. I have tested a 4090 in the meantime and it is stable for me (2x16GB)

[FabianIsensee]

fyi occupying 4 RAM slots makes it more difficult for the IMC to keep up with XMP/RAMP and a possible solution could be to dial back timings & clocks. If you don't know what all of this is I suggest to contact your IT department

[thegooby]

Sorry for not updating the post. I was able to get it to work with 4x16GB by updating to CUDA 12.1 (easier said than done) and messing around with some pytorch versions. The most recent versions of all of the packages have greatly improved training speed (from 111s in the first picture to 45s that it currently takes).

[FabianIsensee]

Yeah a lot is happening which is great. nnunet now also supports pytorch compile for another 30% free speed boost. use export nnUNet_compile=True to use it