Run time Error

[yliu315]

Hello, I was reproducing your work on the leaves dataset.
The training was good at the beginning, but after around 700 epochs. The following RuntimeError occurs. Could you please help me with it? What could cause the problem?

Thank you very much.

Epoch 708
Traceback (most recent call last):
File "train.py", line 522, in
trainIters(args)
File "train.py", line 353, in trainIters
crits, optims, mode=split)
File "train.py", line 108, in runIter
c = args.iou_weight * softIoU(y_true_p, y_pred_i)
File "/home/yliu1/lstmInstanceSpain/rsis/src/utils/hungarian.py", line 83, in softIoU
den = (out+target-out*target).sum(1,True) + e
RuntimeError: CUDA out of memory. Tried to allocate 122.12 MiB (GPU 0; 11.91 GiB total capacity; 10.70 GiB already allocated; 73.06 MiB free; 234.17 MiB cached)