[BUG] OOM errors during fine-tuning with Polyglot 12.8B models

[yum-yeom]

🐛 Bug

hello. I'm trying to train Polyglot 12.8B model using h2o llm-studio, but I'm getting an OOM error.

The GPU I am using is Nvidia A5000 24GB 4 sheets, and the params I used are as follows zip file.

• Using GPU
  
• config & logs
  Uploading logs_polyglot-ko-12.8b_12.8b_koalpaca-v1.1a-row21155_v3_bs1_gas8.zip…

Is there any solution?

[pascal-pfeiffer]

There are multiple ways to bring down memory consumption.
Mainly:

• int4 training
• smaller batchsize
• lower token length for question and answer

There is also an ongoing effort to split the model weights across multiple GPUs during training in this PR: #288