training vram consumption

[adrianpuiu]

training 150 images / hour @ 123 epochs

[seruva19]

Hmm.. So 16 Gb is enough?

[adrianpuiu]

yes, its more than enough, and i'm doing them cropped at 768px .
idk what the guy did with those dreambooth and textual inversion scripts but the vram consumption was way to much .. i mean i could have run the textual inversion one but it was at 22 Gigs + constantly

[seruva19]

Interesting. Makes even more sense now to train prior and unclip parts sequentially.

[adrianpuiu]

yes this is what im doing.
1 by 1
btw , that above picture was for the unclip fp16 decoder , the prior training takes 11.7 Gigs.

[kodxana]

My benchmark on A6000 48 GB GPU:
For prio model I was able to get batch size 140 with 98% memoery usage.
For decoder model max batch size is 4 for 99% usage.

[seruva19]

Tested training on free tier colab, it works fine.
Batch size=1
Prior: ~ 10.8 Gb
Unclip: ~ 13.7 Gb

[adrianpuiu]

btw now that you mentioned colab . theyll need a way to keep the models saved onto the google drive so they eliminate the models downloading times .

[seruva19]

There was already a checkbox in colab notebook for this ;)
As for saving trained checkpoints, you just have to set up correct paths like /content/gdrive/MyDrive/images etc.
And (haven't tested it) perhaps it's possible to resume training later by using already trained checkpoint as source model.

[adrianpuiu]

@kodxana we need your help man . check this one out .. the dude has already a 60k image data set from MJ 5 .. maybe you could help the guy with the training