Model Isn't Learning

[ExponentialML]

Using Stable Diffusion 1.5 on torch 1.13.1, Cuda 11.6, and the latest version of xformers==0.0.16. I cannot build torch 1.12.1 on my machine.
The model won't learn. It simply looks like the first iteration after every epoch.

(0-500 all look like this)

[Ericxgao]

Same issue here, I'm using 3.9.12 with Torch @ 1.12.1, Cuda 11.6

[JulianJuaner]

Same issue. Whether using this repo or the official repo.
It seems the model is not updating during training.

[ExponentialML]

@Ericxgao and @JulianJuaner I found a fix that worked for me. You guys can give it a go and report back.

1. Install from requirements.txt. Torch version must be 1.12.1 on CUDA 11.3 or 11.6.
   pip install torch==1.12.1+cu116 torchvision==0.13.1+cu116 torchaudio==0.12.1 --extra-index-url https://download.pytorch.org/whl/cu116
2. Clone xformers: git clone https://github.com/facebookresearch/xformers
3. After cloning: cd xformers
4. Run :git reset --hard 0bad001ddd56c080524d37c84ff58d9cd030ebfd
5. git submodule update --init --recursive
6. pip install -e .

After install, try running the script.

[JulianJuaner]

@Ericxgao and @JulianJuaner I found a fix that worked for me. You guys can give it a go and report back.

1. Install from requirements.txt. Torch version must be 1.12.1 on CUDA 11.3.
   pip install torch==1.12.1+cu113 torchvision==0.13.1+cu113 torchaudio==0.12.1 --extra-index-url https://download.pytorch.org/whl/cu113
2. Clone xformers: git clone https://github.com/facebookresearch/xformers
3. After cloning: cd xformers
4. Run :git reset --hard 0bad001ddd56c080524d37c84ff58d9cd030ebfd
5. git submodule update --init --recursive
6. pip install -e .

After install, try running the script.

Thanks! It works for me. It seems the version of xformers is essential.

[Ericxgao]

Hmm I'm still having trouble getting this version of xformers installed. What GPU and python version are you @ExponentialML @JulianJuaner ? I'm using a cloud A100.

[ExponentialML]

@Ericxgao If you're using an A100, you should be able to fit the model in 40GB of vram when training, so xformers shouldn't be needed. Is this not the case?

[Ericxgao]

I still get OOM errors - I disabled Adam 8 bit as that was also failing on my system (bitsandbytes doesn't seem to install properly)

[mili-inch]

facebookresearch/xformers#631
This is probably caused by the same issue as this one, but it has been resolved and the model was successfully trained using xFormers in version 0.0.17.dev441.

[ExponentialML]

Closing with solutions from me & @mili-inch . If there are any other issues, feel free to ask for a re-open to discuss.