lora training

[nanaj96]

can we possibly lora training in comfyui with stable-fast?

In ComfyUI it is not possible. But if you use diffusers or any UI using it, I could demonstrate it for you.

[chengzeyi]

In ComfyUI this is hard to implement. But if you use diffusers or any UI backened by it and you are really interested in it, I could demonstrate it for you.

[L4Ph]

Really!
I hope they merge it into sd-script one of these days!

You could really start a revolution when you get there!

[chengzeyi]

I have just done some experiments. Sadly speedup is not significant. 🤔If it needs to be effective, I think I need to build a new compiler frontend with torch.fx and aot_autograd.

Current frontend with torch.jit lacks the ability to optimize backward graph and is incompatible with many features of accelerate, the widely used training acceleration framework. With torch.jit, we cannot explicitly capture the backward graph and wrapping the backward pass of an arbitrary model into a CUDA graph is also non-trivial🤔.

Currently the test results show that the benifit of stable-fast over torch.compile in LoRA training is small or even worse.

[chengzeyi]

And for training fusion is limited since a lot of intermediate buffers have to be generated and saved for calculating gradient during backward pass, so the speedup won't be as significant as pure inference🤔.

[chengzeyi]

Ok, after upgrading torch and refactoring stable-fast, it now works with aot_autograd! But, it is not optimized yet so the speed is not fast! No optimized backward kernels, no optimized decompositions...

https://github.com/chengzeyi/stable-fast/blob/main/examples/optimize_train_text_to_image_lora.sh

[L4Ph]

yupyupyupyup 👍👍👍👍👍👍👍👍👍👍👍👍👍👍👍👍👍👍