Fixed LoRA conversion for kohya_ss

[kovalexal]

diffusers have recently replaced Conv2d and Linear layers with LoraCompatibleConv and LoraCompatibleLinear.

This pull request fixes wrong conversion of kohya_ss trained LoRAs. It should be better to determine whether module is Linear or Conv2d block based on class inheritance, not based on its name.

@BenjaminBossan @pacman100 your review is kindly appreciated.

[BenjaminBossan]

Thanks a lot, this looks good to me. Let's wait for @pacman100's review too, since he knows more about this.

@kovalexal do you have a (small) file at hand that can be loaded with your PR but fails without? Maybe we could use that for some sort of regression test.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

[kovalexal]

@BenjaminBossan hi!

I am not quite sure whether this case can be covered by some sort of test (also, from the perspective that it's just an example, not a full part of the library). I have run this script in an environment with one of the recent versions of diffusers and after conversion succeeded, I've discovered that the results seemed to differ from what I was getting in webui.

So, what I've found out during investigation is that now Linear and Conv layers are not processed properly and that 20-30% of LoRAs files were lost.

[BenjaminBossan]

I am not quite sure whether this case can be covered by some sort of test (also, from the perspective that it's just an example, not a full part of the library). I have run this script in an environment with one of the recent versions of diffusers and after conversion succeeded, I've discovered that the results seemed to differ from what I was getting in webui.

It would probably not be part of PEFT proper, but we could, e.g., run a weekly GH action to load a checkpoint and compare the results to some expected outcome, similar to what you describe. That way, we can catch these types of regressions automatically in the future.

[kovalexal]

@BenjaminBossan sounds like a great idea, I'll provide all the details soon!

Also, do you need my help in terms of modifying GH actions?

[BenjaminBossan]

I'll provide all the details soon!

Fantastic.

Also, do you need my help in terms of modifying GH actions?

As you wish. We can also add it in a separate PR later.

[kovalexal]

@BenjaminBossan hi!

Here is a sample notebook which shows the loss of weights size when converting with the old script and the results after fix.

It can be used as some sort of reference for writing test.

Would you mind if I'll work on this test in a separate PR later? I have some plans on heavily refactoring this script for proper support of other models and would like to do it in a separate PR.

[BenjaminBossan]

It can be used as some sort of reference for writing test.

Sounds good.

Would you mind if I'll work on this test in a separate PR later? I have some plans on heavily refactoring this script for proper support of other models and would like to do it in a separate PR.

Yeah, we can add it later.

Let me ping @pacman100 or @younesbelkada for a potential second review, but I think this should be good to go.

[younesbelkada]

Sounds great thanks! Can you maybe indicate through a comment at the beginning of the file the version of diffusers used? I would assume diffusers>=0.20.0?
If the feature is on main and not published yet we should maybe wait until the next release

[kovalexal]

@younesbelkada hi!

As far as I know, diffusers added these layers in version 0.19.0

This script remains backward compatible with older versions of diffusers. So, are you sure that this comment will be meaningful?

[younesbelkada]

Thanks for clarifying!