I met a problem when I use torch=2.0.0, I can't use loftq config in loraConfig

[LiManyuan663]

when I use loftq config, I met the problem:

AttributeError Traceback (most recent call last)
Cell In[13], line 44
25 unet_loha_config = LoHaConfig(
26 r=args.rank,
27 alpha=args.rank,
(...)
32 # use_effective_conv2d=True,
33 )
35 unet_lokr_config = LoKrConfig(
36 r=args.rank,
37 alpha=args.rank,
(...)
42 # use_effective_conv2d=True,
43 )
---> 44 unet.add_adapter(unet_lora_config)
45 for p in unet.parameters():
46 print(p.shape,p.dtype)

File ~/miniconda3/envs/jit/lib/python3.9/site-packages/diffusers/loaders/peft.py:70, in PeftAdapterMixin.add_adapter(self, adapter_config, adapter_name)
67 # Unlike transformers, here we don't need to retrieve the name_or_path of the unet as the loading logic is
68 # handled by the load_lora_layers or LoraLoaderMixin. Therefore we set it to None here.
69 adapter_config.base_model_name_or_path = None
---> 70 inject_adapter_in_model(adapter_config, self, adapter_name)
71 self.set_adapter(adapter_name)
...
---> 88 values = values.sort().values
89 values /= values.max()
90 return values

AttributeError: 'tuple' object has no attribute 'values'

It seems that torch>=1.8 will meet this problem, writevalues,_ = values.sort().values can solve it, but I don't know how to fix my own code.

[younesbelkada]

Hi @LiManyuan663
Thanks for the report ! Can you share a simple reproducible snippet for us to easily repro the issue ? 🙏

[LiManyuan663]

this is my own code, I want to use 'loftq' config in loraConfig:

unet_lora_config = LoraConfig(
    r=args.rank,
    lora_alpha=args.rank,
    init_lora_weights="loftq",
    loftq_config = LoftQConfig(loftq_bits = 4),
    # use_rslora=True,
    use_dora=False,
    target_modules=["to_k", "to_q", "to_v", "to_out.0", "add_k_proj", "add_v_proj"],
    lora_dropout=0.01,
)
unet.add_adapter(unet_lora_config)

then it report an error：

in this file :peft/utils/loftq_utils.py:88, in NFQuantizer.create_normal_map(offset, symmetric, num_bits)

whatever I pass the parameters, I always meet this problem, It seems it occurs in line 88values = values.sort().values. It reported that AttributeError: 'tuple' object has no attribute 'values'

[LiManyuan663]

you can only test the method create_normal_map() on default mode, I used pytorch=2.0.0 then the problem occured.

[younesbelkada]

Thanks @LiManyuan663 !
Could you also share how do you initialize the Unet module?

[LiManyuan663]

OK！I use “jittor” to initialize it:

unet = JDiffusion.UNet2DConditionModel.from_pretrained(
    args.pretrained_model_name_or_path, subfolder="unet", revision=args.revision
)

it's simple.
then I addAdapter and so on...
it works well until I pass the config params into diffusers and peft, and your method NFQuantizer.create_normal_map also successfully get my parammeter num_bits=4 in my loraconfig, other parameters set to default.
then I asked ai,

I don't know if it is right.hope you can check it then.
Thaaaaaaaannnnks!

[BenjaminBossan]

Btw do you have bitsandbytes installed? If not, could you install it and try again?

[LiManyuan663]

Btw do you have bitsandbytes installed? If not, could you install it and try again?
yesyes, I have installed it. I am sad that it didn't work.

[BenjaminBossan]

yesyes, I have installed it. I am sad that it didn't work.

That's strange, because according to your screen shot, you reach this line:

peft/src/peft/utils/loftq_utils.py

Lines 209 to 210 in ad8f7cb

	if not is_bnb_4bit_available() or num_bits in [2, 8]:
	quantizer = NFQuantizer(num_bits=num_bits, device=device, method="normal", block_size=64)

But that should only happen if bitsandbytes is not installed or you're using 2 or 8 bits, both of which are not true. Any idea what could be happening? Could you maybe jump in with a debugger and check why the condition is met?

[LiManyuan663]

I am so sorry that I forgot I have installed bitsandbytes in current environment. I will try again later!
But why I meet the problem when I set num_bits=2?
Thank you for your patience in answering my questions!!

[BenjaminBossan]

But why I meet the problem when I set num_bits=2?

2 bit cannot use bitsandbytes (which has 4 and 8 bits), so this uses a custom LoftQ implementation. I would recommend to stick with bitsandbytes, i.e. use 4 bits. If you do that and install a recent version of bitsandbytes, you should hopefully not encounter any errors.

That said, there probably is indeed a bug in the custom NFQuantizer in the block you showed above.

[BenjaminBossan]

That said, there probably is indeed a bug in the custom NFQuantizer in the block you showed above.

Okay, I dug a bit deeper. The .sort method returns a namedtuple, which is why the code normally works. However, in older PyTorch versions, it returns a normal tuple, which seems to be happening for you. I could not identify the exact PyTorch version, but it seems like the namedtuple is returned at least from PyTorch v1.6.0 onwards. Is it possible that you're using a very old PyTorch version?

[LiManyuan663]

That said, there probably is indeed a bug in the custom NFQuantizer in the block you showed above.

Okay, I dug a bit deeper. The .sort method returns a namedtuple, which is why the code normally works. However, in older PyTorch versions, it returns a normal tuple, which seems to be happening for you. I could not identify the exact PyTorch version, but it seems like the namedtuple is returned at least from PyTorch v1.6.0 onwards. Is it possible that you're using a very old PyTorch version?

Thanks for your reply! I tried again today and it works well now!
I am so sorry for the trouble and thank you so much for your patience and outstanding work!
as for the tuple, I guess it's because I used jittor and it returns a tuple. It seems like Its compatibility is not so good now :(

[BenjaminBossan]

Great that it works now. I'll close the issue then but feel free to re-open if something new comes up.

As for jittor, it may be worth opening an issue there about the incorrect return type.