LoftQ: Add LoftQ method integrated into LoRA. Add example code for LoftQ usage. #1150

yxli2123 · 2023-11-19T18:17:19Z

I have added the LoftQ method to LoRA. I have run the make style command.

HuggingFaceDocBuilderDev · 2023-11-20T09:51:41Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

BenjaminBossan · 2023-11-20T11:25:54Z

examples/loftq_finetuning/train_gsm8k_llama.py

+
+def extract_answer_number(sentence: str) -> float:
+    sentence = sentence.replace(",", "")
+    pred = [s for s in re.findall(r"-?\d+\.?\d*", sentence)]


Apparently, ruff complains that [s for s in re.findall(r"-?\d+\.?\d*", sentence)] should be list(re.findall(r"-?\d+\.?\d*", sentence)). Fine I guess, but when I tried, findall already returns a list, so neither should be necessary.

What you could do on top is pre-compile the regex. So something like

PATTERN_NUMBER = re.compile(r"-?\d+\.?\d*") def extract_answer_number(sentence: str) -> float: sentence = sentence.replace(",", "") pred = PATTERN_NUMBER.findall(sentence) ... pred_answer = PATTERN_NUMBER.findall(segment[1]) ...

This should be a bit faster and avoid repeating the same regex.

Thanks for your suggestion. I have edited it. Let me know if there is still something I need to do.

change peft model path on HF auto float16/32 for bnb

yxli2123 · 2023-11-21T15:29:21Z

I failed the test because scipy is not installed. Is it possible to add scipy package to the dependency? I may need this package to implement norm.ppf function.

BenjaminBossan · 2023-11-22T14:16:37Z

I failed the test because scipy is not installed. Is it possible to add scipy package to the dependency? I may need this package to implement norm.ppf function.

Ah, too bad that this isn't implemented in PyTorch. I would suggest not to add scipy as a standard dependency, as we want to keep those lean. But we can add it as a dev dependency and use a local import of scipy.stats.norm inside of create_normal_map.
On top of that, let's alert users who want to use LoftQ as early as possible if they don't have scipy installed. For this, we can check in the __post_init__ of LoraConfig that scipy is installed if self.init_lora_weights == "loftq".

Alternatively, we could add our own norm.ppf function to PEFT. I checked the scipy code and it's unfortunately not straightforward to just copy it. It seems to be based on special.ndtri, which PyTorch also implements, but I'd have to brush up my math to check how to do this correctly. At least on surface, it seems possible:

(ndtri leads to many NaNs though)

WDYT?

pacman100

Thank you @yxli2123 for super cool work wrt adding LoftQ initialization method for LoRA adapters! 🤩

wrt scipy error, we can overcome by importing from peft.utils.loftq_utils import loftq_init in the def loftq_init(self, adapter_name) method wherein we first raise and error if scipy is not installed.

Apart from that, left a nit suggestion and a question regarding loftq_fake.

examples/loftq_finetuning/README.md

pacman100 · 2023-11-24T10:49:56Z

src/peft/tuners/lora/config.py

+    )
+    loftq_bits: int = field(default=4, metadata={"help": "Quantization bits for LoftQ"})
+    loftq_iter: int = field(default=1, metadata={"help": "Alternating iterations for LoftQ"})
+    loftq_fake: bool = field(


This isn't used anywhere in loftq_utils.py, is it?

The loftq_bits and loftq_iter are used. loftq_fake and bits_pattern are not used, but could be the future feature. Anyway, I have deleted the loftq_fake and bits_pattern.

Co-authored-by: Sourab Mangrulkar <13534540+pacman100@users.noreply.github.com>

yxli2123 · 2023-11-28T14:12:27Z

Hi, thanks for providing constructive suggestions. I have edited the code. Could you review the code and run test?

BenjaminBossan

Thanks a lot, I think we're almost finished with the PR.

Could you please run make style so that CI passes? Also, I found a few minor issues, please check my comments. Finally, the license snippet is missing on some files, would you please be so kind to add it? Thanks!

BenjaminBossan · 2023-11-28T14:31:04Z

src/peft/utils/loftq_utils.py

+try:
+    from scipy.stats import norm
+except ImportError:
+    raise ImportError("The required package 'scipy' is not installed. Please install it to continue.")


Could you please import norm inside of create_normal_map, just so that we can be super safe that we don't accidentally introduce a dependency on it? Also, inside of __post_init__ of LoraConfig, let's check that we can successfully import scipy if self.init_lora_weights == "loftq". That way, we can fail as early as possible.

src/peft/utils/loftq_utils.py

BenjaminBossan · 2023-11-28T14:34:35Z

src/peft/utils/loftq_utils.py

+        values /= values.max()
+        # print(values)
+        return values
+        # assert values.


BenjaminBossan · 2023-11-28T14:35:24Z

src/peft/utils/loftq_utils.py

+        return weight
+
+    def quantize_block(self, weight):
+        assert len(weight.shape) == 2 and weight.shape[0] * weight.shape[1] % self.block_size == 0


Ideally, we could replace this assert and the ones below by a proper ValueError, including a helpful error message for users.

examples/loftq_finetuning/README.md

Co-authored-by: Benjamin Bossan <BenjaminBossan@users.noreply.github.com>

…sg, edit example docs

yxli2123 · 2023-11-28T15:48:29Z

Thanks again. I have added license to loftq_utils.py, changed assert style to if ... raise ValueError, removed unnecessary print, moved the scipy check to __post_init__, and run make style.

BenjaminBossan

Thanks a lot for addressing the remaining comments. This is a great PR and a very useful feature to have in PEFT.

For me, the PR is in a state that it can be merged. I'll check if we want to have another final review by a colleague. For the future, we also should add documentation and unit tests. If no one else wants to work on those, I'll try to find some time next week to tackle that.

pacman100

Great job @yxli2123 on adding LoftQ and addressing all the comments! 🔥🚀✨

BenjaminBossan · 2023-11-29T11:43:28Z

Ah, sorry @yxli2123 I merged #1189, which created some merge conflicts, I should have merged this PR first. Would you be so kind to fix them? Otherwise, LMK if I should push the fixes on top of your PR.

yxli2123 · 2023-11-29T15:34:17Z

I have edited my branch with the conflicts. It should be fine to just accept my branch when conflicts happen.

src/peft/tuners/lora/config.py

BenjaminBossan · 2023-11-29T16:08:47Z

Thanks a lot @yxli2123, great PR and super cool feature, I hope it will find a lot of adoption.

yxli2123 · 2023-11-29T16:28:09Z

Thank you! We are glad to contribute to this wonderful open-source project.

--------- Co-authored-by: Sourab Mangrulkar <13534540+pacman100@users.noreply.github.com> Co-authored-by: Benjamin Bossan <BenjaminBossan@users.noreply.github.com>

BenjaminBossan · 2023-12-04T15:31:29Z

Hi @yxli2123 I think I discovered a small bug. When I try to use LoftQConfig(loftq_bits=8), I get an issue in the following lines:

peft/src/peft/utils/loftq_utils.py

Lines 201 to 216 in 1b1091c

    
           if not is_bnb_4bit_available(): 
        
               quantizer = NFQuantizer(num_bits=num_bits, device=device, method="normal", block_size=64) 
        
           weight = weight.to(torch.float32) 
        
           res = weight.clone() 
        
           for i in range(num_iter): 
        
               torch.cuda.empty_cache() 
        
               # Quantization 
        
               if num_bits == 4 and is_bnb_4bit_available(): 
        
                   qweight = bnb.nn.Params4bit( 
        
                       res.to("cpu"), requires_grad=False, compress_statistics=False, quant_type="nf4" 
        
                   ).to(device) 
        
                   dequantized_weight = bnb.functional.dequantize_4bit(qweight.data, qweight.quant_state) 
        
               else: 
        
                   quantized_weight, max_abs, shape = quantizer.quantize_block(res) 
        
                   dequantized_weight = quantizer.dequantize_block(quantized_weight, max_abs, shape)

Since is_bnb_4bit_available() is True, quantizer is undefined. However, the first check for num_bits == 4 is False, so we go to the else condition, which tries to use quantizer, resulting in a NameError. Since there is no bnb.functional.dequantize_8bit, we could proceed similar to how we do when merging LoRA bnb 8bit. Or could we use the NFQuantizer for 8 bit?

The same error would occur for 2 bit, but I don't know how relevant that is right now.

Another error (lower priority) that I encountered was when trying to pass a model that is already quantized. Of course, this is incorrect usage and we should document that. But maybe we can raise a nice error message when we encountered that, as right now, at first it appears that everything works and we only get an error during the forward pass.

Edit: Ran one test and NFQuantizer seems to work with 8bit.

Add GPU tests for LoftQ with 4bit quantization. Notes Tests for 8bit quantization are already there but not run at the moment, see this comment: huggingface#1150 (comment) In my testing, 8bit passes when using NFQuantizer, so if the original author is fine with using that, I can make the adjustment.

yxli2123 · 2023-12-08T15:41:02Z

Hi @BenjaminBossan, thanks for pointing it out. As for 8bit, LoftQ doesn't have advantages over QLoRA, so we leave 8bit for experimental programs. However, if it leads to unwanted errors, we could remove 8bit. As for 2bit, since bitsandbytes hasn't supported 2bit yet, we will use Linear4bit as fake replacement. We are working on real 2bit Linear class.

BenjaminBossan · 2023-12-08T16:28:18Z

Thanks for commenting on this. What would you think about enabling 8bit with the NFQuantizer, even if you didn't find it to work so well empirically? Maybe some users find more success. And it's better than raising an error :) We could add a note on the docs that 4bit is recommended. Regarding 2bit, we could raise an error for the time being with the message that 2bit is not yet supported but might be in the future.

Edit: Solves in #1276.

Add GPU tests for LoftQ with 4bit quantization. Notes Tests for 8bit quantization are already there but not run at the moment, see this comment: #1150 (comment) In my testing, 8bit passes when using NFQuantizer, so if the original author is fine with using that, I can make the adjustment. --------- Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com>

Add LoftQ method ingrated into LoRA. Add example code for LoftQ usage.

25f015d

BenjaminBossan reviewed Nov 20, 2023

View reviewed changes

yxli2123 added 4 commits November 20, 2023 09:26

faster regex

667529a

add scipy to requirements

86ba774

change to nf4

d01ccfd

to float32 when svd

e04fab2

change peft model path on HF auto float16/32 for bnb

BenjaminBossan mentioned this pull request Nov 21, 2023

saftetensor in subfolder not supported #1154

Closed

4 tasks

pacman100 reviewed Nov 24, 2023

View reviewed changes

yxli2123 and others added 2 commits November 26, 2023 11:37

Update examples/loftq_finetuning/README.md

6aba235

Co-authored-by: Sourab Mangrulkar <13534540+pacman100@users.noreply.github.com>

import scipy at loftq_init

d4523c8

Merge branch 'huggingface:main' into loftq

bcde4c3

BenjaminBossan requested changes Nov 28, 2023

View reviewed changes

yxli2123 and others added 2 commits November 28, 2023 10:21

Update examples/loftq_finetuning/README.md

fe9e84a

Co-authored-by: Benjamin Bossan <BenjaminBossan@users.noreply.github.com>

add license, import scipy at post_init, change assert to ValueError m…

cf68839

…sg, edit example docs

BenjaminBossan approved these changes Nov 29, 2023

View reviewed changes

pacman100 approved these changes Nov 29, 2023

View reviewed changes

BenjaminBossan mentioned this pull request Nov 29, 2023

DOC: List of TODOs for the documentation #1089

Closed

10 tasks

fix merge conflicts

88a90c6

BenjaminBossan reviewed Nov 29, 2023

View reviewed changes

src/peft/tuners/lora/config.py Show resolved Hide resolved

add loftq to Literal

aaff315

BenjaminBossan merged commit 2b901ee into huggingface:main Nov 29, 2023
14 checks passed

This was referenced Dec 4, 2023

TST: Add tests for 4bit LoftQ #1208

Merged

[LoftQConfig + LoraConfig] throws size matmul mismatch error #1240

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

LoftQ: Add LoftQ method integrated into LoRA. Add example code for LoftQ usage. #1150

LoftQ: Add LoftQ method integrated into LoRA. Add example code for LoftQ usage. #1150

yxli2123 commented Nov 19, 2023

HuggingFaceDocBuilderDev commented Nov 20, 2023

BenjaminBossan Nov 20, 2023

yxli2123 Nov 20, 2023

yxli2123 commented Nov 21, 2023 •

edited

Loading

BenjaminBossan commented Nov 22, 2023

pacman100 left a comment •

edited

Loading

pacman100 Nov 24, 2023

yxli2123 Nov 26, 2023

yxli2123 commented Nov 28, 2023

BenjaminBossan left a comment

BenjaminBossan Nov 28, 2023

BenjaminBossan Nov 28, 2023

BenjaminBossan Nov 28, 2023

yxli2123 commented Nov 28, 2023

BenjaminBossan left a comment

pacman100 left a comment

BenjaminBossan commented Nov 29, 2023

yxli2123 commented Nov 29, 2023

BenjaminBossan commented Nov 29, 2023

yxli2123 commented Nov 29, 2023

BenjaminBossan commented Dec 4, 2023 •

edited

Loading

yxli2123 commented Dec 8, 2023

BenjaminBossan commented Dec 8, 2023 •

edited

Loading

LoftQ: Add LoftQ method integrated into LoRA. Add example code for LoftQ usage. #1150

LoftQ: Add LoftQ method integrated into LoRA. Add example code for LoftQ usage. #1150

Conversation

yxli2123 commented Nov 19, 2023

HuggingFaceDocBuilderDev commented Nov 20, 2023

BenjaminBossan Nov 20, 2023

Choose a reason for hiding this comment

yxli2123 Nov 20, 2023

Choose a reason for hiding this comment

yxli2123 commented Nov 21, 2023 • edited Loading

BenjaminBossan commented Nov 22, 2023

pacman100 left a comment • edited Loading

Choose a reason for hiding this comment

pacman100 Nov 24, 2023

Choose a reason for hiding this comment

yxli2123 Nov 26, 2023

Choose a reason for hiding this comment

yxli2123 commented Nov 28, 2023

BenjaminBossan left a comment

Choose a reason for hiding this comment

BenjaminBossan Nov 28, 2023

Choose a reason for hiding this comment

BenjaminBossan Nov 28, 2023

Choose a reason for hiding this comment

BenjaminBossan Nov 28, 2023

Choose a reason for hiding this comment

yxli2123 commented Nov 28, 2023

BenjaminBossan left a comment

Choose a reason for hiding this comment

pacman100 left a comment

Choose a reason for hiding this comment

BenjaminBossan commented Nov 29, 2023

yxli2123 commented Nov 29, 2023

BenjaminBossan commented Nov 29, 2023

yxli2123 commented Nov 29, 2023

BenjaminBossan commented Dec 4, 2023 • edited Loading

yxli2123 commented Dec 8, 2023

BenjaminBossan commented Dec 8, 2023 • edited Loading

yxli2123 commented Nov 21, 2023 •

edited

Loading

pacman100 left a comment •

edited

Loading

BenjaminBossan commented Dec 4, 2023 •

edited

Loading

BenjaminBossan commented Dec 8, 2023 •

edited

Loading