GPTQ Integration

[SunMarc]

What does this PR do ?

This PR adds the possibility to train lora + adalora adapters on top of GPTQ quantized model.

convert to peft model for training

causal_lm_model_id = "marcsun13/opt-350m-gptq-4bit"
tokenizer  = AutoTokenizer.from_pretrained(causal_lm_model_id)
model = AutoModelForCausalLM.from_pretrained(
    causal_lm_model_id,
    torch_dtype=torch.float16,
    device_map="auto"
)
model = prepare_model_for_kbit_training(model)
config = LoraConfig(
      r=16,
      lora_alpha=32,
      target_modules=["q_proj", "v_proj"],
      lora_dropout=0.05,
      bias="none",
      task_type="CAUSAL_LM",
)
model = get_peft_model(model, config)

save adapters after training

model.cpu().save_pretrained(save_dir)

load saved adapters

model = AutoModelForCausalLM.from_pretrained(
    causal_lm_model_id,
    torch_dtype=torch.float16,
    device_map="auto"
)
model = PeftModel.from_pretrained(model ,save_dir)# load saved adapters

to do

• finetune llama2
• merge after transformers PR (the doc PR test will then be fixed)

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[pacman100]

Hello @SunMarc, Thank you for adding AutoGPTQ support 🚀. Left few comments

[BenjaminBossan]

Thanks a lot. This PR looks really good, I only have minor comments. I don't have any experience with GPTQ itself (yet), so I cannot really judge the more technical parts of the implementation.

A more general question: Does GPTQ generally not work with IA³ or is it just a matter of implementing it later?

[younesbelkada]

Looking great, thanks a lot @SunMarc ! I left one comment
We can add docs and an example script later in a follow up PR
Thanks! 🚀

[BenjaminBossan]

The PR is looking already quite good from my POV. Unfortunately, the tests are still failing, I assume because they require the corresponding changes to land in transformers.

I have a few comments, but none of them are deal breakers.

[BenjaminBossan]

This is an interesting deviation from how the other lora layers are implemented. Here, we pass the original layer (quant_linear_module) and use it under the hood. For the normal Linear lora layer, we don't get the layer, instead basically creating a new linear layer:

# in __init__
nn.Linear.__init__(self, in_features, out_features, **kwargs)
# in forward
result = F.linear(x, transpose(self.weight, self.fan_in_fan_out), bias=self.bias)

I actually prefer the solution here but wonder if there was a specific reason why this approach was not taken originally. If so, would that same reason apply here or are we good with having two different approaches? Hopefully, the others can clarify this.

[SunMarc]

I did that because I wanted to put the new QuantLinear class in the same place as the others for conformity. If we want to go with the same approach, we will need to create this new class in a function so that the auto_gptq import is protected ( to avoid circular import as it is also importing peft ) . LMK what you think about this solution and I will add it in another PR.

[SunMarc]

Unfortunately, the tests are still failing, I assume because they require the corresponding changes to land in transformers.

I don't want to break the tests so i hardcoded the const value. I will change them back when we will have the next release of transformers