Add AutoGPTQ's cpu kernel.

[qwopqwop200]

Currently only supports 2,4 bits, desc_act does not.
Enabling use_qigen overrides all settings related to cuda.

import os

from transformers import AutoTokenizer, TextGenerationPipeline
from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig
import numpy as np
import torch
import torch.nn as nn

pretrained_model_dir = "facebook/opt-125m"
quantized_model_dir = "opt-125m-4bit-128g"

tokenizer = AutoTokenizer.from_pretrained(pretrained_model_dir, use_fast=True)
model = AutoGPTQForCausalLM.from_quantized(quantized_model_dir, use_qigen=True)
print(tokenizer.decode(model.generate(**tokenizer("test is", return_tensors="pt"),max_new_tokens=5)[0]))

opt-125m	Bits	group-size	Wikitext2
cuda	4	128	29.8402
QIgen	4	128	29.8416

[PanQiWei]

Hi, I just leave some reviews to make sure we can have a good shape for this pr, so that it can be easily extend and mantained in the future.

[PanQiWei]

I would suggest to make_quant_cpu first and then load checkpoint. And of course this requires a futher modification in make_quant_cpu(), will describe it on there.

[PanQiWei]

TBH, I'm not fond of the way it make quant here, basically it load checkpoint directly when init QuantLinear, which breaks the API format to be different with other QuantLinear, and may cause difficulty of maintainance in the future.

My suggestions is only replace module in make_quant functions and load state dicts after this function is called in from_quantized()

[PanQiWei]

At least those five arguments' name and order should be same with other QuantLinear: bits, group_size, infeatures, outfeatures, bias

[qwopqwop200]

My code is complete. Please review again.

[fxmarty]

@qwopqwop200 Did not look at the code but given that CPUs usually do not support fp16 compute, is the compute rather done in fp32? With dequantization e.g. int4 -> fp32?

[qwopqwop200]

Probably so. But I didn't design this kernel, so I'm not sure how it works.

[fxmarty]

Where does it come from?

[qwopqwop200]

Where does it come from?

https://github.com/IST-DASLab/QIGen/tree/master

[qwopqwop200]

oh sorry I merged them by mistake. I've checked that it's working now, so there shouldn't be any problems.