fix cpu bnb path

[jiqing-feng]

Hi @Rocketknight1 @ArthurZucker @SunMarc @gante

We have "cpu" in hf_device_map when using bnb model in CPU. The cpu device bnb model should be accepted by transformers because CPU backend has been enabled in BNB. Please take a review, thx!

[Rocketknight1]

Looks good, but do you have an example of code that failed before, that is fixed by this PR?

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[jiqing-feng]

Looks good, but do you have an example of code that failed before, that is fixed by this PR?

I run this case in a CPU-only device

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

model_id = "Felladrin/Llama-68M-Chat-v1"

text = ["I am happy because", "This is"]
tokenizer = AutoTokenizer.from_pretrained(model_id)
tokenizer.pad_token = tokenizer.eos_token
input_ids = tokenizer(text, return_tensors="pt", padding=True)

quantization_config = BitsAndBytesConfig(load_in_8bit=True)

model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto", quantization_config=quantization_config)
model.generation_config.cache_implementation = "static"
model.generate(**input_ids)

Error: IndexError: list index out of range

[jiqing-feng]

Hi @Rocketknight1 . The script can reproduce this error easily both on AWQ and BNB. Besides, it's not safe to get index without checking the list length.

[jiqing-feng]

Hi @Rocketknight1 @ArthurZucker @SunMarc @gante @zucchini-nlp , do you mind reviewing this change? Thanks

[jiqing-feng]

Hi @Titus-von-Koeller . This change is needed for bitsandbytes cpu path, can you help to review it?

Besides, it's also needed for AWQ cpu path which is already enabled here: #33460

[jiqing-feng]

The following script can reproduce the AWQ error:
I run this case on a CPU-only device.

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig, AwqConfig

model_id = "PrunaAI/JackFram-llama-68m-AWQ-4bit-smashed"

text = ["I am happy because", "This is"]
tokenizer = AutoTokenizer.from_pretrained(model_id)
tokenizer.pad_token = tokenizer.eos_token
input_ids = tokenizer(text, return_tensors="pt", padding=True)

# quantization_config = BitsAndBytesConfig(load_in_8bit=True)
quantization_config = AwqConfig(version="ipex")

model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto", quantization_config=quantization_config)
model.generation_config.cache_implementation = "static"
model.generate(**input_ids)

[jiqing-feng]

Hi @aymeric-roucher @LysandreJik , do you have time to review this bug fix? Thanks!

[zucchini-nlp]

@jiqing-feng sorry for late reply, the transformers team was off last week and someone will review the PR soon. Seems like you tagged all relevant people already

[SunMarc]

Thanks for the fix ! I've suggested a similar fix that follows more closely what we have in accelerate. LMK if this works for you !

[jiqing-feng]

Hi @SunMarc , I have applied your changes, thanks!

[SunMarc]

Nice, thanks !

[ArthurZucker]

LGTM 🤗