Community contribution: Adding GGUF support for more architectures

[SunMarc]

Feature request

Recently, we have added the ability to load gguf files within transformers.

The goal was to offer the possibility to users to further train/fine-tune their gguf models.

See Workflow

1) Load gguf file in transformers: we dequantize the weights to fp32, then we load the weights to be used with PyTorch.

from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF"
filename = "tinyllama-1.1b-chat-v1.0.Q6_K.gguf"

tokenizer = AutoTokenizer.from_pretrained(model_id, gguf_file=filename)
model = AutoModelForCausalLM.from_pretrained(model_id, gguf_file=filename)

2. train/finetune

3. Convert the model back to gguf to use in the ggml ecosystem using convert_hf_to_gguf script or using gguf-my-repo space if you pushed your model on the hub :

tokenizer.save_pretrained('directory')
model.save_pretrained('directory')

!python ${path_to_llama_cpp}/convert-hf-to-gguf.py ${directory}

Let's try to add GGUF support for more architectures! Currently supported architectures are

• Llama
• Mistral
• Qwen2

It would be great to add the support for more architectures such as

• Phi3 Add support for GGUF Phi-3 #31844
• Qwen2Moe Add Qwen2Moe GGUF loading support  #33264
• Gemma2
• T5 Add T5 GGUF loading support #33389
• Falcon Add falcon gguf #33437
• Bloom Add gguf support for bloom #33473
• Codestrall
• dbrx
• Deepseek (when it will be added to transformers)
• StableLM Add gguf support for StableLM #33793
• gpt2 Add gguf support for gpt2 #34044
• starcoder2 Add GGUF for starcoder2 #34094

... and many more (Feel free to suggest more architectures ! The model needs to integrated in transformers)

Adding this feature would require to follow the same protocol as in this PR :

1. Update GGUF_TENSOR_MAPPING and GGUF_CONFIG_MAPPING in order to map the tensor/config of the gguf file to the one on transformers.
2. Create a GGUFXXXConverter(XXXConverter) class to convert the gguf tokenizer to a transformers one.
3. Write tests

If you are interested to take up the challenge, comment below with the architecture name you want to integrate and open a PR!

Once you open a PR, feel free to ping @SunMarc @LysandreJik @ArthurZucker for a review !

Motivation

Support for more gguf models

Your contribution

Reviewing PRs and possibly adding the support for more models

[VladOS95-cyber]

@SunMarc I am going to take Qwen2Moe

[KingNish24]

@SunMarc I want to take Gemma2

[junejae]

@SunMarc May I suggest & take T5? Seems GGUF version of T5 encoder is highly used for getting along with FLUX.

[010kim]

@SunMarc Hello! Unless someone else is working on this model already, may I take MiniCPM-V?

[SunMarc]

@SunMarc May I suggest & take T5? Seems GGUF version of T5 encoder is highly used for getting along with FLUX.

Added @junejae !

@SunMarc Hello! Unless someone else is working on this model already, may I take MiniCPM-V?

Hi @010kim, thanks for the interest ! MiniCPM-V model relies on trust_remote_code=True, so I don't think we can add this model for now with gguf support. We don't want to have code in transformers that relies on modeling files that are on the hub. I will think about extending trust_remote_code=True to gguf support, so that the author of the model can add it himself !

[010kim]

Hi @010kim, thanks for the interest ! MiniCPM-V model relies on trust_remote_code=True, so I don't think we can add this model for now with gguf support. We don't want to have code in transformers that relies on modeling files that are on the hub. I will think about extending trust_remote_code=True to gguf support, so that the author of the model can add it himself !

@SunMarc Thank you so much for your response. It also makes sense the author should work on it. What about Cohere? Can I take it?

[jungnerd]

Hi @SunMarc 👋🏻
May I work with CLIP model if nobody is working on it?

[SunMarc]

Hey @jungnerd ! The model you choose needs to be in the conversion script from hf to gguf. See the script here

[g-prz]

Hey @SunMarc 🙋‍♂️
I'd like to try my chance to contribute to this issue, can I take Falcon? 🦅

[VladOS95-cyber]

Hi @SunMarc, I take bloom if nobody is working on it

[fabxoe]

Hi @SunMarc, I'd like to handle the work related to Codestrall :)

[jungnerd]

Hey @jungnerd ! The model you choose needs to be in the conversion script from hf to gguf. See the script here

There is conversion script for clip model(clip.cpp). Can I use this to contribute?

[cjfghk5697]

Hi @SunMarc,
I'm interested in this issue. Would it be okay if I worked on the BLIP model?

[cjfghk5697]

Hi @SunMarc, I'm interested in this issue. Would it be okay if I worked on the BLIP model?

Hi @SunMarc, I'd like to work on the BLIP model, but after researching, I found that it might be challenging due to the Vision model structure. Would it be alright if I switched to working on the Smol model instead?

[g-prz]

Hey @SunMarc 🤗
Gonna continue with granite 🪨

[cjfghk5697]

@SunMarc
I checked the Smol model and confirmed that it's already functioning well without needing any further work. In the issue mentioned that supporting the Smol model would be beneficial, but is there any specific work required?

If not, I’ll proceed with switching to the dbrx model.

[SunMarc]

@SunMarc
I checked the Smol model and confirmed that it's already functioning well without needing any further work. In the issue mentioned that supporting the Smol model would be beneficial, but is there any specific work required?

If not, I’ll proceed with switching to the dbrx model.

Oh indeed, this is because it is a llama architecture.

[VladOS95-cyber]

Hi @SunMarc! I am going to start working on StableLM model

[yijun-lee]

Is any work being done on the Gemma2? If not, I would like to proceed with it!
@SunMarc @KingNish24

[VladOS95-cyber]

Hi @SunMarc! I suppose GPT2 gguf is not supported yet, if this is a case, I'll take it

[fabxoe]

Hi @SunMarc, I'd like to handle the work related to Codestrall :)

Codestrall's tokenizer was just llama tokenizer.
It looks like I don't need to handle codes of Codestrall

[010kim]

@SunMarc Thank you so much for your response. It also makes sense the author should work on it. What about Cohere? Can I take it?

I went through the codes, and i was able to to load Cohere gguf model, but could not load the tokenizer. This is because Cohere slow tokenizer is not implemented in HuggingFace. (Only FastTokenizer is available for Cohere) Is there a way around to fix this? @SunMarc

[VladOS95-cyber]

Hey @SunMarc! I"ll take Starcoder2 as next model

[VladOS95-cyber]

Hi @SunMarc! I am going to start working on Mamba

[farrosalferro]

Are you still working on Gemma2? @yijun-lee @KingNish24 ? If not, is it possible for me to try working on it? Thank you!

[yijun-lee]

Are you still working on Gemma2? @yijun-lee @KingNish24 ? If not, is it possible for me to try working on it? Thank you!

I’m running behind schedule, but I’m making progress! I’ll handle it.

[farrosalferro]

I’m running behind schedule, but I’m making progress! I’ll handle it.

Glad to know! Then is it possible for me to try working on Nemotron? @SunMarc

[farrosalferro]

Could you please kindly check my PR @SunMarc? Thank you
Add Nemotron GGUF Loading Support