Please support gemma arch

[NeonBohdan]

It's a unique model
With 256K tokenizer like MT5 but decoder only
So hoping for good multi language capabilities comapared to llama tokenizer version

Hoping it will be easy enough(like llama -> mistal)
As I see this project is now harder to maintain
But better than llama.cpp or vllm in my opinion

https://huggingface.co/google/gemma-7b-it

Maybe this will help:
ggerganov/llama.cpp#5631
vllm-project/vllm#2960

[minhthuc2502]

Support Gemma soon with #1631

[ymoslem]

Hi @NeonBohdan Have you tried Gemma with CTranslate2? Does it generate the same output as Transformers? It seems it starts with a good generation, and then continues with repeated words. However, I might be missing something.

[NeonBohdan]

@ymoslem, I haven't compiled ctranslate2 to test it yet, and I'm waiting for the release.
However, there seems to be an issue with Gemma-it, compared to Mistral.
The situation strengthens with quantization.

You can try using a repetition penalty, but overall, I've observed this problem as well.

[ymoslem]

Thanks, @NeonBohdan for your response! I tried repetition_penalty but it does not seem to help.
I suspect it can be an issue with quantization. I will try without it and see.

[NeonBohdan]

@ymoslem for sure you want to left repetition_penalty=0

It's the most stable generation case(llama.cpp issues info)

The problem with quantization

[ymoslem]

Thanks @NeonBohdan! Are you aware of any quantization implementation that solves this issue, or is it something to do with the model itself?

[NeonBohdan]

@ymoslem read this

Right now Gemma isn't fully fixed
Two biggest problems

1. gelu vs apriximation
2. different dtype for different layer for best stability

So even at bfloat16 or float16 you may see problems (Maybe float16 less problematic, because of embedding scaling)

And if you add quantization for this model it may be critical to leave some layers unquantized (which ctranslate2 don't support)

---

So I don't really see a solution exept of waiting
Or trying:
vllm (without quantization may work better, with may not work)
llama.cpp (quantization works, but no beam search)

But I like ctranslate2 the most
It's the most production ready package

[ymoslem]

Thanks @NeonBohdan for the explanations.