Steps to support the Dolly model

[devkral]

What

• The current glob for model files is very restricted, I relaxed it a little so it could find the dolly model.
• support for ByteStorage was added. As far as I can see it is uint8.
• new: PretrainedVocab is added

Why

I want to use the dolly model with llama.cpp. They use some ByteStorage stuff of torch

Remaining issues

The vocab file is in a completely different format to sentencepiece (it uses a pretrained tokenizer):

Somehow it must get converted or another Vocab must be added to convert.py

• dolly uses a gpt_neox format which is different from what llama.cpp understands. Needs conversion

[j-f1]

Suggested change

	return f"<SentencePieceVocab with {self.vocab_size_base} base tokens and {len(self.added_tokens_list)} added tokens>"
	return f"<PretrainedVocab with {self.vocab_size_base} base tokens and {len(self.added_tokens_list)} added tokens>"

[ggerganov]

For a GPT-NeoX implementation using ggml see the StableLM example.
It will require some extra work to integrate this in llama.cpp.

But before doing that, we need to add a ggml example for Dolly and make sure that it works correctly.

[devkral]

nice, sry for ghost posting but why llama.cpp exists if you have the ggml repo?

I am a beginner in terms of AI.

[mverrilli]

Hi @ggerganov

But before doing that, we need to add a ggml example for Dolly and make sure that it works correctly.

I created this one. I can PR it if it is ok.

https://github.com/mverrilli/ggml/tree/dolly-v2/examples/dolly-v2

[ggerganov]

@devkral

Yes, please open a PR.
In your experience, do the ggml results look OK when you compare to the reference Python implementation?
I'll probably do some more rigorous testing later, but would like to get an additional opinion.

[mverrilli]

@ggerganov ggml-org/ggml#132

It is pretty comparable. The Q5_0 is significantly faster for the larger model. I was not getting good results until I added the special token handling since it would split the special token into two. I posted some sample runs in the README.

[j-f1]

nice, sry for ghost posting but why llama.cpp exists if you have the ggml repo?

I am a beginner in terms of AI.

GGML is a general purpose matrix API that doesn’t include support for running specific models directly (I believe). This repo exists to use GGML to implement the specific structures of LLaMA.

[devkral]

The ggml dolly example results look good.

[xingchensong]

Hi teams， any update？

[devkral]

wrong repository

[xingchensong]

wrong repository

is ggml the right one？