Does llama.cpp support gpt-2-q4?

[realcarlos]

I noticed ggml could run gpt-2 , I wonder if llama.cpp support too, I have download gpt-2-q4 model from huggingface ,but failed to run.

[realcarlos]

I tried to convert Cerebras-GPT-111M for llama.cpp, it could run with ggml , but I got error like below

❯ ./quantize ./models/Cerebras-GPT-111M/ggml-model-f16.bin ./models/output/cerebras-q4-0.bin q4_0
llama.cpp: loading model from ./models/Cerebras-GPT-111M/ggml-model-f16.bin
llama_model_quantize: failed to quantize: missing tok_embeddings.weight
main: failed to quantize model from './models/Cerebras-GPT-111M/ggml-model-f16.bin'

[realcarlos]

❯ python3 convert.py ./models/Cerebras-GPT-111M/
Loading model file models/Cerebras-GPT-111M/ggml-model-f16.bin
Traceback (most recent call last):
File "/Users/bytedance/llama.cpp/convert.py", line 1149, in
main()
File "/Users/bytedance/llama.cpp/convert.py", line 1137, in main
vocab = load_vocab(vocab_dir)
File "/Users/bytedance/llama.cpp/convert.py", line 1078, in load_vocab
raise FileNotFoundError(f"Could not find tokenizer.model in {path} or its parent; if it's in another directory, pass the directory as --vocab-dir")
FileNotFoundError: Could not find tokenizer.model in models/Cerebras-GPT-111M or its parent; if it's in another directory, pass the directory as --vocab-dir

[realcarlos]

this is the screenshot of "models/Cerebras-GPT-111M"

[LostRuins]

Cerebras is a GPT2 based model, not llama, and won't work with this repo.

You can run it with the gpt2 example main.cpp example in the ggml repo, or via KoboldCpp which automatically detects format.

(Disclaimer: I am a koboldcpp dev)

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.