Where to Get the Tokenizer When Converting GPT4ALL

[reynolds087]

I'm trying to follow the instructions for GPT4ALL.
Here is the first line of code it says to run:

python3 convert-gpt4all-to-ggml.py models/gpt4all-7B/gpt4all-lora-quantized.bin ./models/tokenizer.model

I am getting an error because of the "tokenizer.model" which does not exist in this repository in the specified location, as far as I can tell. It also does not come with the model if you download it from nomic-ai's github. Where is this file supposed to originate from?

Traceback (most recent call last):
File "E:\Programs\llama.cpp\convert-gpt4all-to-ggml.py", line 107, in
main()
File "E:\Programs\llama.cpp\convert-gpt4all-to-ggml.py", line 102, in main
tokenizer = SentencePieceProcessor(args.tokenizer_model)
File "C:\Temp\WPy64-31001\python-3.10.0.amd64\lib\site-packages\sentencepiece_init_.py", line 447, in Init
self.Load(model_file=model_file, model_proto=model_proto)
File "C:\Temp\WPy64-31001\python-3.10.0.amd64\lib\site-packages\sentencepiece_init_.py", line 905, in Load
return self.LoadFromFile(model_file)
File "C:\Temp\WPy64-31001\python-3.10.0.amd64\lib\site-packages\sentencepiece_init_.py", line 310, in LoadFromFile
return _sentencepiece.SentencePieceProcessor_LoadFromFile(self, arg)
OSError: Not found: "./models/tokenizer.model": No such file or directory Error #2

[clxyder]

https://huggingface.co/decapoda-research/llama-7b-hf/blob/main/tokenizer.model

Download this and drop it in the models directory.

[clxyder]

Hey @reynolds087 if this solved your issue can you close the issue?

[lordadamson]

Could you guys add this to the documentation, I had the same issue and expected someone would ask it in the issues, and luckily I did.