Skip to content

jackboyla/create-gguf

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

6 Commits
 
 
 
 
 
 
 
 
 
 

Repository files navigation

  1. Run download_snapshot.py for your desired model
  2. Clone llama.cpp. Ideally use the latest version, but if there is no convert_hf_to_gguf.py file, you can run git checkout 19d8762.
  3. pip install -r llama.cpp/requirements.txt
  4. Check that your model has a tokenizer.model file. If not, you'll need to get it from the base model. e.g for Phi-3-mini-4k-instruct-graph, there was no such file, so I downloaded it from the original Phi-3-mini repo. Put this file in your downloaded model's dir. PLEASE NOTE: if the tokenizer/vocab was modified from the base model to your desried finetuned model, this approach will likely cause issues.
  5. Run:
python llama.cpp/convert_hf_to_gguf.py create-gguf/Phi-3-mini-4k-instruct-graph \
  --outfile create-gguf/Phi-3-mini-4k-instruct-graph.Q8_0.gguf \
  --outtype q8_0

To add to a huggingface model repo, follow these steps

To clear huggingface cache:

pip install -U "huggingface_hub[cli]"
huggingface-cli delete-cache

About

No description, website, or topics provided.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published