Failed at model.save_pretrained_gguf #341

ch-tseng · 2024-04-16T15:31:55Z

I use the model: https://huggingface.co/taide/TAIDE-LX-7B-Chat to fine-tune, but always got the error. training is OK, but model.save_pretrained_gguf failed.

==((====))== Unsloth: Fast Llama patching release 2024.4
\ /| GPU: NVIDIA GeForce RTX 3090. Max memory: 23.691 GB. Platform = Linux.
O^O/ _/ \ Pytorch: 2.2.2+cu121. CUDA = 8.6. CUDA Toolkit = 12.1.
\ / Bfloat16 = TRUE. Xformers = 0.0.25.post1. FA = False.
"--" Free Apache license: http://github.com/unslothai/unsloth
Loading checkpoint shards: 100%|████████████████████████████████████████████████████████| 3/3 [00:03<00:00, 1.02s/it]
You set add_prefix_space. The tokenizer needs to be converted from the slow tokenizers
Unsloth 2024.4 patched 32 layers with 32 QKV layers, 32 O layers and 32 MLP layers.
Setting pad_token_id to eos_token_id:2 for open-end generation.
['Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\n\n### Instruction:\n\n\n### Input:\n上上禮拜持續出現頭痛、噁心、頭暈的症狀，有時睡一下起來還是沒有緩解，大概都痛在太陽穴上面一點，有時痛在頭腦勺（較少），躺著起來頭暈頻率越來越高（本身有貧血，但近期只要姿勢一轉換就會頭暈眼前接近黑色），容易疲累，想問一下這些症狀有需要到醫院去檢查嗎？\n\n### Response:\n\n您好：\n根據您的描述，您可能有以下幾種可能的原因：\n1. 貧血：貧血是常見的問題，若沒有定期檢查，可能會導致頭暈、頭痛、疲累等症狀。\n2. 內耳問題：內耳有平衡器官，若內耳有問題，可能會導致頭暈、頭痛、噁心等症狀。\n3. 其他疾病：如甲狀腺疾病、心臟疾病、糖尿病、高血壓等，都可能會引起頭暈、頭痛、噁心等症狀。\n建議您前往醫院，讓醫師為您做詳細的檢查，以確定病因，並接受適當的治療。\n祝健康！ ']
Unsloth: Merging 4bit and LoRA weights to 16bit...
Unsloth: Will use up to 46.9 out of 62.57 RAM for saving.
100%|█████████████████████████████████████████████████████████████████████████████████| 32/32 [00:00<00:00, 90.79it/s]
Unsloth: Saving tokenizer... Done.
Unsloth: Saving model... This might take 5 minutes for Llama-7b...
Done.
Unsloth: Converting llama model. Can use fast conversion = True.
==((====))== Unsloth: Conversion from QLoRA to GGUF information
\ /| [0] Installing llama.cpp will take 3 minutes.
O^O/ _/ \ [1] Converting HF to GUUF 16bits will take 3 minutes.
\ / [2] Converting GGUF 16bits to q4_k_m will take 20 minutes.
"--" In total, you will have to wait around 26 minutes.

Unsloth: [0] Installing llama.cpp. Unsloth: [1] Converting model The output location will be ./ch_taide This will take 3 minutes...
Loading model file ch_taide_medicine.g Loading model file ch_taide_medicine.g Loading model file ch_taide_medicine.g Loading model file ch_taide_medicine.g params = Params(n_vocab=56064, Loaded vocab file PosixPath('ch_taide_ Vocab info: <LlamaHfVocab Special vocab info: <SpecialVocab Permuting layer 0
Permuting layer 1
Permuting layer 2
Permuting layer 3
Permuting layer 4
Permuting layer 5
Permuting layer 6
Permuting layer 7
Permuting layer 8
Permuting layer 9
Permuting layer 10
Permuting layer 11
Permuting layer 12
Permuting layer 13
Permuting layer 14
Permuting layer 15
Permuting layer 16
Permuting layer 17
Permuting layer 18
Permuting layer 19
Permuting layer 20
Permuting layer 21
Permuting layer 22
Permuting layer 23
Permuting layer 24
Permuting layer 25
Permuting layer 26
Permuting layer 27
Permuting layer 28
Permuting layer 29
Permuting layer 30
Permuting layer 31
model.embed_tokens.weight model.layers.0.input_layernorm.weight model.layers.0.mlp.down_proj.weight model.layers.0.mlp.gate_proj.weight model.layers.0.mlp.up_proj.weight model.layers.0.post_attention_layernorm.weight model.layers.0.self_attn.k_proj.weight model.layers.0.self_attn.o_proj.weight model.layers.0.self_attn.q_proj.weight model.layers.0.self_attn.v_proj.weight model.layers.1.input_layernorm.weight model.layers.1.mlp.down_proj.weight model.layers.1.mlp.gate_proj.weight model.layers.1.mlp.up_proj.weight model.layers.1.post_attention_layernorm.weight model.layers.1.self_attn.k_proj.weight model.layers.1.self_attn.o_proj.weight model.layers.1.self_attn.q_proj.weight model.layers.1.self_attn.v_proj.weight model.layers.10.input_layernorm.weight model.layers.10.mlp.down_proj.weight model.layers.10.mlp.gate_proj.weight model.layers.10.mlp.up_proj.weight model.layers.10.post_attention_layernorm.weight model.layers.10.self_attn.k_proj.weight model.layers.10.self_attn.o_proj.weight model.layers.10.self_attn.q_proj.weight model.layers.10.self_attn.v_proj.weight model.layers.11.self_attn.k_proj.weight model.layers.11.self_attn.q_proj.weight model.layers.2.input_layernorm.weight model.layers.2.mlp.down_proj.weight model.layers.2.mlp.gate_proj.weight model.layers.2.mlp.up_proj.weight model.layers.2.post_attention_layernorm.weight model.layers.2.self_attn.k_proj.weight model.layers.2.self_attn.o_proj.weight model.layers.2.self_attn.q_proj.weight model.layers.2.self_attn.v_proj.weight model.layers.3.input_layernorm.weight model.layers.3.mlp.down_proj.weight model.layers.3.mlp.gate_proj.weight model.layers.3.mlp.up_proj.weight model.layers.3.post_attention_layernorm.weight model.layers.3.self_attn.k_proj.weight model.layers.3.self_attn.o_proj.weight model.layers.3.self_attn.q_proj.weight model.layers.3.self_attn.v_proj.weight model.layers.4.input_layernorm.weight model.layers.4.mlp.down_proj.weight model.layers.4.mlp.gate_proj.weight model.layers.4.mlp.up_proj.weight model.layers.4.post_attention_layernorm.weight model.layers.4.self_attn.k_proj.weight model.layers.4.self_attn.o_proj.weight model.layers.4.self_attn.q_proj.weight model.layers.4.self_attn.v_proj.weight model.layers.5.input_layernorm.weight model.layers.5.mlp.down_proj.weight model.layers.5.mlp.gate_proj.weight model.layers.5.mlp.up_proj.weight model.layers.5.post_attention_layernorm.weight model.layers.5.self_attn.k_proj.weight model.layers.5.self_attn.o_proj.weight model.layers.5.self_attn.q_proj.weight model.layers.5.self_attn.v_proj.weight model.layers.6.input_layernorm.weight model.layers.6.mlp.down_proj.weight model.layers.6.mlp.gate_proj.weight model.layers.6.mlp.up_proj.weight model.layers.6.post_attention_layernorm.weight model.layers.6.self_attn.k_proj.weight model.layers.6.self_attn.o_proj.weight model.layers.6.self_attn.q_proj.weight model.layers.6.self_attn.v_proj.weight model.layers.7.input_layernorm.weight model.layers.7.mlp.down_proj.weight model.layers.7.mlp.gate_proj.weight model.layers.7.mlp.up_proj.weight model.layers.7.post_attention_layernorm.weight model.layers.7.self_attn.k_proj.weight model.layers.7.self_attn.o_proj.weight model.layers.7.self_attn.q_proj.weight model.layers.7.self_attn.v_proj.weight model.layers.8.input_layernorm.weight model.layers.8.mlp.down_proj.weight model.layers.8.mlp.gate_proj.weight model.layers.8.mlp.up_proj.weight model.layers.8.post_attention_layernorm.weight model.layers.8.self_attn.k_proj.weight model.layers.8.self_attn.o_proj.weight model.layers.8.self_attn.q_proj.weight model.layers.8.self_attn.v_proj.weight model.layers.9.input_layernorm.weight model.layers.9.mlp.down_proj.weight model.layers.9.mlp.gate_proj.weight model.layers.9.mlp.up_proj.weight model.layers.9.post_attention_layernorm.weight model.layers.9.self_attn.k_proj.weight model.layers.9.self_attn.o_proj.weight model.layers.9.self_attn.q_proj.weight model.layers.9.self_attn.v_proj.weight model.layers.11.input_layernorm.weight model.layers.11.mlp.down_proj.weight model.layers.11.mlp.gate_proj.weight model.layers.11.mlp.up_proj.weight model.layers.11.post_attention_layernorm.weight model.layers.11.self_attn.o_proj.weight model.layers.11.self_attn.v_proj.weight model.layers.12.input_layernorm.weight model.layers.12.mlp.down_proj.weight model.layers.12.mlp.gate_proj.weight model.layers.12.mlp.up_proj.weight model.layers.12.post_attention_layernorm.weight model.layers.12.self_attn.k_proj.weight model.layers.12.self_attn.o_proj.weight model.layers.12.self_attn.q_proj.weight model.layers.12.self_attn.v_proj.weight model.layers.13.input_layernorm.weight model.layers.13.mlp.down_proj.weight model.layers.13.mlp.gate_proj.weight model.layers.13.mlp.up_proj.weight model.layers.13.post_attention_layernorm.weight model.layers.13.self_attn.k_proj.weight model.layers.13.self_attn.o_proj.weight model.layers.13.self_attn.q_proj.weight model.layers.13.self_attn.v_proj.weight model.layers.14.input_layernorm.weight model.layers.14.mlp.down_proj.weight model.layers.14.mlp.gate_proj.weight model.layers.14.mlp.up_proj.weight model.layers.14.post_attention_layernorm.weight model.layers.14.self_attn.k_proj.weight model.layers.14.self_attn.o_proj.weight model.layers.14.self_attn.q_proj.weight model.layers.14.self_attn.v_proj.weight model.layers.15.input_layernorm.weight model.layers.15.mlp.down_proj.weight model.layers.15.mlp.gate_proj.weight model.layers.15.mlp.up_proj.weight model.layers.15.post_attention_layernorm.weight model.layers.15.self_attn.k_proj.weight model.layers.15.self_attn.o_proj.weight model.layers.15.self_attn.q_proj.weight model.layers.15.self_attn.v_proj.weight model.layers.16.input_layernorm.weight model.layers.16.mlp.down_proj.weight model.layers.16.mlp.gate_proj.weight model.layers.16.mlp.up_proj.weight model.layers.16.post_attention_layernorm.weight model.layers.16.self_attn.k_proj.weight model.layers.16.self_attn.o_proj.weight model.layers.16.self_attn.q_proj.weight model.layers.16.self_attn.v_proj.weight model.layers.17.input_layernorm.weight model.layers.17.mlp.down_proj.weight model.layers.17.mlp.gate_proj.weight model.layers.17.mlp.up_proj.weight model.layers.17.post_attention_layernorm.weight model.layers.17.self_attn.k_proj.weight model.layers.17.self_attn.o_proj.weight model.layers.17.self_attn.q_proj.weight model.layers.17.self_attn.v_proj.weight model.layers.18.input_layernorm.weight model.layers.18.mlp.down_proj.weight model.layers.18.mlp.gate_proj.weight model.layers.18.mlp.up_proj.weight model.layers.18.post_attention_layernorm.weight model.layers.18.self_attn.k_proj.weight model.layers.18.self_attn.o_proj.weight model.layers.18.self_attn.q_proj.weight model.layers.18.self_attn.v_proj.weight model.layers.19.input_layernorm.weight model.layers.19.mlp.down_proj.weight model.layers.19.mlp.gate_proj.weight model.layers.19.mlp.up_proj.weight model.layers.19.post_attention_layernorm.weight model.layers.19.self_attn.k_proj.weight model.layers.19.self_attn.o_proj.weight model.layers.19.self_attn.q_proj.weight model.layers.19.self_attn.v_proj.weight model.layers.20.input_layernorm.weight model.layers.20.mlp.down_proj.weight model.layers.20.mlp.gate_proj.weight model.layers.20.mlp.up_proj.weight model.layers.20.post_attention_layernorm.weight model.layers.20.self_attn.k_proj.weight model.layers.20.self_attn.o_proj.weight model.layers.20.self_attn.q_proj.weight model.layers.20.self_attn.v_proj.weight model.layers.21.input_layernorm.weight model.layers.21.mlp.down_proj.weight model.layers.21.mlp.gate_proj.weight model.layers.21.mlp.up_proj.weight model.layers.21.post_attention_layernorm.weight model.layers.21.self_attn.k_proj.weight model.layers.21.self_attn.o_proj.weight model.layers.21.self_attn.q_proj.weight model.layers.21.self_attn.v_proj.weight model.layers.22.input_layernorm.weight model.layers.22.mlp.down_proj.weight model.layers.22.mlp.gate_proj.weight model.layers.22.mlp.up_proj.weight model.layers.22.post_attention_layernorm.weight model.layers.22.self_attn.k_proj.weight model.layers.22.self_attn.o_proj.weight model.layers.22.self_attn.q_proj.weight model.layers.22.self_attn.v_proj.weight model.layers.23.self_attn.k_proj.weight model.layers.23.self_attn.o_proj.weight model.layers.23.self_attn.q_proj.weight model.layers.23.self_attn.v_proj.weight lm_head.weight model.layers.23.input_layernorm.weight model.layers.23.mlp.down_proj.weight model.layers.23.mlp.gate_proj.weight model.layers.23.mlp.up_proj.weight model.layers.23.post_attention_layernorm.weight model.layers.24.input_layernorm.weight model.layers.24.mlp.down_proj.weight model.layers.24.mlp.gate_proj.weight model.layers.24.mlp.up_proj.weight model.layers.24.post_attention_layernorm.weight model.layers.24.self_attn.k_proj.weight model.layers.24.self_attn.o_proj.weight model.layers.24.self_attn.q_proj.weight model.layers.24.self_attn.v_proj.weight model.layers.25.input_layernorm.weight model.layers.25.mlp.down_proj.weight model.layers.25.mlp.gate_proj.weight model.layers.25.mlp.up_proj.weight model.layers.25.post_attention_layernorm.weight model.layers.25.self_attn.k_proj.weight model.layers.25.self_attn.o_proj.weight model.layers.25.self_attn.q_proj.weight model.layers.25.self_attn.v_proj.weight model.layers.26.input_layernorm.weight model.layers.26.mlp.down_proj.weight model.layers.26.mlp.gate_proj.weight model.layers.26.mlp.up_proj.weight model.layers.26.post_attention_layernorm.weight model.layers.26.self_attn.k_proj.weight model.layers.26.self_attn.o_proj.weight model.layers.26.self_attn.q_proj.weight model.layers.26.self_attn.v_proj.weight model.layers.27.input_layernorm.weight model.layers.27.mlp.down_proj.weight model.layers.27.mlp.gate_proj.weight model.layers.27.mlp.up_proj.weight model.layers.27.post_attention_layernorm.weight model.layers.27.self_attn.k_proj.weight model.layers.27.self_attn.o_proj.weight model.layers.27.self_attn.q_proj.weight model.layers.27.self_attn.v_proj.weight model.layers.28.input_layernorm.weight model.layers.28.mlp.down_proj.weight model.layers.28.mlp.gate_proj.weight model.layers.28.mlp.up_proj.weight model.layers.28.post_attention_layernorm.weight model.layers.28.self_attn.k_proj.weight model.layers.28.self_attn.o_proj.weight model.layers.28.self_attn.q_proj.weight model.layers.28.self_attn.v_proj.weight model.layers.29.input_layernorm.weight model.layers.29.mlp.down_proj.weight model.layers.29.mlp.gate_proj.weight model.layers.29.mlp.up_proj.weight model.layers.29.post_attention_layernorm.weight model.layers.29.self_attn.k_proj.weight model.layers.29.self_attn.o_proj.weight model.layers.29.self_attn.q_proj.weight model.layers.29.self_attn.v_proj.weight model.layers.30.input_layernorm.weight model.layers.30.mlp.down_proj.weight model.layers.30.mlp.gate_proj.weight model.layers.30.mlp.up_proj.weight model.layers.30.post_attention_layernorm.weight model.layers.30.self_attn.k_proj.weight model.layers.30.self_attn.o_proj.weight model.layers.30.self_attn.q_proj.weight model.layers.30.self_attn.v_proj.weight model.layers.31.input_layernorm.weight model.layers.31.mlp.down_proj.weight model.layers.31.mlp.gate_proj.weight model.layers.31.mlp.up_proj.weight model.layers.31.post_attention_layernorm.weight model.layers.31.self_attn.k_proj.weight model.layers.31.self_attn.o_proj.weight model.layers.31.self_attn.q_proj.weight model.layers.31.self_attn.v_proj.weight model.norm.weight Writing ch_taide_medicine.gguf-unsloth.F16.gguf, Traceback (most recent call last):
File "/GPUData/working/unsloth/convert if True: model.save_pretrained_gguf("c File "/home/chtseng/envs/LM2/lib/pytho file_location = save_to_gguf(model_type, File "/home/chtseng/envs/LM2/lib/pytho raise RuntimeError(
RuntimeError: Unsloth: Quantization You might have to compile llama.cpp You do not need to close this You must run this in the same git clone h_taide_medicine.gguf", tokenizer, quantization_method = "quantized")
n3.10/site-packages/unsloth/save.py", line 1333, in unsloth_save_pretrained_gguf
new_save_directory, quantization_method, first_conversion, makefile)
n3.10/site-packages/unsloth/save.py", line 957, in save_to_gguf
failed for ./ch_taide_medicine.gguf-unsloth.F16.gguf
yourself, then run this again.
Python program. Run the following commands in a new terminal:
folder as you're saving your model.
by.my.eu.org/ggerganov/llama.cpp">https://github.com/ggerganov/llama.cpp
clean && LLAMA_CUDA=1 make all -j
n.

The text was updated successfully, but these errors were encountered:

danielhanchen · 2024-04-16T17:54:16Z

@ch-tseng Sorry on the issue - you'll have to unfortunately manually convert it to GGUF via https://github.com/unslothai/unsloth/wiki#manually-saving-to-gguf.

danielhanchen · 2024-04-21T19:34:51Z

I'm unsure if the latest release fixes this

GeneralProtectionFault · 2024-09-07T20:46:57Z

This appears to still be an issue.

My setup:
AMD Ryzen 5
Nvidia RTX 3060 12 GB
96 GB RAM
Garuda (Arch) Linux

Here's my specific line that fails (The training is just literally 2 prompts to test the process--training works fine):
model.save_pretrained_gguf(os.path.join(os.getcwd(), "Llama-3-8B-Instruct-TEST"), tokenizer, quantization_method = "q4_k_m", maximum_memory_usage = 0.5)

...which results in the familiar error:

git clone --recursive https://github.com/ggerganov/llama.cpp cd llama.cpp && make clean && make all -j Once that's done, redo the quantization.

I performed those exact steps and it does nothing. I also tried the workaround mentioned here:
#748
...which is just using an earlier commit of llama.cpp to address that problem--no change.

I get the same problem whether it's llama3 or llama3.1.

To be clear--it seems to successfully create "the model"--I see the 4 .safetensors files, but fails to create the quantized .gguf model.

@danielhanchen I see your mention of the manual workaround, and I see this code there:

`git clone --recursive https://github.com/ggerganov/llama.cpp
make clean -C llama.cpp
make all -j -C llama.cpp
pip install gguf protobuf

python llama.cpp/convert-hf-to-gguf.py FOLDER --outfile OUTPUT --outtype f16`

...but from what I can tell, that is simply recreating the step which save_pretrained_gguf did successfully.
It's the quantizing that failed.
How do we save the quantized model from our fine-tuning, manually or otherwise?

GeneralProtectionFault · 2024-09-07T23:55:03Z

Ok I think I have at least a clue, in case it helps.
I have not tested running the model yet, but I did manage to convert it.

So, after fine-tuning using the typical unsloth notebooks as a guide for the code, running this:
model.save_pretrained_merged("merged_model", tokenizer, save_method = "merged_16bit")

...should export the safetensors files (I believe this is the Hugging Face format).
Again, I think this all that is accomplished by the code I saw on the page to work around this bug manually, but I could be wrong.

So, going directly to llama.cpp Github, it seems the Readme is out of date. It references "convert.py" but that script is not in there. It looks like they replaced it with more specific scripts:

convert_hf_to_gguf.py
convert_hf_to_gguf_update.py (I have no idea what the "update" is, I didn't try this one)
convert_llama_ggml_to_gguf.py
convert_lora_to_gguf.py

Also, it mentions "quantize" as the binary which does that 2nd step. After building (w/ make), I don't see that. But, I do see this, which appears to be a simple name change:

llama-quantize

So essentially, given the Hugging Face safetensors files, we need to convert to gguf--WITHOUT quantizing (FP16 or FP32 is my understanding), i.e.:

python convert_hf_to_gguf.py {the_folder_with_your_model_files}

Then:

llama-quantize {source_fp16_gguf_file} {destination_filename} Q4_K_M

**** BUT ****
This did not work when I tried it from the llama.cpp in the unsloth folder. If I tried, I got missing requirements. If I installed the requirements, I got a dependency mess because llama.cpp uses an older version of torch, prior to 2.4, that causes xformers conflicts, etc, dependency hell confirmed....

So, I made a separate virtual Python environment for unsloth & llama.cpp, and cloned llama.cpp into a separate location and just installed it's requirements from its own virtual environment, ran make in that folder, then ran the commands I just put above ^^^ from there on the HF model which unsloth was at least able to spit out.

I don't know enough about unsloth to know what the best solution is, or if this is even the root cause. But that seems to work. I don't know if there's a better way around that, like matching unsloth's torch dependency to llama.cpp's, etc...

GeneralProtectionFault · 2024-09-08T20:37:52Z

To illustrate the point w/ a concrete example--this works, however inelegant:

IMPORTANT

This python code is run from a virtual environment that installed UNSLOTH's requirements

The first subprocess calls the python binary from a 2nd virtual environment that installed LLAMA.CPP's requirements.

This lets us run the Python scripts we get from llama.cpp without running into a dependency conflict.

llamacpp_venv_bin = "/home/user/Documents/Python/venv_llamacpp/bin/python" llamacpp_hf_gguf_script = "/home/user/Documents/Python/llama.cpp/convert_hf_to_gguf.py" llamacpp_quantize_bin = "/home/user/Documents/Python/llama.cpp/llama-quantize"

new_fp16_model_name = "Llama3.1-TestModel-FP16" new_quantized_model_name = "Llama3.1-TestModel-4bit" quantize_options = ["Q8_0", "Q6_K", "Q5_K_M", "Q5_K_S", "Q4_K_M", "Q4_K_S", "Q3_K_L", "Q3_K_M", "Q3_K_S", "Q2_K"] quantize_method = "Q4_K_M"

This will save the model as safetensors files (Hugging Face format)

Creates a folder under the current working directory for whatever name is passed in as the 1st argument (that's why it will be used below in determining the paths)

model.save_pretrained_merged(new_fp16_model_name, tokenizer, save_method = "merged_16bit", maximum_memory_usage = 0.5)

This calls the python binary from the VENV made specifically for llama.cpp. This is necessary since unsloth & llama.cpp use different versions of torch.

Use the llama.cpp python scripts to convert the Hugging Face model to GGUF (this will still be 16-bit, not quantized!)

subprocess.call(shlex.split(f"{llamacpp_venv_bin} {llamacpp_hf_gguf_script} {os.path.join(os.getcwd(), new_fp16_model_name)}"))

Now, quantize it with the binary from llama.cpp

This calls the llama-quantize binary from llama.cpp, followed by the source filename and destination filename.

subprocess.call(shlex.split(f"{llamacpp_quantize_bin} {os.path.join(os.getcwd(), new_fp16_model_name, source_model_name)}-F16.gguf \ {os.path.join(os.getcwd(), new_fp16_model_name, new_quantized_model_name)}.gguf {quantize_method}"))

danielhanchen · 2024-09-10T08:28:19Z

@GeneralProtectionFault Apologies on the delay!

Yep so model.save_pretrained_merged will merge to 16bit, then use llama.cpp to first convert it to F16 format, then use llama-quantize to quantize it. I'm trying to make the whole process more stable

fullstackwebdev · 2024-09-10T20:35:38Z

Is this related to my issue to gemma 2 27b bnb 4bit?

vllm/aphrodite/sglang all return an error
KeyError: 'model.layers.0.mlp.down_proj.weight

https://huggingface.co/unsloth/gemma-2-27b-it-bnb-4bit/discussions/5

danielhanchen · 2024-09-14T08:32:44Z

@fullstackwebdev Apologies on the delay - I'm assuming the bnb version is not supported in vLLM / SGLang - please try https://huggingface.co/unsloth/gemma-2-27b-it

shimmyshimmer · 2025-01-19T07:26:58Z

Hopefully everything is solved now?

sozercan mentioned this issue Apr 19, 2024

[REQ] update unsloth to llama3 release sozercan/aikit#199

Closed

1 task

shimmyshimmer closed this as completed Jan 19, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Failed at model.save_pretrained_gguf #341

Failed at model.save_pretrained_gguf #341

ch-tseng commented Apr 16, 2024

danielhanchen commented Apr 16, 2024

danielhanchen commented Apr 21, 2024

GeneralProtectionFault commented Sep 7, 2024 •

edited

Loading

GeneralProtectionFault commented Sep 7, 2024 •

edited

Loading

GeneralProtectionFault commented Sep 8, 2024 •

edited

Loading

danielhanchen commented Sep 10, 2024

fullstackwebdev commented Sep 10, 2024

danielhanchen commented Sep 14, 2024

shimmyshimmer commented Jan 19, 2025

Failed at model.save_pretrained_gguf #341

Failed at model.save_pretrained_gguf #341

Comments

ch-tseng commented Apr 16, 2024

danielhanchen commented Apr 16, 2024

danielhanchen commented Apr 21, 2024

GeneralProtectionFault commented Sep 7, 2024 • edited Loading

GeneralProtectionFault commented Sep 7, 2024 • edited Loading

GeneralProtectionFault commented Sep 8, 2024 • edited Loading

IMPORTANT

This python code is run from a virtual environment that installed UNSLOTH's requirements

The first subprocess calls the python binary from a 2nd virtual environment that installed LLAMA.CPP's requirements.

This lets us run the Python scripts we get from llama.cpp without running into a dependency conflict.

This will save the model as safetensors files (Hugging Face format)

Creates a folder under the current working directory for whatever name is passed in as the 1st argument (that's why it will be used below in determining the paths)

This calls the python binary from the VENV made specifically for llama.cpp. This is necessary since unsloth & llama.cpp use different versions of torch.

Use the llama.cpp python scripts to convert the Hugging Face model to GGUF (this will still be 16-bit, not quantized!)

Now, quantize it with the binary from llama.cpp

This calls the llama-quantize binary from llama.cpp, followed by the source filename and destination filename.

danielhanchen commented Sep 10, 2024

fullstackwebdev commented Sep 10, 2024

danielhanchen commented Sep 14, 2024

shimmyshimmer commented Jan 19, 2025

GeneralProtectionFault commented Sep 7, 2024 •

edited

Loading

GeneralProtectionFault commented Sep 7, 2024 •

edited

Loading

GeneralProtectionFault commented Sep 8, 2024 •

edited

Loading