New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

How to convert the AWQ model after the quantization into safetensors #232

Open

vladimiralbrekhtccr opened this issue Oct 31, 2024 · 1 comment

vladimiralbrekhtccr commented Oct 31, 2024

Sorry.
I searched everywhere but I don't found any script to convert the quantized .pt file to the safetensors.

I received the Llama-3.1-Nemotron-70B-Instruct-HF-w4-g128.pt
After using this script:

python -m awq.entry --model_path $MODEL \
    --tasks wikitext \
    --w_bit $w_bit --q_group_size 128 \
    --load_quant quant_cache/$MODEL-w$w_bit-g128-awq.pt \
    --max_memory $max_memory \
    --parallel

Please maybe someone can help me with that.

The text was updated successfully, but these errors were encountered:

cute149q commented Dec 9, 2024

Hi did you solve it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment