AWQ is not working

[endomorphosis]

System Info

Transformers fails with the following error, when trying to use AWQ with TGI / neural compression enginer, or optimum habana
ValueError: AWQ is only available on GPU

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

.

Expected behavior

.

[regisss]

Is it supposed to work on Gaudi?

[endomorphosis]

The primary goal is to get llama405b on a single gaudi node

I had read originally that huggingface TGI was supposed to use awq, but i was unable to use any sort of quantization method at all, provided by huggingface quants, including GPTQ, uint4, etc, its just spread amongst different issues.

[regisss]

I think GPTQ should work on Gaudi no?

[endomorphosis]

no, neither generating quantized models with the intel neural compressor nor does https://huggingface.co/hugging-quants/Meta-Llama-3.1-405B-Instruct-GPTQ-INT4 work on tgi_gaudi, nor does fp8 work with INC on a single node.