fix bug

[horheynm]

Before:
Bug on running from llm-compressor

from transformers import AutoTokenizer, AutoModelForCausalLM
from llmcompressor.transformers import SparseAutoModelForCausalLM


MODEL_ID = "nm-testing/Meta-Llama-3-8B-Instruct-fp8-hf_compat"
#MODEL_ID = "/home/dsikka/llm-compressor/examples/quantization_w4a16/new_quant_format"
model = AutoModelForCausalLM.from_pretrained(MODEL_ID, device_map="cuda")
"""
model = SparseAutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    device_map="cuda",
)
"""


tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
input_ids = tokenizer("Hello my name is", return_tensors="pt").input_ids.to("cuda")
output = model.generate(input_ids.to("cuda"), max_new_tokens=100)
print(tokenizer.decode(output[0]))

Error:

Traceback (most recent call last):
  File "/home/dsikka/llm-compressor/examples/quantization_w4a16/run_script.py", line 19, in <module>
    output = model.generate(input_ids.to("cuda"), max_new_tokens=100)
  File "/home/dsikka/venv/hf_env/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/home/dsikka/venv/hf_env/lib/python3.10/site-packages/transformers/generation/utils.py", line 2048, in generate
    result = self._sample(
  File "/home/dsikka/venv/hf_env/lib/python3.10/site-packages/transformers/generation/utils.py", line 3044, in _sample
    next_tokens = torch.multinomial(probs, num_samples=1).squeeze(1)
RuntimeError: probability tensor contains either `inf`, `nan` or element < 0

After:
Error is caused by not passing in force_zero_point, which may populate zero point to null.
Hence causes fake quant-ed value to be null

[dsikka]

awesome thank you!