[Bug]: FP8 W8A8 quantized model doesn't work properly on V1 with default compilation_config (use_cudagraph = True) #13212
Description
Your current environment
The output of `python collect_env.py`
PyTorch version: 2.5.1+cu124
Is debug build: False
CUDA used to build PyTorch: 12.4
ROCM used to build PyTorch: N/A
OS: TencentOS Server 3.1 (Final) (x86_64)
GCC version: (GCC) 8.5.0 20210514 (TencentOS 8.5.0-18)
Clang version: 16.0.6 (Red Hat 16.0.6-2.module+el8.8.0+557+454507bd)
CMake version: Could not collect
Libc version: glibc-2.28
Python version: 3.12.8 | packaged by conda-forge | (main, Dec 5 2024, 14:24:40) [GCC 13.3.0] (64-bit runtime)
Python platform: Linux-5.4.119-19.0009.40-x86_64-with-glibc2.28
Is CUDA available: True
CUDA runtime version: 12.2.140
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration:
GPU 0: NVIDIA L40
Nvidia driver version: 535.161.07
cuDNN version: Probably one of the following:
/usr/local/cuda-12.4/targets/x86_64-linux/lib/libcudnn.so.8
/usr/local/cuda-12.4/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8
/usr/local/cuda-12.4/targets/x86_64-linux/lib/libcudnn_adv_train.so.8
/usr/local/cuda-12.4/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8
/usr/local/cuda-12.4/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8
/usr/local/cuda-12.4/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8
/usr/local/cuda-12.4/targets/x86_64-linux/lib/libcudnn_ops_train.so.8
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True
CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 384
On-line CPU(s) list: 0-383
Thread(s) per core: 2
Core(s) per socket: 96
Socket(s): 2
NUMA node(s): 2
Vendor ID: AuthenticAMD
CPU family: 25
Model: 17
Model name: AMD EPYC 9K84 96-Core Processor
Stepping: 0
CPU MHz: 2600.026
BogoMIPS: 5200.05
Hypervisor vendor: KVM
Virtualization type: full
L1d cache: 32K
L1i cache: 32K
L2 cache: 1024K
L3 cache: 32768K
NUMA node0 CPU(s): 0-191
NUMA node1 CPU(s): 192-383
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm rep_good nopl cpuid extd_apicid amd_dcm tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core invpcid_single ibpb vmmcall fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 avx512_bf16 clzero xsaveerptr wbnoinvd arat avx512vbmi umip avx512_vbmi2 vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid fsrm
Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] nvidia-cublas-cu12==12.4.5.8
[pip3] nvidia-cuda-cupti-cu12==12.4.127
[pip3] nvidia-cuda-nvrtc-cu12==12.4.127
[pip3] nvidia-cuda-runtime-cu12==12.4.127
[pip3] nvidia-cudnn-cu12==9.1.0.70
[pip3] nvidia-cufft-cu12==11.2.1.3
[pip3] nvidia-curand-cu12==10.3.5.147
[pip3] nvidia-cusolver-cu12==11.6.1.9
[pip3] nvidia-cusparse-cu12==12.3.1.170
[pip3] nvidia-nccl-cu12==2.21.5
[pip3] nvidia-nvjitlink-cu12==12.4.127
[pip3] nvidia-nvtx-cu12==12.4.127
[pip3] pyzmq==26.2.1
[pip3] torch==2.5.1
[pip3] torchaudio==2.5.1
[pip3] torchvision==0.20.1
[pip3] transformers==4.48.3
[pip3] triton==3.1.0
[conda] numpy 1.26.4 pypi_0 pypi
[conda] nvidia-cublas-cu12 12.4.5.8 pypi_0 pypi
[conda] nvidia-cuda-cupti-cu12 12.4.127 pypi_0 pypi
[conda] nvidia-cuda-nvrtc-cu12 12.4.127 pypi_0 pypi
[conda] nvidia-cuda-runtime-cu12 12.4.127 pypi_0 pypi
[conda] nvidia-cudnn-cu12 9.1.0.70 pypi_0 pypi
[conda] nvidia-cufft-cu12 11.2.1.3 pypi_0 pypi
[conda] nvidia-curand-cu12 10.3.5.147 pypi_0 pypi
[conda] nvidia-cusolver-cu12 11.6.1.9 pypi_0 pypi
[conda] nvidia-cusparse-cu12 12.3.1.170 pypi_0 pypi
[conda] nvidia-nccl-cu12 2.21.5 pypi_0 pypi
[conda] nvidia-nvjitlink-cu12 12.4.127 pypi_0 pypi
[conda] nvidia-nvtx-cu12 12.4.127 pypi_0 pypi
[conda] pyzmq 26.2.1 pypi_0 pypi
[conda] torch 2.5.1 pypi_0 pypi
[conda] torchaudio 2.5.1 pypi_0 pypi
[conda] torchvision 0.20.1 pypi_0 pypi
[conda] transformers 4.48.3 pypi_0 pypi
[conda] triton 3.1.0 pypi_0 pypi
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.1.dev1+g0ccd876 (git sha: 0ccd876
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
NCCL_CUMEM_ENABLE=0
TORCHINDUCTOR_COMPILE_THREADS=1
CUDA_MODULE_LOADING=LAZY
🐛 Describe the bug
- Using vLLM V1 with
enforce_eager=False - Model loaded and generate output without any error, but the output content is gibberish
An temporary approach?
Inspired by this comment, I turned compilation_config.use_cudagraph from True to False (diff: imkero@92116c3, should change the source code in vllm/config.py because it always override compilation_config), and then it works as expected.
Lines 3237 to 3249 in 009439c
Should this problem be addressed by modifying compilation_config, or some bug should be fixed instead?
Code and model to reproduce
Model: nm-testing/DeepSeek-R1-Distill-Qwen-14B-FP8-Dynamic
Code:
vLLM latest main: 0ccd876, and this script:
import os
os.environ["VLLM_USE_V1"] = "1"
os.environ["VLLM_ENABLE_V1_MULTIPROCESSING"] = "0"
from vllm import LLM, SamplingParams
from transformers import AutoTokenizer
MODEL = "nm-testing/DeepSeek-R1-Distill-Qwen-14B-FP8-Dynamic"
tokenizer = AutoTokenizer.from_pretrained(MODEL)
def chat_template(question):
return tokenizer.apply_chat_template(
[
{
"role": "user",
"content": question,
},
],
tokenize=False,
add_generation_prompt=True,
)
prompts = [
chat_template(question)
for question in [
"Hello",
"Which one is greater: 9.11 or 9.9?"
]
]
sampling_params = SamplingParams(temperature=0, max_tokens=128)
llm = LLM(
model=MODEL,
gpu_memory_utilization=0.9,
max_model_len=1024,
)
outputs = llm.generate(prompts, sampling_params)
for output in outputs:
prompt = output.prompt
generated_text = output.outputs[0].text
print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")Outputs
vLLM main branch code output:
Prompt: '<｜begin▁of▁sentence｜><｜User｜>Hello<｜Assistant｜>', Generated text: '开头\n空气质量格佯抽 bist exc� 3好00\n210的，的,100000<think>10<think>0100329003021 the0，111的1的1 2的1,1，，0的1 的,的1的的，ence，的 的0ence,\n1的4的的的1的,的,0201<think>，，1001，002的，的，的<think>的frac1，的<think><think>的的的的,'
Prompt: '<｜begin▁of▁sentence｜><｜User｜>Which one is greater: 9.11 or 9.9?<｜Assistant｜>', Generated text: '开头\n reogra�001-001<think> record pre��使用2=1，00011000，，，11,,,0，,,，,1\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n'
Exptected output:
Prompt: '<｜begin▁of▁sentence｜><｜User｜>Hello<｜Assistant｜>', Generated text: '<think>\n\n</think>\n\nHello! How can I assist you today? 😊'
Prompt: '<｜begin▁of▁sentence｜><｜User｜>Which one is greater: 9.11 or 9.9?<｜Assistant｜>', Generated text: "<think>\nFirst, I'll compare the whole number parts of both numbers. Both 9.11 and 9.9 have the same whole number, which is 9.\n\nNext, I'll look at the decimal parts. In 9.11, the decimal part is 0.11, and in 9.9, it's 0.9.\n\nTo make a clear comparison, I'll express 0.9 as 0.90. Now, comparing 0.90 and 0.11, it's evident that 0.90 is greater than 0."
Before submitting a new issue...
- Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.