[Bug]: compressed tensors moe (nvfp4) consume an unusually large amount of vram

[ReinForce-II]

Your current environment

The output of python collect_env.py

Collecting environment information...                                                                                                                                                                                                                                09:12:45 [99/556]
uv is set
==============================
        System Info
==============================
OS                           : Ubuntu 24.04.1 LTS (x86_64)
GCC version                  : (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0
Clang version                : Could not collect
CMake version                : Could not collect
Libc version                 : glibc-2.39

==============================
       PyTorch Info
==============================
PyTorch version              : 2.8.0+cu128
Is debug build               : False                                                                                                                                                                                                                                                  CUDA used to build PyTorch   : 12.8                                                                                                                                                                                                                                                   ROCM used to build PyTorch   : N/A                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          ==============================                                                                                                                                                                                                                                                              Python Environment                                                                                                                                                                                                                                                              ==============================                                                                                                                                                                                                                                                        Python version               : 3.12.3 (main, Aug 14 2025, 17:47:21) [GCC 13.3.0] (64-bit runtime)
Python platform              : Linux-5.15.0-152-generic-x86_64-with-glibc2.39

==============================
       CUDA / GPU Info
==============================
Is CUDA available            : True
CUDA runtime version         : 12.8.93
CUDA_MODULE_LOADING set to   : LAZY
GPU models and configuration : GPU 0: NVIDIA GeForce RTX 5090
Nvidia driver version        : 575.64.05
cuDNN version                : Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.9.8.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv.so.9.8.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn.so.9.8.0
/usr/lib/x86_64-linux-gnu/libcudnn_engines_precompiled.so.9.8.0
/usr/lib/x86_64-linux-gnu/libcudnn_engines_runtime_compiled.so.9.8.0
/usr/lib/x86_64-linux-gnu/libcudnn_graph.so.9.8.0
/usr/lib/x86_64-linux-gnu/libcudnn_heuristic.so.9.8.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops.so.9.8.0
HIP runtime version          : N/A
MIOpen runtime version       : N/A
Is XNNPACK available         : True

==============================
          CPU Info
==============================
Architecture:                            x86_64
CPU op-mode(s):                          32-bit, 64-bit
Address sizes:                           52 bits physical, 57 bits virtual
Byte Order:                              Little Endian
CPU(s):                                  256
On-line CPU(s) list:                     0-255
Vendor ID:                               AuthenticAMD
Model name:                              AMD EPYC 9754 128-Core Processor
CPU family:                              25
Model:                                   160
Thread(s) per core:                      2
Core(s) per socket:                      128
Socket(s):                               1
Stepping:                                2
Frequency boost:                         enabled
CPU(s) scaling MHz:                      68%
CPU max MHz:                             2250.0000
CPU min MHz:                             1500.0000
BogoMIPS:                                4493.32
Flags:                                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq moni
tor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 invpcid_single hw
_pstate ssbd mba ibrs ibpb stibp ibrs_enhanced vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_lo
cal avx512_bf16 clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512
_vnni avx512_bitalg avx512_vpopcntdq la57 rdpid overflow_recov succor smca fsrm flush_l1d
Virtualization:                          AMD-V
L1d cache:                               4 MiB (128 instances)
L1i cache:                               4 MiB (128 instances)
L2 cache:                                128 MiB (128 instances)
L3 cache:                                256 MiB (16 instances)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-255
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Mitigation; safe RET
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:                Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; Enhanced / Automatic IBRS; IBPB disabled; STIBP disabled; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds:                     Not affected
Vulnerability Tsx async abort:           Not affected

==============================
Versions of relevant libraries
==============================
[pip3] flashinfer-python==0.4.0
[pip3] numpy==2.2.6
[pip3] nvidia-cublas-cu12==12.8.4.1
[pip3] nvidia-cuda-cupti-cu12==12.8.90
[pip3] nvidia-cuda-nvrtc-cu12==12.8.93
[pip3] nvidia-cuda-runtime-cu12==12.8.90
[pip3] nvidia-cudnn-cu12==9.10.2.21
[pip3] nvidia-cudnn-frontend==1.15.0
[pip3] nvidia-cufft-cu12==11.3.3.83
[pip3] nvidia-cufile-cu12==1.13.1.3
[pip3] nvidia-curand-cu12==10.3.9.90
[pip3] nvidia-cusolver-cu12==11.7.3.90
[pip3] nvidia-cusparse-cu12==12.5.8.93
[pip3] nvidia-cusparselt-cu12==0.7.1
[pip3] nvidia-cutlass-dsl==4.2.1
[pip3] nvidia-ml-py==13.580.82
[pip3] nvidia-nccl-cu12==2.27.3
[pip3] nvidia-nvjitlink-cu12==12.8.93
[pip3] nvidia-nvtx-cu12==12.8.90
[pip3] pyzmq==27.1.0
[pip3] torch==2.8.0+cu128
[pip3] torchaudio==2.8.0+cu128
[pip3] torchvision==0.23.0+cu128
[pip3] transformers==4.57.0
[pip3] triton==3.4.0
[conda] Could not collect

==============================
         vLLM Info
==============================
ROCM Version                 : Could not collect
vLLM Version                 : 0.11.0
vLLM Build Flags:
  CUDA Archs: Not Set; ROCm: Disabled
GPU Topology:
        GPU0    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      0-255   0               N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

==============================
     Environment Variables
==============================
NVIDIA_VISIBLE_DEVICES=all
NVIDIA_REQUIRE_CUDA=cuda>=12.8 brand=unknown,driver>=470,driver<471 brand=grid,driver>=470,driver<471 brand=tesla,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=nvidiartx,driver>=470,dr
iver<471 brand=vapps,driver>=470,driver<471 brand=vpc,driver>=470,driver<471 brand=vcs,driver>=470,driver<471 brand=vws,driver>=470,driver<471 brand=cloudgaming,driver>=470,driver<471 brand=unknown,driver>=535,driver<536 brand=grid,driver>=535,driver<536 brand=tesla,driver>=535
,driver<536 brand=nvidia,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=vapps,driver>=535,driver<536 brand=vpc,driver>=535,driver<536 brand=vcs,driver>=535,driver<536 brand=vws,drive
r>=535,driver<536 brand=cloudgaming,driver>=535,driver<536 brand=unknown,driver>=550,driver<551 brand=grid,driver>=550,driver<551 brand=tesla,driver>=550,driver<551 brand=nvidia,driver>=550,driver<551 brand=quadro,driver>=550,driver<551 brand=quadrortx,driver>=550,driver<551 br
and=nvidiartx,driver>=550,driver<551 brand=vapps,driver>=550,driver<551 brand=vpc,driver>=550,driver<551 brand=vcs,driver>=550,driver<551 brand=vws,driver>=550,driver<551 brand=cloudgaming,driver>=550,driver<551 brand=unknown,driver>=560,driver<561 brand=grid,driver>=560,driver
<561 brand=tesla,driver>=560,driver<561 brand=nvidia,driver>=560,driver<561 brand=quadro,driver>=560,driver<561 brand=quadrortx,driver>=560,driver<561 brand=nvidiartx,driver>=560,driver<561 brand=vapps,driver>=560,driver<561 brand=vpc,driver>=560,driver<561 brand=vcs,driver>=56
0,driver<561 brand=vws,driver>=560,driver<561 brand=cloudgaming,driver>=560,driver<561 brand=unknown,driver>=565,driver<566 brand=grid,driver>=565,driver<566 brand=tesla,driver>=565,driver<566 brand=nvidia,driver>=565,driver<566 brand=quadro,driver>=565,driver<566 brand=quadror
tx,driver>=565,driver<566 brand=nvidiartx,driver>=565,driver<566 brand=vapps,driver>=565,driver<566 brand=vpc,driver>=565,driver<566 brand=vcs,driver>=565,driver<566 brand=vws,driver>=565,driver<566 brand=cloudgaming,driver>=565,driver<566
NCCL_VERSION=2.25.1-1
NVIDIA_DRIVER_CAPABILITIES=compute,utility
NVIDIA_PRODUCT_NAME=CUDA
CUDA_VERSION=12.8.1
LD_LIBRARY_PATH=/usr/local/cuda/lib64
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
CUDA_MODULE_LOADING=LAZY

🐛 Describe the bug

• Merged support flashinfer_fp4 moe for 5090 gpu #26669

VLLM_USE_FLASHINFER_MOE_FP4=1 vllm serve RedHatAI/Qwen3-30B-A3B-NVFP4

(EngineCore_DP0 pid=6016) INFO 10-14 08:10:58 [default_loader.py:267] Loading weights took 7.45 seconds
(EngineCore_DP0 pid=6016) INFO 10-14 08:10:59 [gpu_model_runner.py:2653] Model loading took 25.9087 GiB and 8.627268 seconds
(EngineCore_DP0 pid=6016) INFO 10-14 08:11:08 [backends.py:548] Using cache directory: /root/.cache/vllm/torch_compile_cache/00507ceb63/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=6016) INFO 10-14 08:11:08 [backends.py:559] Dynamo bytecode transform time: 8.67 s
(EngineCore_DP0 pid=6016) INFO 10-14 08:11:13 [backends.py:197] Cache the graph for dynamic shape for later use
(EngineCore_DP0 pid=6016) INFO 10-14 08:11:59 [backends.py:218] Compiling a graph for dynamic shape takes 5.96 s
(EngineCore_DP0 pid=6016) INFO 10-14 08:17:26 [monitor.py:34] torch.compile takes 9.64 s in total

It takes 25.9087 GiB here, about 9 GiB larger than expected.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[BenasdTW]

I encountered the same issue. I quantized a Qwen3-MoE model using llm-compressor and ran inference with vLLM, which produced a similar result.

The issue does not occur in models quantized with modelopt.

VLLM_USE_FLASHINFER_MOE_FP4=1 vllm serve Benasd/Qwen3-30B-A3B-Instruct-2507-NVFP4 --max_model_len 8192

Result: Model loading took 25.9092 GiB but the model is ~18GB in size

(APIServer pid=20863) INFO 10-14 09:50:50 [api_server.py:1873] vLLM API server version 0.11.0rc2.dev479+g577d49821
(APIServer pid=20863) INFO 10-14 09:50:50 [utils.py:243] non-default args: {'model_tag': 'Benasd/Qwen3-30B-A3B-Instruct-2507-NVFP4', 'host': '0.0.0.0', 'model': 'Benasd/Qwen3-30B-A3B-Instruct-2507-NVFP4', 'max_model_len': 8192}
(APIServer pid=20863) INFO 10-14 09:50:51 [model.py:646] Resolved architecture: Qwen3MoeForCausalLM
(APIServer pid=20863) `torch_dtype` is deprecated! Use `dtype` instead!
(APIServer pid=20863) INFO 10-14 09:50:51 [model.py:1721] Using max model len 8192
(APIServer pid=20863) INFO 10-14 09:50:52 [scheduler.py:225] Chunked prefill is enabled with max_num_batched_tokens=2048.
INFO 10-14 09:50:55 [__init__.py:224] Automatically detected platform cuda.
(EngineCore_DP0 pid=21007) INFO 10-14 09:50:58 [core.py:729] Waiting for init message from front-end.
(EngineCore_DP0 pid=21007) INFO 10-14 09:50:58 [core.py:96] Initializing a V1 LLM engine (v0.11.0rc2.dev479+g577d49821) with config: model='Benasd/Qwen3-30B-A3B-Instruct-2507-NVFP4', speculative_config=None, tokenizer='Benasd/Qwen3-30B-A3B-Instruct-2507-NVFP4', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None,trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=8192, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=compressed-tensors, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=Benasd/Qwen3-30B-A3B-Instruct-2507-NVFP4, enable_prefix_caching=True, chunked_prefill_enabled=True, pooler_config=None, compilation_config={'level': 3, 'debug_dump_path': None, 'cache_dir': '', 'backend': 'inductor', 'custom_ops':['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention', 'vllm::sparse_attn_indexer'], 'use_inductor': None, 'compile_sizes': [], 'inductor_compile_config': {'enable_auto_functionalized_v2': False}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 512, 'local_cache_dir': None}
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
(EngineCore_DP0 pid=21007) INFO 10-14 09:50:59 [parallel_state.py:1325] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0
(EngineCore_DP0 pid=21007) INFO 10-14 09:50:59 [topk_topp_sampler.py:58] Using FlashInfer for top-p & top-k sampling.
(EngineCore_DP0 pid=21007) INFO 10-14 09:50:59 [gpu_model_runner.py:2849] Starting to load model Benasd/Qwen3-30B-A3B-Instruct-2507-NVFP4...
(EngineCore_DP0 pid=21007) INFO 10-14 09:50:59 [compressed_tensors_w4a4_nvfp4.py:48] Using flashinfer-cutlass for FP4
(EngineCore_DP0 pid=21007) INFO 10-14 09:50:59 [cuda.py:405] Using Flash Attention backend on V1 engine.
(EngineCore_DP0 pid=21007) INFO 10-14 09:50:59 [nvfp4_moe_support.py:38] Using FlashInfer kernels for CompressedTensorsW4A4MoeMethod.
(EngineCore_DP0 pid=21007) INFO 10-14 09:51:00 [weight_utils.py:419] Using model weights format ['*.safetensors']
Loading safetensors checkpoint shards:   0% Completed | 0/4 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  25% Completed | 1/4 [00:02<00:06,  2.10s/it]
Loading safetensors checkpoint shards:  50% Completed | 2/4 [00:04<00:04,  2.34s/it]
Loading safetensors checkpoint shards:  75% Completed | 3/4 [00:07<00:02,  2.41s/it]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:08<00:00,  2.00s/it]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:08<00:00,  2.12s/it]
(EngineCore_DP0 pid=21007)
(EngineCore_DP0 pid=21007) INFO 10-14 09:51:09 [default_loader.py:314] Loading weights took 8.58 seconds
(EngineCore_DP0 pid=21007) INFO 10-14 09:51:09 [gpu_model_runner.py:2910] Model loading took 25.9092 GiB and 9.599436 seconds
(EngineCore_DP0 pid=21007) INFO 10-14 09:51:17 [backends.py:608] Using cache directory: /root/.cache/vllm/torch_compile_cache/c2d76fe916/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=21007) INFO 10-14 09:51:17 [backends.py:622] Dynamo bytecode transform time: 8.24 s
(EngineCore_DP0 pid=21007) WARNING 10-14 09:51:18 [partition_rules.py:41] Failed to resolve operator for Inductor partition: vllm::mamba_mixer2
(EngineCore_DP0 pid=21007) WARNING 10-14 09:51:18 [partition_rules.py:41] Failed to resolve operator for Inductor partition: vllm::mamba_mixer
(EngineCore_DP0 pid=21007) WARNING 10-14 09:51:18 [partition_rules.py:41] Failed to resolve operator for Inductor partition: vllm::short_conv
(EngineCore_DP0 pid=21007) WARNING 10-14 09:51:18 [partition_rules.py:41] Failed to resolve operator for Inductor partition: vllm::linear_attention
(EngineCore_DP0 pid=21007) WARNING 10-14 09:51:18 [partition_rules.py:41] Failed to resolve operator for Inductor partition: vllm::plamo2_mamba_mixer
(EngineCore_DP0 pid=21007) WARNING 10-14 09:51:18 [partition_rules.py:41] Failed to resolve operator for Inductor partition: vllm::gdn_attention
(EngineCore_DP0 pid=21007) INFO 10-14 09:51:22 [backends.py:206] Directly load the compiled graph(s) for dynamic shape from the cache, took 3.538 s
(EngineCore_DP0 pid=21007) INFO 10-14 09:51:23 [monitor.py:33] torch.compile takes 11.78 s in total
(EngineCore_DP0 pid=21007) INFO 10-14 09:51:24 [gpu_worker.py:315] Available KV cache memory: 1.66 GiB
(EngineCore_DP0 pid=21007) INFO 10-14 09:51:24 [kv_cache_utils.py:1199] GPU KV cache size: 18,144 tokens
(EngineCore_DP0 pid=21007) INFO 10-14 09:51:24 [kv_cache_utils.py:1204] Maximum concurrency for 8,192 tokens per request: 2.21x
(EngineCore_DP0 pid=21007) 2025-10-14 09:51:24,321 - INFO - autotuner.py:256 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(EngineCore_DP0 pid=21007) 2025-10-14 09:51:25,118 - INFO - autotuner.py:262 - flashinfer.jit: [Autotuner]: Autotuning process ends
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|█████████████████████████████████████████████████████████████████████████████████| 67/67 [00:08<00:00,  7.48it/s]
Capturing CUDA graphs (decode, FULL): 100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 35/35 [00:07<00:00,  4.75it/s]
(EngineCore_DP0 pid=21007) INFO 10-14 09:51:42 [gpu_model_runner.py:3820] Graph capturing finished in 17 secs, took 0.26 GiB
(EngineCore_DP0 pid=21007) INFO 10-14 09:51:42 [core.py:242] init engine (profile, create kv cache, warmup model) took 32.79 seconds
(APIServer pid=20863) INFO 10-14 09:51:43 [loggers.py:191] Engine 000: vllm cache_config_info with initialization after num_gpu_blocks is: 1134
(EngineCore_DP0 pid=21007) INFO 10-14 09:51:43 [gc_utils.py:40] GC Debug Config. enabled:False,top_objects:-1
(APIServer pid=20863) INFO 10-14 09:51:43 [api_server.py:1629] Supported tasks: ['generate']
(APIServer pid=20863) WARNING 10-14 09:51:43 [model.py:1574] Default sampling parameters have been overridden by the model's Hugging Face generation config recommended from the model creator. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(APIServer pid=20863) INFO 10-14 09:51:43 [serving_responses.py:146] Using default chat sampling params from model: {'temperature': 0.7, 'top_k': 20, 'top_p': 0.8}
(APIServer pid=20863) INFO 10-14 09:51:43 [serving_chat.py:130] Using default chat sampling params from model: {'temperature': 0.7, 'top_k': 20, 'top_p': 0.8}
(APIServer pid=20863) INFO 10-14 09:51:43 [serving_completion.py:67] Using default completion sampling params from model: {'temperature': 0.7, 'top_k': 20, 'top_p': 0.8}
(APIServer pid=20863) INFO 10-14 09:51:43 [api_server.py:1942] Starting vLLM API server 0 on http://0.0.0.0:8000