[Bug]: sleep level 2 causes gibberish outputs

[qgallouedec]

Your current environment

The output of python collect_env.py

==============================
        System Info
==============================
OS                           : Ubuntu 20.04.6 LTS (x86_64)
GCC version                  : (Ubuntu 10.5.0-1ubuntu1~20.04) 10.5.0
Clang version                : Could not collect
CMake version                : version 3.27.7
Libc version                 : glibc-2.31

==============================
       PyTorch Info
==============================
PyTorch version              : 2.9.0+cu128
Is debug build               : False
CUDA used to build PyTorch   : 12.8
ROCM used to build PyTorch   : N/A

==============================
      Python Environment
==============================
Python version               : 3.12.12 | packaged by Anaconda, Inc. | (main, Oct 21 2025, 20:16:04) [GCC 11.2.0] (64-bit runtime)
Python platform              : Linux-5.15.0-1048-aws-x86_64-with-glibc2.31

==============================
       CUDA / GPU Info
==============================
Is CUDA available            : True
CUDA runtime version         : 12.1.105
CUDA_MODULE_LOADING set to   : 
GPU models and configuration : GPU 0: NVIDIA H100 80GB HBM3
Nvidia driver version        : 575.57.08
cuDNN version                : Could not collect
HIP runtime version          : N/A
MIOpen runtime version       : N/A
Is XNNPACK available         : True

==============================
          CPU Info
==============================
Architecture:                       x86_64
CPU op-mode(s):                     32-bit, 64-bit
Byte Order:                         Little Endian
Address sizes:                      48 bits physical, 48 bits virtual
CPU(s):                             96
On-line CPU(s) list:                0-95
Thread(s) per core:                 1
Core(s) per socket:                 48
Socket(s):                          2
NUMA node(s):                       2
Vendor ID:                          AuthenticAMD
CPU family:                         25
Model:                              1
Model name:                         AMD EPYC 7R13 Processor
Stepping:                           1
CPU MHz:                            3491.190
BogoMIPS:                           5299.99
Hypervisor vendor:                  KVM
Virtualization type:                full
L1d cache:                          3 MiB
L1i cache:                          3 MiB
L2 cache:                           48 MiB
L3 cache:                           384 MiB
NUMA node0 CPU(s):                  0-47
NUMA node1 CPU(s):                  48-95
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit:        Not affected
Vulnerability L1tf:                 Not affected
Vulnerability Mds:                  Not affected
Vulnerability Meltdown:             Not affected
Vulnerability Mmio stale data:      Not affected
Vulnerability Retbleed:             Not affected
Vulnerability Spec rstack overflow: Mitigation; safe RET
Vulnerability Spec store bypass:    Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:           Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:           Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP always-on, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds:                Not affected
Vulnerability Tsx async abort:      Not affected
Flags:                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf tsc_known_freq pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch topoext perfctr_core invpcid_single ssbd ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru wbnoinvd arat npt nrip_save vaes vpclmulqdq rdpid

==============================
Versions of relevant libraries
==============================
[pip3] flashinfer-python==0.5.2
[pip3] mypy_extensions==1.1.0
[pip3] numpy==2.2.6
[pip3] nvidia-cublas-cu12==12.8.4.1
[pip3] nvidia-cuda-cupti-cu12==12.8.90
[pip3] nvidia-cuda-nvrtc-cu12==12.8.93
[pip3] nvidia-cuda-runtime-cu12==12.8.90
[pip3] nvidia-cudnn-cu12==9.10.2.21
[pip3] nvidia-cudnn-frontend==1.16.0
[pip3] nvidia-cufft-cu12==11.3.3.83
[pip3] nvidia-cufile-cu12==1.13.1.3
[pip3] nvidia-curand-cu12==10.3.9.90
[pip3] nvidia-cusolver-cu12==11.7.3.90
[pip3] nvidia-cusparse-cu12==12.5.8.93
[pip3] nvidia-cusparselt-cu12==0.7.1
[pip3] nvidia-cutlass-dsl==4.3.0
[pip3] nvidia-ml-py==13.580.82
[pip3] nvidia-nccl-cu12==2.27.5
[pip3] nvidia-nvjitlink-cu12==12.8.93
[pip3] nvidia-nvshmem-cu12==3.3.20
[pip3] nvidia-nvtx-cu12==12.8.90
[pip3] pyzmq==27.1.0
[pip3] torch==2.9.0
[pip3] torchaudio==2.9.0
[pip3] torchvision==0.24.0
[pip3] transformers==4.57.2
[pip3] triton==3.5.0
[conda] flashinfer-python         0.5.2                    pypi_0    pypi
[conda] numpy                     2.2.6                    pypi_0    pypi
[conda] nvidia-cublas-cu12        12.8.4.1                 pypi_0    pypi
[conda] nvidia-cuda-cupti-cu12    12.8.90                  pypi_0    pypi
[conda] nvidia-cuda-nvrtc-cu12    12.8.93                  pypi_0    pypi
[conda] nvidia-cuda-runtime-cu12  12.8.90                  pypi_0    pypi
[conda] nvidia-cudnn-cu12         9.10.2.21                pypi_0    pypi
[conda] nvidia-cudnn-frontend     1.16.0                   pypi_0    pypi
[conda] nvidia-cufft-cu12         11.3.3.83                pypi_0    pypi
[conda] nvidia-cufile-cu12        1.13.1.3                 pypi_0    pypi
[conda] nvidia-curand-cu12        10.3.9.90                pypi_0    pypi
[conda] nvidia-cusolver-cu12      11.7.3.90                pypi_0    pypi
[conda] nvidia-cusparse-cu12      12.5.8.93                pypi_0    pypi
[conda] nvidia-cusparselt-cu12    0.7.1                    pypi_0    pypi
[conda] nvidia-cutlass-dsl        4.3.0                    pypi_0    pypi
[conda] nvidia-ml-py              13.580.82                pypi_0    pypi
[conda] nvidia-nccl-cu12          2.27.5                   pypi_0    pypi
[conda] nvidia-nvjitlink-cu12     12.8.93                  pypi_0    pypi
[conda] nvidia-nvshmem-cu12       3.3.20                   pypi_0    pypi
[conda] nvidia-nvtx-cu12          12.8.90                  pypi_0    pypi
[conda] pyzmq                     27.1.0          py312hcf8288c_1  
[conda] torch                     2.9.0                    pypi_0    pypi
[conda] torchaudio                2.9.0                    pypi_0    pypi
[conda] torchvision               0.24.0                   pypi_0    pypi
[conda] transformers              4.57.2                   pypi_0    pypi
[conda] triton                    3.5.0                    pypi_0    pypi

==============================
         vLLM Info
==============================
ROCM Version                 : Could not collect
vLLM Version                 : 0.11.2
vLLM Build Flags:
  CUDA Archs: Not Set; ROCm: Disabled
GPU Topology:
        GPU0    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      0-10    0               N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

==============================
     Environment Variables
==============================
LD_LIBRARY_PATH=/fsx/qgallouedec/miniconda3/envs/trl/lib/python3.12/site-packages/nvidia/nvjitlink/lib:/fsx/qgallouedec/miniconda3/envs/trl/lib/python3.12/site-packages/nvidia/nvjitlink/lib:/opt/amazon/efa/lib:/opt/amazon/openmpi/lib:/opt/aws-ofi-nccl/lib:/usr/local/cuda-12.1/lib:/usr/local/cuda-12.1/lib64:/usr/local/cuda-12.1:/usr/local/cuda-12.1/targets/x86_64-linux/lib/:/usr/local/cuda-12.1/extras/CUPTI/lib64:/usr/local/lib:/usr/lib:/fsx/qgallouedec/miniconda3/envs/trl/lib/python3.12/site-packages/nvidia/nvjitlink/lib:/opt/amazon/efa/lib:/opt/amazon/openmpi/lib:/opt/aws-ofi-nccl/lib:/usr/local/cuda-12.1/lib:/usr/local/cuda-12.1/lib64:/usr/local/cuda-12.1:/usr/local/cuda-12.1/targets/x86_64-linux/lib/:/usr/local/cuda-12.1/extras/CUPTI/lib64:/usr/local/lib:/usr/lib::/opt/amazon/openmpi/lib:/opt/amazon/efa/lib:/opt/amazon/openmpi/lib:/opt/amazon/efa/lib
VLLM_LOGGING_LEVEL=ERROR
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1

🐛 Describe the bug

When using the sleep mode level 2, the model produces gibberish completions:

from vllm import LLM

llm = LLM(model="openai/gpt-oss-20b", enable_sleep_mode=True)

llm.sleep(level=2)
llm.wake_up()

prompts = [[{"role": "user", "content": "Where is the Machu Picchu located?"}]]
outputs = llm.chat(prompts)
print(repr(outputs[0].outputs[0].text))

'ocado \'" chemical optimal UriWord Beef nwanyị Mehmet性质Usuarioಡೆಯ profanity դեպիSleep Columbia'

notes:

• with sleep level = 1, there is no such issue:

  'analysisUser asks location. Need to answer: Machu Picchu located'
  

• the same issue occurs with version 0.10.2, 0.11.0, 0.11.1 and 0.11.2
• the same issue occurs with transformers 4.57.0 and 5.0.0.dev0
• the same issue occurs with models Qwen3 and GPT-OSS, so I guess it affects all models.
• model_impl="transformers" doesn't solve the issue

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[david6666666]

https://docs.vllm.ai/en/latest/features/sleep_mode/#offline-inference, maybe llm.wake_up() lack of tags, llm.wake_up() has nothing to reload

[qgallouedec]

Hi @qgallouedec , same issue: #25800, but I have submitted fix PR, waitting maintainer to Audit. see this PR: #28783

I think this issue is different. There isn't FP8 quantization involved here.

https://docs.vllm.ai/en/latest/features/sleep_mode/#offline-inference, maybe llm.wake_up() lack of tags, llm.wake_up() has nothing to reload

No, llm.wake_up wakes up everything by default.

[david6666666]

Hi @qgallouedec , same issue: #25800, but I have submitted fix PR, waitting maintainer to Audit. see this PR: #28783

I think this issue is different. There isn't FP8 quantization involved here.

https://docs.vllm.ai/en/latest/features/sleep_mode/#offline-inference, maybe llm.wake_up() lack of tags, llm.wake_up() has nothing to reload

No, llm.wake_up wakes up everything by default.

I test in main branch. llm.wake_up llm.wake_up wakes up everything by default.

(EngineCore_DP0 pid=104939) INFO 11-25 03:21:48 [gpu_worker.py:123] Sleep mode freed 71.09 GiB memory, 1.58 GiB memory is still in use.
(EngineCore_DP0 pid=104939) INFO 11-25 03:21:48 [abstract.py:306] It took 0.124753 seconds to fall asleep.
(EngineCore_DP0 pid=104939) INFO 11-25 03:21:48 [abstract.py:324] It took 0.140853 seconds to wake up tags {'kv_cache', 'weights'}.
INFO 11-25 03:21:51 [chat_utils.py:574] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
WARNING 11-25 03:21:52 [model.py:1568] Default sampling parameters have been overridden by the model's Hugging Face generation config recommended from the model creator. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
Adding requests: 100%|███████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 1805.55it/s]
Processed prompts: 100%|███████| 1/1 [00:00<00:00, 20.04it/s, est. speed input: 341.64 toks/s, output: 321.43 toks/s]
'Lon")). exem newValue deleting gap每一位peated.Field Legend� ))\n\n⁅𝔱 Jwtfulness'

[david6666666]

Otherwise I use offline-inference example is right

# Sleep level 2
# Put the engine to sleep (level=2: discard both weights and KV cache)
llm.sleep(level=2)

# Reallocate weights memory only
llm.wake_up(tags=["weights"])

# Load weights in-place
llm.collective_rpc("reload_weights")

# Reallocate KV cache
llm.wake_up(tags=["kv_cache"])

(EngineCore_DP0 pid=105994) INFO 11-25 03:26:13 [abstract.py:306] It took 0.052879 seconds to fall asleep.
(EngineCore_DP0 pid=105994) INFO 11-25 03:26:13 [abstract.py:324] It took 0.067806 seconds to wake up tags ['weights'].
(EngineCore_DP0 pid=105994) INFO 11-25 03:26:13 [gpu_model_runner.py:3436] Reloading weights inplace...
(EngineCore_DP0 pid=105994) INFO 11-25 03:26:14 [weight_utils.py:481] No model.safetensors.index.json found in remote.
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  5.08it/s]
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  5.07it/s]
(EngineCore_DP0 pid=105994) 
(EngineCore_DP0 pid=105994) INFO 11-25 03:26:14 [default_loader.py:308] Loading weights took 0.23 seconds
(EngineCore_DP0 pid=105994) INFO 11-25 03:26:14 [abstract.py:324] It took 0.016932 seconds to wake up tags ['kv_cache'].
INFO 11-25 03:26:17 [chat_utils.py:574] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
WARNING 11-25 03:26:17 [model.py:1568] Default sampling parameters have been overridden by the model's Hugging Face generation config recommended from the model creator. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
Adding requests: 100%|███████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 1848.53it/s]
Processed prompts: 100%|███████| 1/1 [00:00<00:00, 19.99it/s, est. speed input: 340.83 toks/s, output: 320.68 toks/s]
'<think>\nOkay, the user is asking where Machu Picchu is located.'

[qgallouedec]

It appears that the issue arises because llm.collective_rpc("reload_weights") must be called after llm.wake_up() when level=2. IMO, this isn’t very intuitive or user-friendly. I’d recommend either triggering this call automatically or raising a warning (ideally even an error) when generation starts without the weights having been reloaded.

[david6666666]

It appears that the issue arises because llm.collective_rpc("reload_weights") must be called after llm.wake_up() when level=2. IMO, this isn’t very intuitive or user-friendly. I’d recommend either triggering this call automatically or raising a warning (ideally even an error) when generation starts without the weights having been reloaded.

maybe cc @22quinn , ask his adv

[hmellor]

The blog on this feature states:

For Level 2 sleep, you must call reload_weights and reset_prefix_cache after waking. Level 1 sleep doesn’t require these extra steps.

So I think this is expected behaviour, we just need to:

• Decide if it's the best UX
• Document it better

cc @tanpinsiang @tjtanaa