[Bug]: Phi-4-Multimodal Attribute error LoRALRUCache

[lhcavalcanti]

🐛 Describe the bug

vLLM Engine Version 0.

Parameters:

{"tensor_parallel_size": 1, "limit_mm_per_prompt": {"image": 2}, "max_seq_len_to_capture": 131072, "enable_lora": true, "max_loras":1, "max_lora_rank": 320, "lora_modules": {"vision": "vision-lora"}}

Attribute error in LoRALRUCache

``` INFO 04-14 05:43:09 [__init__.py:239] Automatically detected platform cuda. INFO 04-14 05:43:10 [config.py:209] Replacing legacy 'type' key with 'rope_type' INFO 04-14 05:43:17 [config.py:600] This model supports multiple tasks: {'classify', 'generate', 'score', 'reward', 'embed'}. Defaulting to 'generate'. WARNING 04-14 05:43:17 [arg_utils.py:1708] ['Phi4MMForCausalLM'] is not supported by the V1 Engine. Falling back to V0. WARNING 04-14 05:43:17 [arg_utils.py:1581] The model has a long context length (131072). This may causeOOM during the initial memory profiling phase, or result in low performance due to small KV cache size. Consider setting --max-model-len to a smaller value. INFO 04-14 05:43:17 [llm_engine.py:242] Initializing a V0 LLM engine (v0.8.3) with config: model='/models/Phi-4-multimodal-instruct', speculative_config=None, tokenizer='/models/Phi-4-multimodal-instruct', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=131072, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='xgrammar', reasoning_backend=None), observability_config=ObservabilityConfig(show_hidden_metrics=False, otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=None, served_model_name=default-model, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=False, chunked_prefill_enabled=False, use_async_output_proc=True, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"splitting_ops":[],"compile_sizes":[],"cudagraph_capture_sizes":[256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"max_capture_size":256}, use_cached_outputs=False, INFO 04-14 05:43:18 [cuda.py:292] Using Flash Attention backend. INFO 04-14 05:43:19 [parallel_state.py:957] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0 INFO 04-14 05:43:19 [model_runner.py:1110] Starting to load model /models/Phi-4-multimodal-instruct... INFO 04-14 05:43:19 [cuda.py:266] Cannot use FlashAttention-2 backend for head size 72. INFO 04-14 05:43:19 [cuda.py:289] Using XFormers backend. ```

INFO 04-14 05:43:22 [model_runner.py:1146] Model loading took 9.7031 GiB and 2.520181 seconds
[rank0]: Traceback (most recent call last):
[rank0]:   File "/code/score.py", line 260, in <module>
[rank0]:     engine = AsyncLLMEngine.from_engine_args(
[rank0]:   File "/opt/miniconda/envs/python39/lib/python3.9/site-packages/vllm/engine/async_llm_engine.py", line 680, in from_engine_args
[rank0]:     return async_engine_cls.from_vllm_config(
[rank0]:   File "/opt/miniconda/envs/python39/lib/python3.9/site-packages/vllm/engine/async_llm_engine.py", line 653, in from_vllm_config
[rank0]:     return cls(
[rank0]:   File "/opt/miniconda/envs/python39/lib/python3.9/site-packages/vllm/engine/async_llm_engine.py", line 608, in __init__
[rank0]:     self.engine = self._engine_class(*args, **kwargs)
[rank0]:   File "/opt/miniconda/envs/python39/lib/python3.9/site-packages/vllm/engine/async_llm_engine.py", line 267, in __init__
[rank0]:     super().__init__(*args, **kwargs)
[rank0]:   File "/opt/miniconda/envs/python39/lib/python3.9/site-packages/vllm/engine/llm_engine.py", line 284, in __init__
[rank0]:     self._initialize_kv_caches()
[rank0]:   File "/opt/miniconda/envs/python39/lib/python3.9/site-packages/vllm/engine/llm_engine.py", line 433, in _initialize_kv_caches
[rank0]:     self.model_executor.determine_num_available_blocks())
[rank0]:   File "/opt/miniconda/envs/python39/lib/python3.9/site-packages/vllm/executor/executor_base.py", line 103, in determine_num_available_blocks
[rank0]:     results = self.collective_rpc("determine_num_available_blocks")
[rank0]:   File "/opt/miniconda/envs/python39/lib/python3.9/site-packages/vllm/executor/uniproc_executor.py", line 56, in collective_rpc
[rank0]:     answer = run_method(self.driver_worker, method, args, kwargs)
[rank0]:   File "/opt/miniconda/envs/python39/lib/python3.9/site-packages/vllm/utils.py", line 2347, in run_method
[rank0]:     return func(*args, **kwargs)
[rank0]:   File "/opt/miniconda/envs/python39/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
[rank0]:     return func(*args, **kwargs)
[rank0]:   File "/opt/miniconda/envs/python39/lib/python3.9/site-packages/vllm/worker/worker.py", line 229, in determine_num_available_blocks
[rank0]:     self.model_runner.profile_run()
[rank0]:   File "/opt/miniconda/envs/python39/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
[rank0]:     return func(*args, **kwargs)
[rank0]:   File "/opt/miniconda/envs/python39/lib/python3.9/site-packages/vllm/worker/model_runner.py", line 1243, in profile_run
[rank0]:     self._dummy_run(max_num_batched_tokens, max_num_seqs)
[rank0]:   File "/opt/miniconda/envs/python39/lib/python3.9/site-packages/vllm/worker/model_runner.py", line 1369, in _dummy_run
[rank0]:     self.execute_model(model_input, kv_caches, intermediate_tensors)
[rank0]:   File "/opt/miniconda/envs/python39/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
[rank0]:     return func(*args, **kwargs)
[rank0]:   File "/opt/miniconda/envs/python39/lib/python3.9/site-packages/vllm/worker/model_runner.py", line 1697, in execute_model
[rank0]:     self.set_active_loras(model_input.lora_requests,
[rank0]:   File "/opt/miniconda/envs/python39/lib/python3.9/site-packages/vllm/worker/model_runner.py", line 1385, in set_active_loras
[rank0]:     self.lora_manager.set_active_adapters(lora_requests, lora_mapping)
[rank0]:   File "/opt/miniconda/envs/python39/lib/python3.9/site-packages/vllm/lora/worker_manager.py", line 167, in set_active_adapters
[rank0]:     set_active_adapters_worker(requests, mapping, self._apply_adapters,
[rank0]:   File "/opt/miniconda/envs/python39/lib/python3.9/site-packages/vllm/adapter_commons/utils.py", line 54, in set_active_adapters_worker
[rank0]:     apply_adapters_func(requests)
[rank0]:   File "/opt/miniconda/envs/python39/lib/python3.9/site-packages/vllm/lora/worker_manager.py", line 227, in _apply_adapters
[rank0]:     self.add_adapter(lora)
[rank0]:   File "/opt/miniconda/envs/python39/lib/python3.9/site-packages/vllm/lora/worker_manager.py", line 250, in add_adapter
[rank0]:     self._adapter_manager.activate_adapter(lora_request.lora_int_id)
[rank0]:   File "/opt/miniconda/envs/python39/lib/python3.9/site-packages/vllm/lora/models.py", line 752, in activate_adapter
[rank0]:     self._active_adapters.touch(lora_id)
[rank0]:   File "/opt/miniconda/envs/python39/lib/python3.9/site-packages/vllm/utils.py", line 275, in touch
[rank0]:     self._LRUCache__update(key)  # type: ignore
[rank0]: AttributeError: 'LoRALRUCache' object has no attribute '_LRUCache__update'
INFO 04-14 05:58:52 [multiproc_worker_utils.py:124] Killing local vLLM worker processes
[rank0]:[W414 05:58:52.627573283 ProcessGroupNCCL.cpp:1496] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
/opt/miniconda/envs/python39/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown

Your current environment

``` INFO 04-14 05:48:25 [__init__.py:239] Automatically detected platform cuda. Collecting environment information... /opt/miniconda/envs/python39/lib/python3.9/site-packages/_distutils_hack/__init__.py:33: UserWarning: Setuptools is replacing distutils. warnings.warn("Setuptools is replacing distutils.") PyTorch version: 2.6.0+cu124 Is debug build: False CUDA used to build PyTorch: 12.4 ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.5 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.31

Python version: 3.9.18 (main, Sep 11 2023, 13:41:44) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-6.8.0-1026-azure-x86_64-with-glibc2.31
Is CUDA available: True
CUDA runtime version: 12.1.66
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration:
GPU 0: NVIDIA A100 80GB PCIe
GPU 1: NVIDIA A100 80GB PCIe
GPU 2: NVIDIA A100 80GB PCIe
GPU 3: NVIDIA A100 80GB PCIe

Nvidia driver version: 550.120
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.8.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.8.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.8.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.8.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.8.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.8.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.8.1
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 48 bits physical, 48 bits virtual
CPU(s): 96
On-line CPU(s) list: 0-95
Thread(s) per core: 1
Core(s) per socket: 48
Socket(s): 2
NUMA node(s): 4
Vendor ID: AuthenticAMD
CPU family: 25
Model: 1
Model name: AMD EPYC 7V13 64-Core Processor
Stepping: 1
CPU MHz: 2445.441
BogoMIPS: 4890.88
Hypervisor vendor: Microsoft
Virtualization type: full
L1d cache: 3 MiB
L1i cache: 3 MiB
L2 cache: 48 MiB
L3 cache: 384 MiB
NUMA node0 CPU(s): 0-23
NUMA node1 CPU(s): 24-47
NUMA node2 CPU(s): 48-71
NUMA node3 CPU(s): 72-95
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Vulnerable: Safe RET, no microcode
Vulnerability Spec store bypass: Vulnerable
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves user_shstk clzero xsaveerptr rdpru arat umip vaes vpclmulqdq rdpid fsrm

Versions of relevant libraries:
[pip3] numpy==2.0.2
[pip3] nvidia-cublas-cu12==12.4.5.8
[pip3] nvidia-cuda-cupti-cu12==12.4.127
[pip3] nvidia-cuda-nvrtc-cu12==12.4.127
[pip3] nvidia-cuda-runtime-cu12==12.4.127
[pip3] nvidia-cudnn-cu12==9.1.0.70
[pip3] nvidia-cufft-cu12==11.2.1.3
[pip3] nvidia-curand-cu12==10.3.5.147
[pip3] nvidia-cusolver-cu12==11.6.1.9
[pip3] nvidia-cusparse-cu12==12.3.1.170
[pip3] nvidia-cusparselt-cu12==0.6.2
[pip3] nvidia-nccl-cu12==2.21.5
[pip3] nvidia-nvjitlink-cu12==12.4.127
[pip3] nvidia-nvtx-cu12==12.4.127
[pip3] pyzmq==26.4.0
[pip3] torch==2.6.0
[pip3] torchaudio==2.6.0
[pip3] torchvision==0.21.0
[pip3] transformers==4.51.2
[pip3] triton==3.2.0
[conda] No relevant packages
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.8.3
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0 GPU1 GPU2 GPU3 NIC0 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X NV12 SYS SYS NODE 0-23 0 N/A
GPU1 NV12 X SYS SYS SYS 24-47 1 N/A
GPU2 SYS SYS X NV12 SYS 48-71 2 N/A
GPU3 SYS SYS NV12 X SYS 72-95 3 N/A
NIC0 NODE SYS SYS SYS X

Legend:

X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks

NIC Legend:

NIC0: mlx5_0

NVIDIA_VISIBLE_DEVICES=all
CUBLAS_VERSION=12.1.0.26
NVIDIA_REQUIRE_CUDA=cuda>=9.0
CUDA_CACHE_DISABLE=1
NCCL_VERSION=2.17.1
NVIDIA_DRIVER_CAPABILITIES=compute,utility,video
NVIDIA_PRODUCT_NAME=Triton Server
CUDA_VERSION=12.1.0.023
CUDNN_VERSION=8.8.1.3+cuda12.0
NVIDIA_TRITON_SERVER_VERSION=23.03
LD_LIBRARY_PATH=/opt/tritonserver/backends/onnxruntime:/usr/local/cuda/compat/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/usr/local/cuda-11/lib64
NVIDIA_BUILD_ID=56086596
CUDA_DRIVER_VERSION=530.30.02
NVIDIA_REQUIRE_JETPACK_HOST_MOUNTS=
NCCL_CUMEM_ENABLE=0
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
CUDA_MODULE_LOADING=LAZY

</details>

[jeejeelee]

I used the latest main branch with the example code and was not able to reproduce your issue.

[JH-BK]

However, the LoRALRUCache object does not implement the _LRUCache__update method, and this method cannot be found anywhere in the repository. This leads to an AttributeError when attempting to use a LoRA adapter.

[jeejeelee]

_LRUCache__update uses python's name mangling feature.
A possible reason is that you implemented a custom LRUCache class but didn't implement the __update method.

[lhcavalcanti]

Interesting, I did not create a custom LRUCache.

@jeejeelee, could you share packages version you are using?

My code is pretty simple:

    engine_args = AsyncEngineArgs.from_cli_args(args)
    engine = AsyncLLMEngine.from_engine_args(
        engine_args, 
        usage_context=UsageContext.API_SERVER,
        stat_loggers={"Geneva": CustomStatsLogger()}
    )

    model_config = asyncio.run(get_model_config(engine))

    base_model_paths = [BaseModelPath(name=args.served_model_name, model_path=args.model)]
    resolved_chat_template = load_chat_template(args.chat_template)
    logger.info("Using supplied chat template:\n%s", resolved_chat_template)

    serving_model = OpenAIServingModels(
        engine_client=engine,
        model_config=model_config,
        base_model_paths=base_model_paths,
        lora_modules=args.lora_modules,
        prompt_adapters=args.prompt_adapters,
    )

    asyncio.run(serving_model.init_static_loras())

    openai_serving_chat = OpenAIServingChat(
        engine,
        model_config,
        serving_model,
        args.response_role,
        request_logger=None,
        chat_template=args.chat_template,
        chat_template_content_format=args.chat_template_content_format,
    )

I'm using packages below:

Package                           Version
--------------------------------- -------------
accelerate                        1.6.0
aiohappyeyeballs                  2.6.1
aiohttp                           3.11.16
aiosignal                         1.3.2
airportsdata                      20250224
annotated-types                   0.7.0
anyio                             4.9.0
astor                             0.8.1
async-timeout                     5.0.1
attrs                             25.3.0
backoff                           2.2.1
blake3                            1.0.4
blinker                           1.9.0
cachetools                        6.0.0b3
certifi                           2025.1.31
cffi                              1.17.1
charset-normalizer                3.4.1
click                             8.1.8
cloudpickle                       3.1.1
compressed-tensors                0.9.2
cryptography                      44.0.2
cupy-cuda12x                      13.4.1
defusedxml                        0.8.0rc2
depyf                             0.18.0
dill                              0.3.9
diskcache                         5.6.3
distro                            1.9.0
dnspython                         2.7.0
einops                            0.8.1
email_validator                   2.2.0
exceptiongroup                    1.2.2
fastapi                           0.115.12
fastapi-cli                       0.0.7
fastrlock                         0.8.3
filelock                          3.18.0
Flask                             2.3.3
fluent-logger                     0.11.1
frozenlist                        1.5.0
fsspec                            2025.3.2
gguf                              0.10.0
gunicorn                          23.0.0
h11                               0.14.0
httpcore                          1.0.8
httptools                         0.6.4
httpx                             0.28.1
huggingface-hub                   0.30.2
idna                              3.10
importlib_metadata                8.6.1
interegular                       0.3.3
itsdangerous                      2.2.0
Jinja2                            3.1.6
jiter                             0.9.0
jsonschema                        4.23.0
jsonschema-specifications         2024.10.1
lark                              1.2.2
llguidance                        0.7.14
llvmlite                          0.43.0
lm-format-enforcer                0.10.11
markdown-it-py                    3.0.0
MarkupSafe                        3.0.2
mdurl                             0.1.2
mistral_common                    1.5.4
mpmath                            1.3.0
msgpack                           1.1.0
msgspec                           0.19.0
multidict                         6.0.5
nanobind                          2.6.1
nest-asyncio                      1.6.0
networkx                          3.2.1
ninja                             1.11.1.4
numba                             0.60.0
numpy                             2.0.2
nvidia-cublas-cu12                12.4.5.8
nvidia-cuda-cupti-cu12            12.4.127
nvidia-cuda-nvrtc-cu12            12.4.127
nvidia-cuda-runtime-cu12          12.4.127
nvidia-cudnn-cu12                 9.1.0.70
nvidia-cufft-cu12                 11.2.1.3
nvidia-curand-cu12                10.3.5.147
nvidia-cusolver-cu12              11.6.1.9
nvidia-cusparse-cu12              12.3.1.170
nvidia-cusparselt-cu12            0.6.2
nvidia-nccl-cu12                  2.21.5
nvidia-nvjitlink-cu12             12.4.127
nvidia-nvtx-cu12                  12.4.127
openai                            1.72.0
opencv-python-headless            4.11.0.86
outlines                          0.1.11
outlines_core                     0.1.26
packaging                         24.2
partial-json-parser               0.2.1.1.post5
peft                              0.14.0
pillow                            11.2.0
pip                               23.3
polymer-commonschemas             4.0.12
polymer-core                      4.4.1
polymer-featurestore              4.0.0
polymer-runtime                   4.9.3
prometheus_client                 0.21.1
prometheus-fastapi-instrumentator 7.1.0
propcache                         0.3.1
protobuf                          4.25.3
psutil                            5.9.8
py-cpuinfo                        9.0.0
pycountry                         24.6.1
pycparser                         2.22
pydantic                          2.11.3
pydantic_core                     2.33.1
Pygments                          2.19.1
pylogctx                          1.12
pyOpenSSL                         25.0.0
python-dotenv                     1.1.0
python-json-logger                3.3.0
python-multipart                  0.0.20
PyYAML                            6.0.2
pyzmq                             26.4.0
ray                               2.43.0
referencing                       0.36.2
regex                             2024.11.6
requests                          2.32.3
rich                              14.0.0
rich-toolkit                      0.14.1
rpds-py                           0.24.0
safetensors                       0.5.3
scipy                             1.13.1
sentencepiece                     0.2.0
setuptools                        68.0.0
shellingham                       1.5.4
sniffio                           1.3.1
starlette                         0.46.1
statsd                            4.0.1
sympy                             1.13.1
tiktoken                          0.9.0
tokenizers                        0.21.1
torch                             2.6.0
torchaudio                        2.6.0
torchvision                       0.21.0
tqdm                              4.67.1
transformers                      4.51.2
triton                            3.2.0
typer                             0.15.2
typing_extensions                 4.13.2
typing-inspection                 0.4.0
urllib3                           2.4.0
uvicorn                           0.34.0
uvloop                            0.21.0
vllm                              0.8.3
watchfiles                        1.0.5
websockets                        15.0.1
Werkzeug                          3.1.3
wheel                             0.41.2
xformers                          0.0.29.post2
xgrammar                          0.1.17
yarl                              1.19.0
zipp                              3.21.0

[jeejeelee]

Package                                  Version              Editable project location
---------------------------------------- -------------------- ------------------------------------
absl-py                                  2.1.0
accelerate                               1.0.1
aiohappyeyeballs                         2.4.3
aiohttp                                  3.10.11
aiohttp-cors                             0.7.0
aiosignal                                1.3.1
airportsdata                             20241001
annotated-types                          0.7.0
anyio                                    4.6.2.post1
application_properties                   0.8.2
argcomplete                              3.5.1
astor                                    0.8.1
asttokens                                2.4.1
async-timeout                            4.0.3
attrs                                    24.2.0
audioread                                3.0.1
awscli                                   1.35.23
backoff                                  2.2.1
bitsandbytes                             0.45.3
black                                    24.10.0
blake3                                   1.0.0
boto3                                    1.35.57
botocore                                 1.35.57
bounded-pool-executor                    0.0.3
buildkite-test-collector                 0.1.9
cachetools                               5.5.0
certifi                                  2024.8.30
cffi                                     1.17.1
cfgv                                     3.4.0
chardet                                  5.2.0
charset-normalizer                       3.4.0
clang-format                             18.1.5
click                                    8.1.7
cloudpickle                              3.1.0
cmake                                    3.31.6
codespell                                2.3.0
colorama                                 0.4.6
colorful                                 0.5.6
Columnar                                 1.4.1
compressed-tensors                       0.9.3
contourpy                                1.3.0
cramjam                                  2.9.0
cupy-cuda12x                             13.3.0
cut-cross-entropy                        24.12.3
cycler                                   0.12.1
datamodel-code-generator                 0.26.3
DataProperty                             1.0.1
datasets                                 3.0.2
decorator                                5.1.1
decord                                   0.6.0
Deprecated                               1.2.18
depyf                                    0.18.0
dill                                     0.3.8
diskcache                                5.6.3
distlib                                  0.3.9
distro                                   1.9.0
dnspython                                2.7.0
docstring_parser                         0.16
docutils                                 0.16
einops                                   0.8.0
einx                                     0.3.0
email_validator                          2.2.0
encodec                                  0.1.1
evaluate                                 0.4.3
exceptiongroup                           1.2.2
executing                                2.1.0
fastapi                                  0.115.2
fastapi-cli                              0.0.7
fastparquet                              2024.11.0
fastrlock                                0.8.2
fastsafetensors                          0.1.10
filelock                                 3.16.1
flashinfer-python                        0.2.3                /root/Code/flashinfer
fonttools                                4.54.1
frozendict                               2.4.6
frozenlist                               1.5.0
fsspec                                   2024.9.0
genai-perf                               0.0.8
genson                                   1.3.0
gguf                                     0.14.0
google-api-core                          2.24.0
google-auth                              2.37.0
googleapis-common-protos                 1.66.0
grpcio                                   1.68.1
h11                                      0.14.0
hf_transfer                              0.1.8
hf-xet                                   1.0.0
hiredis                                  3.0.0
httpcore                                 1.0.6
httptools                                0.6.2
httpx                                    0.27.2
huggingface-hub                          0.30.1
humanize                                 4.11.0
identify                                 2.6.6
idna                                     3.10
importlib_metadata                       8.0.0
importlib_resources                      6.4.5
inflect                                  5.6.2
iniconfig                                2.0.0
interegular                              0.3.3
ipdb                                     0.13.13
ipython                                  8.28.0
isort                                    5.13.2
jedi                                     0.19.1
Jinja2                                   3.1.6
jiter                                    0.6.1
jiwer                                    3.0.5
jmespath                                 1.0.1
joblib                                   1.4.2
jsonlines                                4.0.0
jsonschema                               4.23.0
jsonschema-specifications                2024.10.1
kaleido                                  0.2.1
kiwisolver                               1.4.7
lark                                     1.2.2
lazy_loader                              0.4
libnacl                                  2.1.0
librosa                                  0.10.2.post1
linkify-it-py                            2.0.3
llguidance                               0.7.10
llvmlite                                 0.44.0
lm_eval                                  0.4.4
lm-format-enforcer                       0.10.11
lxml                                     5.3.0
Markdown                                 3.7
markdown-it-py                           3.0.0
MarkupSafe                               3.0.2
matplotlib                               3.9.2
matplotlib-inline                        0.1.7
mbstrdecoder                             1.1.3
mdit-py-plugins                          0.4.2
mdurl                                    0.1.2
memray                                   1.15.0
mistral_common                           1.5.4
more-itertools                           10.5.0
mpmath                                   1.3.0
msgpack                                  1.1.0
msgspec                                  0.18.6
multidict                                6.1.0
multiprocess                             0.70.16
mypy                                     1.11.1
mypy-extensions                          1.0.0
nanobind                                 2.6.1
nest-asyncio                             1.6.0
networkx                                 3.2.1
ninja                                    1.11.1.1
nltk                                     3.9.1
nodeenv                                  1.9.1
numba                                    0.61.2
numexpr                                  2.10.1
numpy                                    1.26.4
nvidia-cublas-cu12                       12.4.5.8
nvidia-cuda-cupti-cu12                   12.4.127
nvidia-cuda-nvrtc-cu12                   12.4.127
nvidia-cuda-runtime-cu12                 12.4.127
nvidia-cudnn-cu12                        9.1.0.70
nvidia-cufft-cu12                        11.2.1.3
nvidia-curand-cu12                       10.3.5.147
nvidia-cusolver-cu12                     11.6.1.9
nvidia-cusparse-cu12                     12.3.1.170
nvidia-cusparselt-cu12                   0.6.2
nvidia-ml-py                             12.560.30
nvidia-nccl-cu12                         2.21.5
nvidia-nvjitlink-cu12                    12.4.127
nvidia-nvtx-cu12                         12.4.127
openai                                   1.58.1
opencensus                               0.11.4
opencensus-context                       0.1.3
opencv-python                            4.10.0.84
opencv-python-headless                   4.11.0.86
opentelemetry-api                        1.26.0
opentelemetry-exporter-otlp              1.26.0
opentelemetry-exporter-otlp-proto-common 1.26.0
opentelemetry-exporter-otlp-proto-grpc   1.26.0
opentelemetry-exporter-otlp-proto-http   1.26.0
opentelemetry-proto                      1.26.0
opentelemetry-sdk                        1.26.0
opentelemetry-semantic-conventions       0.47b0
opentelemetry-semantic-conventions-ai    0.4.3
outlines                                 0.1.11
outlines_core                            0.1.26
packaging                                24.1
pandas                                   2.2.3
parso                                    0.8.4
partial-json-parser                      0.2.1.1.post4
pathspec                                 0.12.1
pathvalidate                             3.2.1
patsy                                    1.0.1
peft                                     0.13.2
pexpect                                  4.9.0
pillow                                   10.4.0
pip                                      24.2
platformdirs                             4.3.6
plotly                                   5.24.1
pluggy                                   1.5.0
polib                                    1.2.0
pooch                                    1.8.2
portalocker                              2.10.1
pqdm                                     0.2.0
pre_commit                               4.0.1
prometheus_client                        0.21.0
prometheus-fastapi-instrumentator        7.0.0
prompt_toolkit                           3.0.48
propcache                                0.2.0
proto-plus                               1.25.0
protobuf                                 4.25.6
psutil                                   6.1.0
ptyprocess                               0.7.0
pure_eval                                0.2.3
py                                       1.11.0
py-cpuinfo                               9.0.0
py-spy                                   0.4.0
pyairports                               2.1.1
pyarrow                                  18.0.0
pyasn1                                   0.6.1
pyasn1_modules                           0.4.1
pybind11                                 2.13.6
pycountry                                24.6.1
pycparser                                2.22
pydantic                                 2.9.2
pydantic_core                            2.23.4
Pygments                                 2.18.0
pyinstrument                             5.0.0
pymarkdownlnt                            0.9.26
pyparsing                                3.2.0
pytablewriter                            1.2.0
pytest                                   8.3.3
pytest-asyncio                           0.24.0
pytest-forked                            1.6.0
pytest-mock                              3.14.0
pytest-rerunfailures                     14.0
pytest-shard                             0.1.2
python-dateutil                          2.9.0.post0
python-dotenv                            1.0.1
python-json-logger                       3.2.1
python-multipart                         0.0.20
python-rapidjson                         1.20
pytz                                     2024.2
PyYAML                                   6.0.2
pyzmq                                    26.2.0
RapidFuzz                                3.12.1
ray                                      2.43.0
redis                                    5.2.0
referencing                              0.35.1
regex                                    2024.9.11
requests                                 2.32.3
responses                                0.25.3
rich                                     13.9.4
rich-toolkit                             0.13.2
rouge_score                              0.1.2
rpds-py                                  0.20.1
rsa                                      4.7.2
ruff                                     0.6.5
runai-model-streamer                     0.11.0
runai-model-streamer-s3                  0.11.0
s3transfer                               0.10.3
sacrebleu                                2.4.3
safetensors                              0.4.5
scikit-learn                             1.5.2
scipy                                    1.13.1
sentence-transformers                    3.2.1
sentencepiece                            0.2.0
setuptools                               75.8.0
setuptools-scm                           8.2.0
shellingham                              1.5.4
shtab                                    1.7.1
six                                      1.16.0
smart-open                               7.1.0
sniffio                                  1.3.1
soundfile                                0.12.1
soxr                                     0.5.0.post1
sphinx-lint                              1.0.0
sqlitedict                               2.1.0
stack-data                               0.6.3
starlette                                0.39.2
statsmodels                              0.14.4
sympy                                    1.13.1
tabledata                                1.3.3
tabulate                                 0.9.0
tcolorpy                                 0.1.6
tenacity                                 9.0.0
tensorboard                              2.18.0
tensorboard-data-server                  0.7.2
tensorizer                               2.9.0
textual                                  1.0.0
threadpoolctl                            3.5.0
tiktoken                                 0.7.0
timm                                     1.0.11
tokenizers                               0.21.0
toml                                     0.10.2
tomli                                    2.0.2
toolz                                    1.0.0
torch                                    2.6.0
torch-tb-profiler                        0.4.3
torchaudio                               2.6.0
torchvision                              0.21.0
tqdm                                     4.66.6
tqdm-multiprocess                        0.0.11
traitlets                                5.14.3
transformers                             4.51.1
transformers-stream-generator            0.0.5
triton                                   3.2.0
tritonclient                             2.51.0
trl                                      0.13.0
typeguard                                4.4.1
typepy                                   1.3.2
typer                                    0.15.2
types-PyYAML                             6.0.12.20240917
types-requests                           2.31.0.6
types-setuptools                         75.1.0.20241014
types-urllib3                            1.26.25.14
typing_extensions                        4.12.2
tyro                                     0.9.5
tzdata                                   2024.2
uc-micro-py                              1.0.3
unsloth                                  2024.12.12
unsloth_zoo                              2024.12.7
urllib3                                  2.2.3
uvicorn                                  0.31.1
uvloop                                   0.21.0
vector-quantize-pytorch                  1.21.2
virtualenv                               20.28.0
vllm                                     0.1.dev5816+g63d2705 /root/Code/vllm
vocos                                    0.1.0
watchfiles                               0.24.0
wcwidth                                  0.2.13
websockets                               13.1
Werkzeug                                 3.1.3
wheel                                    0.44.0
word2number                              1.1
wrapt                                    1.17.0
xformers                                 0.0.29.post2
xgrammar                                 0.1.18
xxhash                                   3.5.0
yapf                                     0.32.0
yarl                                     1.17.1
zipp                                     3.20.2
zstandard                                0.23.0

[JH-BK]

Confirmed that this does not happen when using vllm==0.8.0, but it reproduces when using 0.8.3.

[lhcavalcanti]

Interesting, I got an environment that it works with 0.8.3, but I had to fix some dependencies versions:

Working environment:

Package                           Version
--------------------------------- -------------
accelerate                        1.4.0
aiohappyeyeballs                  2.4.6
aiohttp                           3.11.13
aiosignal                         1.3.2
airportsdata                      20250224
annotated-types                   0.7.0
anyio                             4.8.0
astor                             0.8.1
async-timeout                     5.0.1
attrs                             25.1.0
backoff                           2.2.1
blake3                            1.0.4
blinker                           1.9.0
cachetools                        5.5.2
certifi                           2025.1.31
cffi                              1.17.1
charset-normalizer                3.4.1
click                             8.1.8
cloudpickle                       3.1.1
compressed-tensors                0.9.2
cryptography                      44.0.1
cupy-cuda12x                      13.3.0
defusedxml                        0.8.0rc2
depyf                             0.18.0
dill                              0.3.9
diskcache                         5.6.3
distro                            1.9.0
dnspython                         2.7.0
einops                            0.8.1
email_validator                   2.2.0
exceptiongroup                    1.2.2
fastapi                           0.115.8
fastapi-cli                       0.0.7
fastrlock                         0.8.3
filelock                          3.17.0
Flask                             2.3.3
fluent-logger                     0.11.1
frozenlist                        1.5.0
fsspec                            2025.2.0
gguf                              0.10.0
gunicorn                          23.0.0
h11                               0.14.0
httpcore                          1.0.7
httptools                         0.6.4
httpx                             0.28.1
huggingface-hub                   0.30.2
idna                              3.10
importlib_metadata                8.6.1
iniconfig                         2.0.0
interegular                       0.3.3
itsdangerous                      2.2.0
Jinja2                            3.1.5
jiter                             0.8.2
jsonschema                        4.23.0
jsonschema-specifications         2024.10.1
lark                              1.2.2
llguidance                        0.7.14
llvmlite                          0.43.0
lm-format-enforcer                0.10.11
markdown-it-py                    3.0.0
MarkupSafe                        3.0.2
mdurl                             0.1.2
mistral_common                    1.5.4
mpmath                            1.3.0
msgpack                           1.1.0
msgspec                           0.19.0
multidict                         6.0.5
nanobind                          2.6.1
nest-asyncio                      1.6.0
networkx                          3.2.1
ninja                             1.11.1.4
numba                             0.60.0
numpy                             1.26.4
nvidia-cublas-cu12                12.4.5.8
nvidia-cuda-cupti-cu12            12.4.127
nvidia-cuda-nvrtc-cu12            12.4.127
nvidia-cuda-runtime-cu12          12.4.127
nvidia-cudnn-cu12                 9.1.0.70
nvidia-cufft-cu12                 11.2.1.3
nvidia-curand-cu12                10.3.5.147
nvidia-cusolver-cu12              11.6.1.9
nvidia-cusparse-cu12              12.3.1.170
nvidia-cusparselt-cu12            0.6.2
nvidia-nccl-cu12                  2.21.5
nvidia-nvjitlink-cu12             12.4.127
nvidia-nvtx-cu12                  12.4.127
openai                            1.64.0
opencv-python-headless            4.11.0.86
outlines                          0.1.11
outlines_core                     0.1.26
packaging                         24.2
partial-json-parser               0.2.1.1.post5
peft                              0.14.0
pillow                            11.1.0
pip                               23.3
pluggy                            1.5.0
polymer-commonschemas             4.0.12
polymer-core                      4.4.0
polymer-featurestore              4.0.0
polymer-runtime                   4.9.3
prometheus_client                 0.21.1
prometheus-fastapi-instrumentator 7.0.2
propcache                         0.3.0
protobuf                          4.25.3
psutil                            5.9.8
py-cpuinfo                        9.0.0
pybind11                          2.14.0.dev1
pycountry                         24.6.1
pycparser                         2.22
pydantic                          2.11.0a2
pydantic_core                     2.29.0
Pygments                          2.19.1
pylogctx                          1.12
pyOpenSSL                         25.0.0
pytest                            8.3.4
python-dotenv                     1.0.1
python-json-logger                3.3.0
python-multipart                  0.0.20
PyYAML                            6.0.2
pyzmq                             26.2.1
ray                               2.43.0
referencing                       0.36.2
regex                             2024.11.6
requests                          2.32.3
rich                              13.9.4
rich-toolkit                      0.13.2
rpds-py                           0.23.1
safetensors                       0.5.3
scipy                             1.13.1
sentencepiece                     0.2.0
setuptools                        68.0.0
shellingham                       1.5.4
sniffio                           1.3.1
starlette                         0.45.3
statsd                            4.0.1
sympy                             1.13.1
tiktoken                          0.9.0
tokenizers                        0.21.0
tomli                             2.2.1
torch                             2.6.0
torchaudio                        2.6.0
torchvision                       0.21.0
tqdm                              4.67.1
transformers                      4.51.3
triton                            3.2.0
typer                             0.15.1
typing_extensions                 4.12.2
urllib3                           2.3.0
uvicorn                           0.34.0
uvloop                            0.21.0
vllm                              0.8.3
watchfiles                        1.0.4
websockets                        15.0
Werkzeug                          3.1.3
wheel                             0.41.2
xformers                          0.0.29.post2
xgrammar                          0.1.17
yarl                              1.18.3
zipp                              3.21.0

Environment with Problem

Package                           Version
--------------------------------- -------------
accelerate                        1.6.0
aiohappyeyeballs                  2.6.1
aiohttp                           3.11.16
aiosignal                         1.3.2
airportsdata                      20250224
annotated-types                   0.7.0
anyio                             4.9.0
astor                             0.8.1
async-timeout                     5.0.1
attrs                             25.3.0
backoff                           2.2.1
blake3                            1.0.4
blinker                           1.9.0
cachetools                        6.0.0b3
certifi                           2025.1.31
cffi                              1.17.1
charset-normalizer                3.4.1
click                             8.1.8
cloudpickle                       3.1.1
compressed-tensors                0.9.2
cryptography                      44.0.2
cupy-cuda12x                      13.4.1
defusedxml                        0.8.0rc2
depyf                             0.18.0
dill                              0.3.9
diskcache                         5.6.3
distro                            1.9.0
dnspython                         2.7.0
einops                            0.8.1
email_validator                   2.2.0
exceptiongroup                    1.2.2
fastapi                           0.115.12
fastapi-cli                       0.0.7
fastrlock                         0.8.3
filelock                          3.18.0
Flask                             2.3.3
fluent-logger                     0.11.1
frozenlist                        1.5.0
fsspec                            2025.3.2
gguf                              0.10.0
gunicorn                          23.0.0
h11                               0.14.0
httpcore                          1.0.8
httptools                         0.6.4
httpx                             0.28.1
huggingface-hub                   0.30.2
idna                              3.10
importlib_metadata                8.6.1
interegular                       0.3.3
itsdangerous                      2.2.0
Jinja2                            3.1.6
jiter                             0.9.0
jsonschema                        4.23.0
jsonschema-specifications         2024.10.1
lark                              1.2.2
llguidance                        0.7.14
llvmlite                          0.43.0
lm-format-enforcer                0.10.11
markdown-it-py                    3.0.0
MarkupSafe                        3.0.2
mdurl                             0.1.2
mistral_common                    1.5.4
mpmath                            1.3.0
msgpack                           1.1.0
msgspec                           0.19.0
multidict                         6.0.5
nanobind                          2.6.1
nest-asyncio                      1.6.0
networkx                          3.2.1
ninja                             1.11.1.4
numba                             0.60.0
numpy                             2.0.2
nvidia-cublas-cu12                12.4.5.8
nvidia-cuda-cupti-cu12            12.4.127
nvidia-cuda-nvrtc-cu12            12.4.127
nvidia-cuda-runtime-cu12          12.4.127
nvidia-cudnn-cu12                 9.1.0.70
nvidia-cufft-cu12                 11.2.1.3
nvidia-curand-cu12                10.3.5.147
nvidia-cusolver-cu12              11.6.1.9
nvidia-cusparse-cu12              12.3.1.170
nvidia-cusparselt-cu12            0.6.2
nvidia-nccl-cu12                  2.21.5
nvidia-nvjitlink-cu12             12.4.127
nvidia-nvtx-cu12                  12.4.127
openai                            1.72.0
opencv-python-headless            4.11.0.86
outlines                          0.1.11
outlines_core                     0.1.26
packaging                         24.2
partial-json-parser               0.2.1.1.post5
peft                              0.14.0
pillow                            11.2.0
pip                               23.3
polymer-commonschemas             4.0.12
polymer-core                      4.4.1
polymer-featurestore              4.0.0
polymer-runtime                   4.9.3
prometheus_client                 0.21.1
prometheus-fastapi-instrumentator 7.1.0
propcache                         0.3.1
protobuf                          4.25.3
psutil                            5.9.8
py-cpuinfo                        9.0.0
pycountry                         24.6.1
pycparser                         2.22
pydantic                          2.11.3
pydantic_core                     2.33.1
Pygments                          2.19.1
pylogctx                          1.12
pyOpenSSL                         25.0.0
python-dotenv                     1.1.0
python-json-logger                3.3.0
python-multipart                  0.0.20
PyYAML                            6.0.2
pyzmq                             26.4.0
ray                               2.43.0
referencing                       0.36.2
regex                             2024.11.6
requests                          2.32.3
rich                              14.0.0
rich-toolkit                      0.14.1
rpds-py                           0.24.0
safetensors                       0.5.3
scipy                             1.13.1
sentencepiece                     0.2.0
setuptools                        68.0.0
shellingham                       1.5.4
sniffio                           1.3.1
starlette                         0.46.1
statsd                            4.0.1
sympy                             1.13.1
tiktoken                          0.9.0
tokenizers                        0.21.1
torch                             2.6.0
torchaudio                        2.6.0
torchvision                       0.21.0
tqdm                              4.67.1
transformers                      4.51.2
triton                            3.2.0
typer                             0.15.2
typing_extensions                 4.13.2
typing-inspection                 0.4.0
urllib3                           2.4.0
uvicorn                           0.34.0
uvloop                            0.21.0
vllm                              0.8.3
watchfiles                        1.0.5
websockets                        15.0.1
Werkzeug                          3.1.3
wheel                             0.41.2
xformers                          0.0.29.post2
xgrammar                          0.1.17
yarl                              1.19.0
zipp                              3.21.0

[jeejeelee]

When using cachetools ==6.0.0b3, I can reproduce this issue. It should be a problem with the cachetools version.