[Bug]: TypeError: argument 'id': StreamInput must be either an integer or a list of integers

[zhongzq123]

Your current environment

The output of python collect_env.py

Your output of `python collect_env.py` here

🐛 Describe the bug

APIServer pid=10005) ERROR 09-28 05:39:47 [async_llm.py:480] AsyncLLM output_handler failed.
(APIServer pid=10005) ERROR 09-28 05:39:47 [async_llm.py:480] Traceback (most recent call last):
(APIServer pid=10005) ERROR 09-28 05:39:47 [async_llm.py:480] File "/opt/conda/lib/python3.10/site-packages/vllm/v1/engine/async_llm.py", line 457, in output_handler
(APIServer pid=10005) ERROR 09-28 05:39:47 [async_llm.py:480] processed_outputs = output_processor.process_outputs(
(APIServer pid=10005) ERROR 09-28 05:39:47 [async_llm.py:480] File "/opt/conda/lib/python3.10/site-packages/vllm/v1/engine/output_processor.py", line 435, in process_outputs
(APIServer pid=10005) ERROR 09-28 05:39:47 [async_llm.py:480] stop_string = req_state.detokenizer.update(
(APIServer pid=10005) ERROR 09-28 05:39:47 [async_llm.py:480] File "/opt/conda/lib/python3.10/site-packages/vllm/v1/engine/detokenizer.py", line 118, in update (APIServer pid=10005) ERROR 09-28 05:39:47 [async_llm.py:480] self.output_text += self.decode_next(new_token_id) (APIServer pid=10005) ERROR 09-28 05:39:47 [async_llm.py:480] File "/opt/conda/lib/python3.10/site-packages/vllm/v1/engine/detokenizer.py", line 219, in decode_next (APIServer pid=10005) ERROR 09-28 05:39:47 [async_llm.py:480] token = self._protected_step(next_token_id) (APIServer pid=10005) ERROR 09-28 05:39:47 [async_llm.py:480] File "/opt/conda/lib/python3.10/site-packages/vllm/v1/engine/detokenizer.py", line 241, in _protected_step (APIServer pid=10005) ERROR 09-28 05:39:47 [async_llm.py:480] raise e (APIServer pid=10005) ERROR 09-28 05:39:47 [async_llm.py:480] File "/opt/conda/lib/python3.10/site-packages/vllm/v1/engine/detokenizer.py", line 233, in _protected_step (APIServer pid=10005) ERROR 09-28 05:39:47 [async_llm.py:480] token = self.stream.step(self.tokenizer, next_token_id) (APIServer pid=10005) ERROR 09-28 05:39:47 [async_llm.py:480] TypeError: argument 'id': StreamInput must be either an integer or a list of integers (APIServer pid=10005) INFO: 127.0.0.1:49204 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error (APIServer pid=10005) INFO: 127.0.0.1:39430 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
(APIServer pid=10005) INFO: 127.0.0.1:39440 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error (APIServer pid=10005) INFO: 127.0.0.1:39456 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[nicole-lihui]

Could you share a bit more info?
For example, the inference request(like curl), model name, vllm version. -- otherwise I can't repoduce the problem.

[josephrocca]

Hi @nicole-lihui I'm also getting this issue, caused by my use of -dcp 4 parameter. Here's a minimal reproduction (on 4xH200):

uv pip install vllm[flashinfer]==0.10.2 --torch-backend=auto
vllm serve RedHatAI/DeepSeek-R1-0528-quantized.w4a16 --tensor-parallel-size 4 -dcp 4 --served-model-name default --host 0.0.0.0 --port 3000

Then run:

curl 'http://localhost:3000/v1/completions' -H 'content-type: application/json' --data-raw $'{"model":"default","prompt":"<｜begin▁of▁sentence｜><｜User｜>Write a very long story about a cat.<｜Assistant｜><think>\\n","add_special_tokens":false,"temperature":0.6,"max_tokens":2048}'

(APIServer pid=62073) INFO:     Started server process [62073]
(APIServer pid=62073) INFO:     Waiting for application startup.
(APIServer pid=62073) INFO:     Application startup complete.
(APIServer pid=62073) ERROR 10-01 11:34:45 [async_llm.py:480] AsyncLLM output_handler failed.
(APIServer pid=62073) ERROR 10-01 11:34:45 [async_llm.py:480] Traceback (most recent call last):
(APIServer pid=62073) ERROR 10-01 11:34:45 [async_llm.py:480]   File "/root/venv/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 457, in output_handler
(APIServer pid=62073) ERROR 10-01 11:34:45 [async_llm.py:480]     processed_outputs = output_processor.process_outputs(
(APIServer pid=62073) ERROR 10-01 11:34:45 [async_llm.py:480]                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=62073) ERROR 10-01 11:34:45 [async_llm.py:480]   File "/root/venv/lib/python3.12/site-packages/vllm/v1/engine/output_processor.py", line 435, in process_outputs
(APIServer pid=62073) ERROR 10-01 11:34:45 [async_llm.py:480]     stop_string = req_state.detokenizer.update(
(APIServer pid=62073) ERROR 10-01 11:34:45 [async_llm.py:480]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=62073) ERROR 10-01 11:34:45 [async_llm.py:480]   File "/root/venv/lib/python3.12/site-packages/vllm/v1/engine/detokenizer.py", line 118, in update
(APIServer pid=62073) ERROR 10-01 11:34:45 [async_llm.py:480]     self.output_text += self.decode_next(new_token_id)
(APIServer pid=62073) ERROR 10-01 11:34:45 [async_llm.py:480]                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=62073) ERROR 10-01 11:34:45 [async_llm.py:480]   File "/root/venv/lib/python3.12/site-packages/vllm/v1/engine/detokenizer.py", line 219, in decode_next
(APIServer pid=62073) ERROR 10-01 11:34:45 [async_llm.py:480]     token = self._protected_step(next_token_id)
(APIServer pid=62073) ERROR 10-01 11:34:45 [async_llm.py:480]             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=62073) ERROR 10-01 11:34:45 [async_llm.py:480]   File "/root/venv/lib/python3.12/site-packages/vllm/v1/engine/detokenizer.py", line 241, in _protected_step
(APIServer pid=62073) ERROR 10-01 11:34:45 [async_llm.py:480]     raise e
(APIServer pid=62073) ERROR 10-01 11:34:45 [async_llm.py:480]   File "/root/venv/lib/python3.12/site-packages/vllm/v1/engine/detokenizer.py", line 233, in _protected_step
(APIServer pid=62073) ERROR 10-01 11:34:45 [async_llm.py:480]     token = self.stream.step(self.tokenizer, next_token_id)
(APIServer pid=62073) ERROR 10-01 11:34:45 [async_llm.py:480]             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=62073) ERROR 10-01 11:34:45 [async_llm.py:480] TypeError: argument 'id': StreamInput must be either an integer or a list of integers
(APIServer pid=62073) INFO:     127.0.0.1:52742 - "POST /v1/completions HTTP/1.1" 500 Internal Server Error
(APIServer pid=62073) INFO:     Shutting down
(APIServer pid=62073) INFO:     Waiting for application shutdown.
(APIServer pid=62073) INFO:     Application shutdown complete.
(APIServer pid=62073) INFO:     Finished server process [62073]

I'm still digging into this, and I'll update this thread if I find any clues.

[josephrocca]

The issue is gone when installing at the commit where DCP was merged:

• [Feature] Support Decode Context Parallel (DCP) for MLA #23734

Like so:

export VLLM_COMMIT=ac201a0eaf2af032779db3e7ac96f857cbaa5b7a && uv pip install vllm[flashinfer] --torch-backend=auto --extra-index-url https://wheels.vllm.ai/${VLLM_COMMIT}

So I guess this bug was introduced some time in the last few weeks, or it was somehow 'hidden' (i.e. silent bug just messing with output accuracy or something) until recently.

I tested before DeepSeek v3.2 stuff merged (specifically, I tested e47433b) and got the same crash.

And I tested the ~latest commit as of writing (1405f0c), and the server crashes while starting up with dynamic_func() got multiple values for argument 'HEAD_DIM'. I've opened a separate issue to track that:

• [Bug]: dynamic_func() got multiple values for argument 'HEAD_DIM' crash during startup when using -dcp #26022

Unsure who to tag here, apologies if I'm pinging the wrong person here, or if I shouldn't ping anyone for this. CC @LucasWilkinson

[josephrocca]

I could originally consistently reproduce this, but for some reason this has become flaky or maybe unreproducible. I've marked my previous comments as outdated for now, and am investigating - maybe something weird with compile caches, or something I forgot to include when documenting the above minimal reproduction. Will update if I learn anything new.

[cjackal]

It seems I faced the same error; c.f. #26071. I have left some details there.