[Bug]: [v1/core/block_pool.py] Assertion Failure: prev_block.block_hash is not None

[QierLi]

Environment

vLLM V1
T16 single host
LLAMA4 Maverick TP8
Will crash per hours because of this engine failure.

Code Pointer

In cache_full_blocks()

vllm/vllm/v1/core/block_pool.py

Lines 127 to 139 in 055bd39

	if num_cached_blocks == num_full_blocks:
	return
	new_full_blocks = blocks[num_cached_blocks:num_full_blocks]
	assert len(block_hashes) >= num_cached_blocks
	new_block_hashes = block_hashes[num_cached_blocks:]
	# Update the new blocks with the block hashes through the chain.
	if num_cached_blocks == 0:
	prev_block_hash_value = None
	else:
	prev_block = blocks[num_cached_blocks - 1]
	assert prev_block.block_hash is not None
	prev_block_hash_value = prev_block.block_hash.get_hash_value()

Stack Trace

Line numbers may be inaccurate

ERROR EngineCore encountered a fatal error.
ERROR Traceback (most recent call last):
ERROR   File "engine/core.py", line 640, in run_engine_core
ERROR     engine_core.run_busy_loop()
ERROR   File "engine/core.py", line 667, in run_busy_loop
ERROR     self._process_engine_step()
ERROR   File "engine/core.py", line 692, in _process_engine_step
ERROR     outputs, model_executed = self.step_fn()
ERROR   File "engine/core.py", line 280, in step
ERROR     scheduler_output = self.scheduler.schedule()
ERROR   File "core/sched/scheduler.py", line 440, in schedule
ERROR     new_blocks = self.kv_cache_manager.allocate_slots(
ERROR   File "core/kv_cache_manager.py", line 302, in allocate_slots
ERROR     self.coordinator.cache_blocks(
ERROR   File "core/kv_cache_coordinator.py", line 113, in cache_blocks
ERROR     manager.cache_blocks(request, block_hashes, num_computed_tokens)
ERROR   File "core/single_type_kv_cache_manager.py", line 146, in cache_blocks
ERROR     self.block_pool.cache_full_blocks(
ERROR   File "core/block_pool.py", line 138, in cache_full_blocks
ERROR     assert prev_block.block_hash is not None
ERROR AssertionError
ERROR AsyncLLM output_handler failed.
ERROR Traceback (most recent call last):
ERROR   File "engine/async_llm.py", line 379, in output_handler
ERROR     outputs = await engine_core.get_output_async()
ERROR   File "engine/core_client.py", line 764, in get_output_async
ERROR     raise self._format_exception(outputs) from None
ERROR EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root 

[sarckk]

@QierLi can you share the command you are running?

[hansuelijud]

It seems like I encountered the same issue with vllm 0.10.0 running RedHatAI/Llama-4-Maverick-17B-128E-Instruct-quantized.w4a16 on 4xA100 80GB:

ERROR 07-31 12:46:34 [core.py:634] EngineCore encountered a fatal error.
ERROR 07-31 12:46:34 [core.py:634] Traceback (most recent call last):
ERROR 07-31 12:46:34 [core.py:634]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 625, in run_engine_core
ERROR 07-31 12:46:34 [core.py:634]     engine_core.run_busy_loop()
ERROR 07-31 12:46:34 [core.py:634]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 652, in run_busy_loop
ERROR 07-31 12:46:34 [core.py:634]     self._process_engine_step()
ERROR 07-31 12:46:34 [core.py:634]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 677, in _process_engine_step
ERROR 07-31 12:46:34 [core.py:634]     outputs, model_executed = self.step_fn()
ERROR 07-31 12:46:34 [core.py:634]                               ^^^^^^^^^^^^^^
ERROR 07-31 12:46:34 [core.py:634]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 266, in step
ERROR 07-31 12:46:34 [core.py:634]     scheduler_output = self.scheduler.schedule()
ERROR 07-31 12:46:34 [core.py:634]                        ^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-31 12:46:34 [core.py:634]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/sched/scheduler.py", line 440, in schedule
ERROR 07-31 12:46:34 [core.py:634]     new_blocks = self.kv_cache_manager.allocate_slots(
ERROR 07-31 12:46:34 [core.py:634]                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-31 12:46:34 [core.py:634]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/kv_cache_manager.py", line 302, in allocate_slots
ERROR 07-31 12:46:34 [core.py:634]     self.coordinator.cache_blocks(
ERROR 07-31 12:46:34 [core.py:634]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/kv_cache_coordinator.py", line 113, in cache_blocks
ERROR 07-31 12:46:34 [core.py:634]     manager.cache_blocks(request, block_hashes, num_computed_tokens)
ERROR 07-31 12:46:34 [core.py:634]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/single_type_kv_cache_manager.py", line 146, in cache_blocks
ERROR 07-31 12:46:34 [core.py:634]     self.block_pool.cache_full_blocks(
ERROR 07-31 12:46:34 [core.py:634]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/block_pool.py", line 138, in cache_full_blocks
ERROR 07-31 12:46:34 [core.py:634]     assert prev_block.block_hash is not None
ERROR 07-31 12:46:34 [core.py:634]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-31 12:46:34 [core.py:634] AssertionError
ERROR 07-31 12:46:34 [async_llm.py:416] AsyncLLM output_handler failed.
ERROR 07-31 12:46:34 [async_llm.py:416] Traceback (most recent call last):
ERROR 07-31 12:46:34 [async_llm.py:416]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 375, in output_handler
ERROR 07-31 12:46:34 [async_llm.py:416]     outputs = await engine_core.get_output_async()
ERROR 07-31 12:46:34 [async_llm.py:416]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-31 12:46:34 [async_llm.py:416]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 751, in get_output_async
ERROR 07-31 12:46:34 [async_llm.py:416]     raise self._format_exception(outputs) from None
ERROR 07-31 12:46:34 [async_llm.py:416] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
ERROR 07-31 12:46:34 [serving_chat.py:932] Error in chat completion stream generator.
ERROR 07-31 12:46:34 [serving_chat.py:932] Traceback (most recent call last):
ERROR 07-31 12:46:34 [serving_chat.py:932]   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/serving_chat.py", line 497, in chat_completion_stream_generator
ERROR 07-31 12:46:34 [serving_chat.py:932]     async for res in result_generator:
ERROR 07-31 12:46:34 [serving_chat.py:932]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 323, in generate
ERROR 07-31 12:46:34 [serving_chat.py:932]     out = q.get_nowait() or await q.get()
ERROR 07-31 12:46:34 [serving_chat.py:932]                             ^^^^^^^^^^^^^
ERROR 07-31 12:46:34 [serving_chat.py:932]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 57, in get
ERROR 07-31 12:46:34 [serving_chat.py:932]     raise output
ERROR 07-31 12:46:34 [serving_chat.py:932]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 375, in output_handler
ERROR 07-31 12:46:34 [serving_chat.py:932]     outputs = await engine_core.get_output_async()
ERROR 07-31 12:46:34 [serving_chat.py:932]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-31 12:46:34 [serving_chat.py:932]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 751, in get_output_async
ERROR 07-31 12:46:34 [serving_chat.py:932]     raise self._format_exception(outputs) from None
ERROR 07-31 12:46:34 [serving_chat.py:932] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.

I used the official Docker image with following command:

docker run \
     --gpus '"device=0,1,2,3"'\
     --ipc=host \
     --shm-size=10.24g\
     --rm\
     -v "/raid/models/Llama-4-Maverick-17B-128E-Instruct-quantized.w4a16":"/mnt/model/"\
     -v "/home/user/llm-utils/llama4":"/app/plugins/" \
     -e "VLLM_DISABLE_COMPILE_CACHE=1" \
     -p 8080:8000 vllm/vllm-openai:v0.10.0 \
     --host 0.0.0.0\
     --model="/mnt/model/"\
     --served-model-name alpineai/Llama-4\
     --disable-log-requests\
     --tensor-parallel-size 4\
     --gpu-memory-utilization 0.95\
     --override-generation-config='{"attn_temperature_tuning": true}'\
     --enable-auto-tool-choice\
     --chat-template /app/plugins/llama4_pythonic_chat_template.jinja\
     --tool-parser-plugin /app/plugins/llama4_pythonic_tool_parser.py\
     --tool-call-parser llama4_pythonic \
     --max-model-len 130000\
     --max-seq-len-to-capture 130000\
     --limit-mm-per-prompt '{"images": 10}'

[xhejtman]

Same as above, the same model on 2x H100. I believe, I did not see it without enabled tool parser, or maybe it is just coincidence.

[sarckk]

@luccafong wonder if it's related to hybrid KV + local chunked attn somehow?

[sarckk]

@xhejtman @QierLi @hansuelijud could you try with --disable-hybrid-kv-cache-manager to see if that fixes the issue?

[xhejtman]

--disable-hybrid-kv-cache-manager

ok, restarted model with this option and we shall see.

[sarckk]

The error is because we expect the cached prefix blocks to have a valid block hash, but for chunked local attention we seem to run into cases where this is None. When I print out the ref_cnt of the cached blocks when this error is hit, I see negative ref counts, which seems a bit suspicious to me.

[blk.ref_cnt for blk in new_computed_block_list[0]]
[-381, -381, -381, -381, -381, -381, -381, -381]

I suspect something in remove_skipped_blocks method of ChunkedLocalAttentionManager (https://github.com/vllm-project/vllm/blob/main/vllm/v1/core/single_type_kv_cache_manager.py#L481) might need update. I only hit this error when attn chunking is used.

To reproduce the issue:

1. Download Maverick HF ckpt (I used RedHatAI/Llama-4-Scout-17B-16E-Instruct-quantized.w4a16 as suggested above but I think it should work with normal maverick ckpt) and modify local config.json to change attention chunk size to a small number (e.g. 32).

2. If checkout is past [Perf] Disable chunked local attention by default with llama4 #21761 (which disables hybrid KV cache manager by default for llama4), set VLLM_ALLOW_CHUNKED_LOCAL_ATTN_WITH_HYBRID_KV_CACHE=1.

3. Run

vllm serve $MAVERICK_CKPT -tp 4 --max-model-len 130000 --max-seq-len-to-capture 130000 --enforce-eager

4. Send requests via eval:

lm_eval --model local-completions --tasks gsm8k     --model_args model=/home/yhshin/local/models/maverick_quant_redhat,base_url=http://127.0.0.1:8000/v1/completions,num_concurrent=200,max_retries=3,tokenized_requests=False     --limit 1000

If I use --disable-hybrid-kv-cache-manager, I dont run into the error. Neither when attn chunk size is unchanged.

cc @luccafong @heheda12345