Skip to content

[Multimodal][Speculative Decoding]Eagle Eagle3 mm support, enablement on qwen2.5vl #22872

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 43 commits into from
Sep 27, 2025
Merged

[Multimodal][Speculative Decoding]Eagle Eagle3 mm support, enablement on qwen2.5vl #22872

merged 43 commits into from
Sep 27, 2025
+210 −45

Conversation

@david6666666
Copy link
Contributor

@david6666666 david6666666 commented Aug 14, 2025 •
edited by github-actions bot
Loading

Purpose

follow #20788 Eagle Eagle3 mm support, enablement on qwen2.5vl
model is https://huggingface.co/Rayzl/qwen2.5-vl-7b-eagle3-sgl

Test Plan

Test Result

vllm serve \
    /workspace/models/Qwen2.5-VL-7B-Instruct \
    --port 5580 --host 0.0.0.0 \
    --max-num-seqs 128 --dtype bfloat16 --max-model-len=8192  \
    --no-enable-prefix-caching --trust-remote-code -tp 1\
    --speculative-config '{"method": "eagle3", "model": "/workspace/models/qwen2.5-vl-7b-eagle3-sgl", "prefill_token_shift": false, "num_speculative_tokens": 3, "draft_tensor_parallel_size": 1, "max_model_len": 8192}' \
    --num-lookahead-slots=3 \
    --gpu-memory-utilization=0.93

client

vllm bench serve \
--backend openai-chat \
--endpoint-type openai-chat \
--port 5580 \
--host 127.0.0.1 \
--endpoint /v1/chat/completions \
--model /home/models/Qwen2.5-VL-7B-Instruct \
--dataset-name hf \
--dataset-path Lin-Chen/MMStar \
--hf-split val \
--num-prompts 50 \
--max-concurrency 2 \
--temperature 0 \
--top-p 1.0 \
--seed 123

e2e speed up 1.3x
# eagle3
============ Serving Benchmark Result ============
Successful requests:                     50        
Maximum request concurrency:             2         
Benchmark duration (s):                  32.11     
Total input tokens:                      1226      
Total generated tokens:                  3985      
Request throughput (req/s):              1.56      
Output token throughput (tok/s):         124.09    
Peak output token throughput (tok/s):    82.00     
Peak concurrent requests:                6.00      
Total Token throughput (tok/s):          162.26    
---------------Time to First Token----------------
Mean TTFT (ms):                          50.02     
Median TTFT (ms):                        49.18     
P99 TTFT (ms):                           71.19     
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          14.07     
Median TPOT (ms):                        14.27     
P99 TPOT (ms):                           19.15     
---------------Inter-token Latency----------------
Mean ITL (ms):                           24.34     
Median ITL (ms):                         24.75     
P99 ITL (ms):                            25.99     
==================================================


# without eagle3
tip: install termplotlib and gnuplot to plot the metrics
============ Serving Benchmark Result ============
Successful requests:                     50        
Maximum request concurrency:             2         
Benchmark duration (s):                  44.76     
Total input tokens:                      1226      
Total generated tokens:                  4237      
Request throughput (req/s):              1.12      
Output token throughput (tok/s):         94.66     
Peak output token throughput (tok/s):    99.00     
Peak concurrent requests:                5.00      
Total Token throughput (tok/s):          122.05    
---------------Time to First Token----------------
Mean TTFT (ms):                          43.05     
Median TTFT (ms):                        41.76     
P99 TTFT (ms):                           62.82     
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          20.48     
Median TPOT (ms):                        20.49     
P99 TPOT (ms):                           20.64     
---------------Inter-token Latency----------------
Mean ITL (ms):                           20.24     
Median ITL (ms):                         20.46     
P99 ITL (ms):                            21.60     
==================================================

(Optional) Documentation Update

Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

@mergify mergify bot added new-model Requests to new models qwen Related to Qwen models speculative-decoding v1 labels Aug 14, 2025
gemini-code-assist[bot]
gemini-code-assist bot reviewed Aug 14, 2025
View reviewed changes
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request adds support for Eagle and Eagle3 speculative decoding for the Qwen2.5-VL multimodal model. The changes include adding new model files for the Eagle and Eagle3 variants, updating model registries, and modifying tests. I found one issue related to a buggy condition in the weight loading logic for the Eagle model which could lead to incorrect behavior. My feedback addresses this.

@github-actions
Copy link

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

DarkLight1337
DarkLight1337 reviewed Aug 14, 2025
View reviewed changes
tests/v1/e2e/test_spec_decode.py Outdated
@@ -183,6 +191,9 @@ def test_eagle_correctness(

method, model_name, spec_model_name, tp_size = model_setup

if "Qwen2.5-VL" in model_name and attn_backend == "TREE_ATTN":
pytest.skip("TREE ATTN not support Qwen2.5-VL Model yet")
print(f"model_setup={model_setup}")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
print(f"model_setup={model_setup}")

DarkLight1337
DarkLight1337 reviewed Aug 14, 2025
View reviewed changes
Copy link
Member

@DarkLight1337 DarkLight1337 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @22quinn @morgendave can you help review?

@LJH-LBJ LJH-LBJ force-pushed the Eagle-mulitmodal-support-Qwen2.5vl branch from af49ffc to 9d06a8d Compare August 14, 2025 05:11
[V1] support eagle and eagle3 for qwen2_5vl
8874e16 
Signed-off-by: Junhong <liujunhong11@huawei.com>
@LJH-LBJ LJH-LBJ force-pushed the Eagle-mulitmodal-support-Qwen2.5vl branch from e0fd906 to 8874e16 Compare August 14, 2025 12:32
@22quinn
Copy link
Collaborator

22quinn commented Aug 16, 2025

cc @22quinn @morgendave can you help review?

cc spec decode experts @zixi-qi @charlotte12l

@mergify
Copy link

mergify bot commented Aug 20, 2025

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @david6666666.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify bot added the needs-rebase label Aug 20, 2025
@LJH-LBJ LJH-LBJ force-pushed the Eagle-mulitmodal-support-Qwen2.5vl branch from 828fc1e to f8af6b8 Compare August 21, 2025 11:27
Junhong added 6 commits August 21, 2025 19:28
fix bug
b242260 
Signed-off-by: Junhong <liujunhong11@huawei.com>
support M-RoPE in eagle
6b682de 
Signed-off-by: Junhong <liujunhong11@huawei.com>
fix bug for SupportsEagle3 and graph compile
475ce2b 
Signed-off-by: Junhong <liujunhong11@huawei.com>
fix bug
7df533c 
Signed-off-by: Junhong <liujunhong11@huawei.com>
Merge remote-tracking branch 'origin' into Eagle_support_mrope
fdb7cb2
fix bug
ce18a91 
Signed-off-by: Junhong <liujunhong11@huawei.com>
DarkLight1337
DarkLight1337 reviewed Sep 26, 2025
View reviewed changes
opt
cbe0744 
Signed-off-by: Junhong <liujunhong11@huawei.com>
DarkLight1337
DarkLight1337 reviewed Sep 26, 2025
View reviewed changes
opt
e5e29e0 
Signed-off-by: Junhong <liujunhong11@huawei.com>
DarkLight1337
DarkLight1337 approved these changes Sep 26, 2025
View reviewed changes
Copy link
Member

@DarkLight1337 DarkLight1337 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your patience!

@DarkLight1337 DarkLight1337 added the ready ONLY add when PR is ready to merge/full CI is needed label Sep 26, 2025
fix pre-commit
4e24437 
Signed-off-by: Junhong <liujunhong11@huawei.com>
@DarkLight1337
Copy link
Member

PTAL at the failing test

fix bug
c38907b 
Signed-off-by: Junhong <liujunhong11@huawei.com>
@mergify
Copy link

mergify bot commented Sep 27, 2025

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @david6666666.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify bot added the needs-rebase label Sep 27, 2025
@LJH-LBJ
Merge branch 'main' into Eagle-mulitmodal-support-Qwen2.5vl
6096054 
Signed-off-by: Junhong Liu <98734602+LJH-LBJ@users.noreply.github.com>
@mergify mergify bot removed the needs-rebase label Sep 27, 2025
@LJH-LBJ
Copy link
Contributor

LJH-LBJ commented Sep 27, 2025

PTAL at the failing test

skip the test_eagle_correctness, because this flash attention build does not support headdim not being a multiple of 32

@DarkLight1337 DarkLight1337 enabled auto-merge (squash) September 27, 2025 02:26
@DarkLight1337 DarkLight1337 merged commit f1d53d1 into vllm-project:main Sep 27, 2025
52 checks passed
@taoyun951753
Copy link

The online service, deployed with the Qwen2.5-VL-7B model, encountered a corner case that caused the service to crash. The input_ids processed by the propose function in eagle.py generated an image_token_id, resulting in a shape mismatch with mm_embeds.

@taoyun951753
Copy link

mm embds shape: [torch.Size([546, 3584])]
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m before propose input ids: target token ids: tensor([151644, 8948, 198, 2610, 525, 264, 10950, 17847, 13,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151645, 198, 151644, 872, 198, 151652, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151653, 53481, 41683, 17254, 9370, 45930,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 102073, 43815, 8997, 101140, 37945, 104317, 102650, 15946, 64471,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 101114, 1773, 62244, 102650, 15946, 101475, 37945, 53481, 102650,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 101047, 102122, 24968, 62244, 102650, 15946, 101114, 37945, 53481,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 102650, 15946, 104013, 109318, 101070, 8997, 14880, 23031, 2236,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 68805, 99553, 7, 104346, 3837, 100714, 53481, 3837, 99226,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 98237, 53481, 8, 101124, 66017, 3837, 90919, 104346, 14880,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 11622, 2073, 106468, 5373, 101977, 5373, 42140, 99605, 854,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 104787, 1773, 101475, 102122, 15946, 99226, 98237, 53481, 91680,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 102298, 104766, 5373, 44091, 3837, 100714, 53481, 46448, 108593,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 102298, 102122, 106336, 49567, 104449, 1773, 101114, 102122, 15946,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 99226, 98237, 53481, 91680, 102298, 104013, 5373, 104766, 5373,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 102196, 3837, 100714, 53481, 46448, 108593, 102298, 104013, 104363,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 5373, 104766, 99719, 49567, 100405, 104449, 1773, 151645, 198,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151644, 77091, 198, 57218, 112531, 99194, 38035, 17447, 24562,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 100544, 3837, 104013, 109612, 18830, 17323, 71, 3022, 43,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 46944, 112531, 99194, 38035, 17447], device='cuda:0',
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m dtype=torch.int32), next token ids: tensor([ 4913, 111413, 110607, 100010, 105611], device='cuda:0',
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m dtype=torch.int32) ,mm_embeds shape: [torch.Size([546, 3584])]
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m propose input ids: tensor([ 8948, 198, 2610, 525, 264, 10950, 17847, 13, 151645,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 198, 151644, 872, 198, 151652, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151653, 53481, 41683, 17254, 9370, 45930, 102073,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 43815, 8997, 101140, 37945, 104317, 102650, 15946, 64471, 101114,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 1773, 62244, 102650, 15946, 101475, 37945, 53481, 102650, 101047,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 102122, 24968, 62244, 102650, 15946, 101114, 37945, 53481, 102650,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 15946, 104013, 109318, 101070, 8997, 14880, 23031, 2236, 68805,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 99553, 7, 104346, 3837, 100714, 53481, 3837, 99226, 98237,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 53481, 8, 101124, 66017, 3837, 90919, 104346, 14880, 11622,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 2073, 106468, 5373, 101977, 5373, 42140, 99605, 854, 104787,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 1773, 101475, 102122, 15946, 99226, 98237, 53481, 91680, 102298,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 104766, 5373, 44091, 3837, 100714, 53481, 46448, 108593, 102298,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 102122, 106336, 49567, 104449, 1773, 101114, 102122, 15946, 99226,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 98237, 53481, 91680, 102298, 104013, 5373, 104766, 5373, 102196,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 3837, 100714, 53481, 46448, 108593, 102298, 104013, 104363, 5373,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 104766, 99719, 49567, 100405, 104449, 1773, 151645, 198, 151644,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 77091, 198, 4913, 112531, 111413, 38035, 17447, 24562, 100544,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 3837, 110607, 109612, 18830, 100010, 71, 3022, 43, 46944,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 105611, 99194, 38035, 17447, 151655], device='cuda:0',
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m dtype=torch.int32), mm embeds shape :[torch.Size([546, 3584])]
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m utils.py: is_multimodal sum: 547, emb shape 0:546
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:402: masked_scatter_size_check: block: [0,0,0], thread: [0,0,0] Assertion totalElements <= srcSize failed.
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m ERROR 09-29 03:07:32 [dump_input.py:69] Dumping input data for V1 LLM engine (v0.11.0rc2.dev0+g0d9fe260d.d20250926) with config: model='/lhf-model/Ipc/Qwen2.5-VL-7B-Instruct-20250829', speculative_config=SpeculativeConfig(method='eagle3', model='/mnt/sharedLLMs/qwen2.5-vl-7b-eagle3-specforge-v5-epoch1', num_spec_tokens=4), tokenizer='/lhf-model/Ipc/Qwen2.5-VL-7B-Instruct-20250829', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=8192, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=base_model, enable_prefix_caching=True, chunked_prefill_enabled=True, pooler_config=None, compilation_config={"level":3,"debug_dump_path":"","cache_dir":"/root/.cache/vllm/torch_compile_cache/6cd3220e9b","backend":"","custom_ops":[],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output","vllm.mamba_mixer2","vllm.mamba_mixer","vllm.short_conv","vllm.linear_attention","vllm.plamo2_mamba_mixer","vllm.gdn_attention"],"use_inductor":true,"compile_sizes":[],"inductor_compile_config":{"enable_auto_functionalized_v2":false},"inductor_passes":{},"cudagraph_mode":1,"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[512,504,496,488,480,472,464,456,448,440,432,424,416,408,400,392,384,376,368,360,352,344,336,328,320,312,304,296,288,280,272,264,256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"use_inductor_graph_partition":false,"pass_config":{},"max_capture_size":512,"local_cache_dir":"/root/.cache/vllm/torch_compile_cache/6cd3220e9b/rank_0_0/backbone"},
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m ERROR 09-29 03:07:32 [dump_input.py:76] Dumping scheduler output for model execution: SchedulerOutput(scheduled_new_reqs=[NewRequestData(req_id=chatcmpl-c0150128df3f4637864541d650a02fb9,prompt_token_ids_len=687,mm_features=[MultiModalFeatureSpec(data={'image_grid_thw': MultiModalFieldElem(modality='image', key='image_grid_thw', data=tensor([ 1, 42, 52]), field=MultiModalBatchedField()), 'pixel_values': MultiModalFieldElem(modality='image', key='pixel_values', data=tensor([[1.9297, 1.9297, 1.9297, ..., 2.1406, 2.1406, 2.1406],
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m ERROR 09-29 03:07:32 [dump_input.py:76] [1.9297, 1.9297, 1.9297, ..., 2.1406, 2.1406, 2.1406],
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m ERROR 09-29 03:07:32 [dump_input.py:76] [1.9297, 1.9297, 1.9297, ..., 2.1406, 2.1406, 2.1406],
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m ERROR 09-29 03:07:32 [dump_input.py:76] ...,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m ERROR 09-29 03:07:32 [dump_input.py:76] [1.9297, 1.9297, 1.9297, ..., 2.1406, 2.1406, 2.1406],
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m ERROR 09-29 03:07:32 [dump_input.py:76] [1.9297, 1.9297, 1.9297, ..., 2.1406, 2.1406, 2.1406],
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m ERROR 09-29 03:07:32 [dump_input.py:76] [1.9297, 1.9297, 1.9297, ..., 2.1406, 2.1406, 2.1406]],
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m ERROR 09-29 03:07:32 [dump_input.py:76] dtype=torch.bfloat16), field=MultiModalFlatField(slices=[[slice(0, 2184, None)]], dim=0))}, modality='image', identifier='089654d81821ee9d7bbe74606f03e9df82789e90e7bb59dbf5bf3b1df03c527f', mm_position=PlaceholderRange(offset=15, length=546, is_embed=None))],sampling_params=SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.05, temperature=0.0, top_p=1.0, top_k=0, min_p=0.0, seed=None, stop=[], stop_token_ids=[151643], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=128, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, structured_outputs=None, extra_args=None),block_ids=([1299, 1300, 1301, 1302, 1303, 1304, 1305, 1306, 1307, 1308, 1309, 1310, 1311, 1312, 1313, 1314, 1315, 1316, 1317, 1318, 1319, 1320, 1321, 1322, 1323, 1324, 1325, 1326, 1327, 1328, 1329, 1330, 1331, 1332, 1333, 1334, 1335, 1336, 1337, 1338, 1339, 1340, 1341],),num_computed_tokens=0,lora_request=None,prompt_embeds_shape=None)], scheduled_cached_reqs=CachedRequestData(req_ids=['chatcmpl-0721c0b185fd419c9a4594955b125766', 'chatcmpl-c123b1c2a6124ba4b6751f9295fedb60', 'chatcmpl-aa2319efc98c4342838ea42e267ab5d2', 'chatcmpl-029d33be6527448bb50c1417a1d02b0f'], resumed_from_preemption=[false, false, false, false], new_token_ids=[], new_block_ids=[[[1297]], null, null, [[1298]]], num_computed_tokens=[3880, 899, 679, 633]), num_scheduled_tokens={chatcmpl-c0150128df3f4637864541d650a02fb9: 687, chatcmpl-0721c0b185fd419c9a4594955b125766: 5, chatcmpl-c123b1c2a6124ba4b6751f9295fedb60: 5, chatcmpl-aa2319efc98c4342838ea42e267ab5d2: 5, chatcmpl-029d33be6527448bb50c1417a1d02b0f: 5}, total_num_scheduled_tokens=707, scheduled_spec_decode_tokens={chatcmpl-029d33be6527448bb50c1417a1d02b0f: [112531, 99194, 38035, 17447], chatcmpl-0721c0b185fd419c9a4594955b125766: [100544, 3837, 104013, 109612], chatcmpl-c123b1c2a6124ba4b6751f9295fedb60: [17323, 71, 3022, 43], chatcmpl-aa2319efc98c4342838ea42e267ab5d2: [112531, 99194, 38035, 17447]}, scheduled_encoder_inputs={chatcmpl-c0150128df3f4637864541d650a02fb9: [0]}, num_common_prefix_blocks=[0], finished_req_ids=['chatcmpl-e4e8edd6366042429bfaebe5479951bb'], free_encoder_mm_hashes=['dd8c6601b7dec1227a8dbc366c398edb0c05d6a568ab4153729d1df02fe14e88'], structured_output_request_ids={}, grammar_bitmask=null, kv_connector_metadata=null)
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m ERROR 09-29 03:07:32 [dump_input.py:79] Dumping scheduler stats: SchedulerStats(num_running_reqs=5, num_waiting_reqs=0, step_counter=0, current_wave=0, kv_cache_usage=0.26257668711656446, prefix_cache_stats=PrefixCacheStats(reset=False, requests=1, queries=687, hits=0), spec_decoding_stats=None, kv_connector_stats=None, num_corrupted_reqs=0)
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m ERROR 09-29 03:07:32 [core.py:710] EngineCore encountered a fatal error.
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m ERROR 09-29 03:07:32 [core.py:710] Traceback (most recent call last):
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m ERROR 09-29 03:07:32 [core.py:710] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/utils.py", line 409, in merge_multimodal_embeddings
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m ERROR 09-29 03:07:32 [core.py:710] inputs_embeds.masked_scatter(is_multimodal.unsqueeze(-1),
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m ERROR 09-29 03:07:32 [core.py:710] torch.AcceleratorError: CUDA error: device-side assert triggered
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m ERROR 09-29 03:07:32 [core.py:710] Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m ERROR 09-29 03:07:32 [core.py:710]

@ggg-s
Copy link

ggg-s commented Sep 29, 2025 •
edited
Loading

mm embds shape: [torch.Size([546, 3584])]
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m before propose input ids: target token ids: tensor([151644, 8948, 198, 2610, 525, 264, 10950, 17847, 13,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151645, 198, 151644, 872, 198, 151652, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151653, 53481, 41683, 17254, 9370, 45930,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 102073, 43815, 8997, 101140, 37945, 104317, 102650, 15946, 64471,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 101114, 1773, 62244, 102650, 15946, 101475, 37945, 53481, 102650,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 101047, 102122, 24968, 62244, 102650, 15946, 101114, 37945, 53481,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 102650, 15946, 104013, 109318, 101070, 8997, 14880, 23031, 2236,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 68805, 99553, 7, 104346, 3837, 100714, 53481, 3837, 99226,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 98237, 53481, 8, 101124, 66017, 3837, 90919, 104346, 14880,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 11622, 2073, 106468, 5373, 101977, 5373, 42140, 99605, 854,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 104787, 1773, 101475, 102122, 15946, 99226, 98237, 53481, 91680,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 102298, 104766, 5373, 44091, 3837, 100714, 53481, 46448, 108593,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 102298, 102122, 106336, 49567, 104449, 1773, 101114, 102122, 15946,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 99226, 98237, 53481, 91680, 102298, 104013, 5373, 104766, 5373,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 102196, 3837, 100714, 53481, 46448, 108593, 102298, 104013, 104363,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 5373, 104766, 99719, 49567, 100405, 104449, 1773, 151645, 198,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151644, 77091, 198, 57218, 112531, 99194, 38035, 17447, 24562,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 100544, 3837, 104013, 109612, 18830, 17323, 71, 3022, 43,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 46944, 112531, 99194, 38035, 17447], device='cuda:0',
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m dtype=torch.int32), next token ids: tensor([ 4913, 111413, 110607, 100010, 105611], device='cuda:0',
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m dtype=torch.int32) ,mm_embeds shape: [torch.Size([546, 3584])]
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m propose input ids: tensor([ 8948, 198, 2610, 525, 264, 10950, 17847, 13, 151645,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 198, 151644, 872, 198, 151652, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655, 151655,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 151655, 151655, 151653, 53481, 41683, 17254, 9370, 45930, 102073,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 43815, 8997, 101140, 37945, 104317, 102650, 15946, 64471, 101114,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 1773, 62244, 102650, 15946, 101475, 37945, 53481, 102650, 101047,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 102122, 24968, 62244, 102650, 15946, 101114, 37945, 53481, 102650,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 15946, 104013, 109318, 101070, 8997, 14880, 23031, 2236, 68805,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 99553, 7, 104346, 3837, 100714, 53481, 3837, 99226, 98237,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 53481, 8, 101124, 66017, 3837, 90919, 104346, 14880, 11622,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 2073, 106468, 5373, 101977, 5373, 42140, 99605, 854, 104787,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 1773, 101475, 102122, 15946, 99226, 98237, 53481, 91680, 102298,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 104766, 5373, 44091, 3837, 100714, 53481, 46448, 108593, 102298,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 102122, 106336, 49567, 104449, 1773, 101114, 102122, 15946, 99226,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 98237, 53481, 91680, 102298, 104013, 5373, 104766, 5373, 102196,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 3837, 100714, 53481, 46448, 108593, 102298, 104013, 104363, 5373,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 104766, 99719, 49567, 100405, 104449, 1773, 151645, 198, 151644,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 77091, 198, 4913, 112531, 111413, 38035, 17447, 24562, 100544,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 3837, 110607, 109612, 18830, 100010, 71, 3022, 43, 46944,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m 105611, 99194, 38035, 17447, 151655], device='cuda:0',
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m dtype=torch.int32), mm embeds shape :[torch.Size([546, 3584])]
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m utils.py: is_multimodal sum: 547, emb shape 0:546
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:402: masked_scatter_size_check: block: [0,0,0], thread: [0,0,0] Assertion totalElements <= srcSize failed.
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m ERROR 09-29 03:07:32 [dump_input.py:69] Dumping input data for V1 LLM engine (v0.11.0rc2.dev0+g0d9fe260d.d20250926) with config: model='/lhf-model/Ipc/Qwen2.5-VL-7B-Instruct-20250829', speculative_config=SpeculativeConfig(method='eagle3', model='/mnt/sharedLLMs/qwen2.5-vl-7b-eagle3-specforge-v5-epoch1', num_spec_tokens=4), tokenizer='/lhf-model/Ipc/Qwen2.5-VL-7B-Instruct-20250829', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=8192, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=base_model, enable_prefix_caching=True, chunked_prefill_enabled=True, pooler_config=None, compilation_config={"level":3,"debug_dump_path":"","cache_dir":"/root/.cache/vllm/torch_compile_cache/6cd3220e9b","backend":"","custom_ops":[],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output","vllm.mamba_mixer2","vllm.mamba_mixer","vllm.short_conv","vllm.linear_attention","vllm.plamo2_mamba_mixer","vllm.gdn_attention"],"use_inductor":true,"compile_sizes":[],"inductor_compile_config":{"enable_auto_functionalized_v2":false},"inductor_passes":{},"cudagraph_mode":1,"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[512,504,496,488,480,472,464,456,448,440,432,424,416,408,400,392,384,376,368,360,352,344,336,328,320,312,304,296,288,280,272,264,256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"use_inductor_graph_partition":false,"pass_config":{},"max_capture_size":512,"local_cache_dir":"/root/.cache/vllm/torch_compile_cache/6cd3220e9b/rank_0_0/backbone"},
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m ERROR 09-29 03:07:32 [dump_input.py:76] Dumping scheduler output for model execution: SchedulerOutput(scheduled_new_reqs=[NewRequestData(req_id=chatcmpl-c0150128df3f4637864541d650a02fb9,prompt_token_ids_len=687,mm_features=[MultiModalFeatureSpec(data={'image_grid_thw': MultiModalFieldElem(modality='image', key='image_grid_thw', data=tensor([ 1, 42, 52]), field=MultiModalBatchedField()), 'pixel_values': MultiModalFieldElem(modality='image', key='pixel_values', data=tensor([[1.9297, 1.9297, 1.9297, ..., 2.1406, 2.1406, 2.1406],
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m ERROR 09-29 03:07:32 [dump_input.py:76] [1.9297, 1.9297, 1.9297, ..., 2.1406, 2.1406, 2.1406],
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m ERROR 09-29 03:07:32 [dump_input.py:76] [1.9297, 1.9297, 1.9297, ..., 2.1406, 2.1406, 2.1406],
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m ERROR 09-29 03:07:32 [dump_input.py:76] ...,
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m ERROR 09-29 03:07:32 [dump_input.py:76] [1.9297, 1.9297, 1.9297, ..., 2.1406, 2.1406, 2.1406],
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m ERROR 09-29 03:07:32 [dump_input.py:76] [1.9297, 1.9297, 1.9297, ..., 2.1406, 2.1406, 2.1406],
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m ERROR 09-29 03:07:32 [dump_input.py:76] [1.9297, 1.9297, 1.9297, ..., 2.1406, 2.1406, 2.1406]],
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m ERROR 09-29 03:07:32 [dump_input.py:76] dtype=torch.bfloat16), field=MultiModalFlatField(slices=[[slice(0, 2184, None)]], dim=0))}, modality='image', identifier='089654d81821ee9d7bbe74606f03e9df82789e90e7bb59dbf5bf3b1df03c527f', mm_position=PlaceholderRange(offset=15, length=546, is_embed=None))],sampling_params=SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.05, temperature=0.0, top_p=1.0, top_k=0, min_p=0.0, seed=None, stop=[], stop_token_ids=[151643], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=128, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, structured_outputs=None, extra_args=None),block_ids=([1299, 1300, 1301, 1302, 1303, 1304, 1305, 1306, 1307, 1308, 1309, 1310, 1311, 1312, 1313, 1314, 1315, 1316, 1317, 1318, 1319, 1320, 1321, 1322, 1323, 1324, 1325, 1326, 1327, 1328, 1329, 1330, 1331, 1332, 1333, 1334, 1335, 1336, 1337, 1338, 1339, 1340, 1341],),num_computed_tokens=0,lora_request=None,prompt_embeds_shape=None)], scheduled_cached_reqs=CachedRequestData(req_ids=['chatcmpl-0721c0b185fd419c9a4594955b125766', 'chatcmpl-c123b1c2a6124ba4b6751f9295fedb60', 'chatcmpl-aa2319efc98c4342838ea42e267ab5d2', 'chatcmpl-029d33be6527448bb50c1417a1d02b0f'], resumed_from_preemption=[false, false, false, false], new_token_ids=[], new_block_ids=[[[1297]], null, null, [[1298]]], num_computed_tokens=[3880, 899, 679, 633]), num_scheduled_tokens={chatcmpl-c0150128df3f4637864541d650a02fb9: 687, chatcmpl-0721c0b185fd419c9a4594955b125766: 5, chatcmpl-c123b1c2a6124ba4b6751f9295fedb60: 5, chatcmpl-aa2319efc98c4342838ea42e267ab5d2: 5, chatcmpl-029d33be6527448bb50c1417a1d02b0f: 5}, total_num_scheduled_tokens=707, scheduled_spec_decode_tokens={chatcmpl-029d33be6527448bb50c1417a1d02b0f: [112531, 99194, 38035, 17447], chatcmpl-0721c0b185fd419c9a4594955b125766: [100544, 3837, 104013, 109612], chatcmpl-c123b1c2a6124ba4b6751f9295fedb60: [17323, 71, 3022, 43], chatcmpl-aa2319efc98c4342838ea42e267ab5d2: [112531, 99194, 38035, 17447]}, scheduled_encoder_inputs={chatcmpl-c0150128df3f4637864541d650a02fb9: [0]}, num_common_prefix_blocks=[0], finished_req_ids=['chatcmpl-e4e8edd6366042429bfaebe5479951bb'], free_encoder_mm_hashes=['dd8c6601b7dec1227a8dbc366c398edb0c05d6a568ab4153729d1df02fe14e88'], structured_output_request_ids={}, grammar_bitmask=null, kv_connector_metadata=null)
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m ERROR 09-29 03:07:32 [dump_input.py:79] Dumping scheduler stats: SchedulerStats(num_running_reqs=5, num_waiting_reqs=0, step_counter=0, current_wave=0, kv_cache_usage=0.26257668711656446, prefix_cache_stats=PrefixCacheStats(reset=False, requests=1, queries=687, hits=0), spec_decoding_stats=None, kv_connector_stats=None, num_corrupted_reqs=0)
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m ERROR 09-29 03:07:32 [core.py:710] EngineCore encountered a fatal error.
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m ERROR 09-29 03:07:32 [core.py:710] Traceback (most recent call last):
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m ERROR 09-29 03:07:32 [core.py:710] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/utils.py", line 409, in merge_multimodal_embeddings
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m ERROR 09-29 03:07:32 [core.py:710] inputs_embeds.masked_scatter(is_multimodal.unsqueeze(-1),
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m ERROR 09-29 03:07:32 [core.py:710] torch.AcceleratorError: CUDA error: device-side assert triggered
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m ERROR 09-29 03:07:32 [core.py:710] Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.
�[1;36m(EngineCore_DP0 pid=996225)�[0;0m ERROR 09-29 03:07:32 [core.py:710]

Please print the error logs from the very beginning.

make placeholders and vision patches the same length (trim to match) inside merge_multimodal_embeddings

@DarkLight1337
Copy link
Member

This corner case should be mitigated by #16229, which was merged recently

pdasigi pushed a commit to pdasigi/vllm that referenced this pull request Oct 2, 2025
@david6666666 @LJH-LBJ @pdasigi
[Multimodal][Speculative Decoding]Eagle Eagle3 mm support, enablement…
2d6c910 
… on qwen2.5vl (vllm-project#22872)

Signed-off-by: Junhong <liujunhong11@huawei.com>
Signed-off-by: Junhong Liu <98734602+LJH-LBJ@users.noreply.github.com>
Co-authored-by: Junhong <liujunhong11@huawei.com>
Co-authored-by: LJH-LBJ <98734602+LJH-LBJ@users.noreply.github.com>
yewentao256 pushed a commit that referenced this pull request Oct 3, 2025
@david6666666 @LJH-LBJ @yewentao256
[Multimodal][Speculative Decoding]Eagle Eagle3 mm support, enablement…
38c2df8 
… on qwen2.5vl (#22872)

Signed-off-by: Junhong <liujunhong11@huawei.com>
Signed-off-by: Junhong Liu <98734602+LJH-LBJ@users.noreply.github.com>
Co-authored-by: Junhong <liujunhong11@huawei.com>
Co-authored-by: LJH-LBJ <98734602+LJH-LBJ@users.noreply.github.com>
Signed-off-by: yewentao256 <zhyanwentao@126.com>
@benchislett
Copy link
Collaborator

Does anyone know why this PR removes torch compile support for llama_eagle3?

@benchislett benchislett mentioned this pull request Oct 8, 2025
Merged
5 tasks
@taoyun951753
Copy link

The qwen2.5-vl eagle3 model encounters an error when using @support_torch_compile

Traceback (most recent call last):
(EngineCore_DP0 pid=8139) File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore_DP0 pid=8139) self.run()
(EngineCore_DP0 pid=8139) File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
(EngineCore_DP0 pid=8139) self._target(*self._args, **self._kwargs)
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 716, in run_engine_core
(EngineCore_DP0 pid=8139) raise e
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 703, in run_engine_core
(EngineCore_DP0 pid=8139) engine_core = EngineCoreProc(*args, **kwargs)
(EngineCore_DP0 pid=8139) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 499, in init
(EngineCore_DP0 pid=8139) super().init(vllm_config, executor_class, log_stats,
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 93, in init
(EngineCore_DP0 pid=8139) self._initialize_kv_caches(vllm_config)
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 208, in _initialize_kv_caches
(EngineCore_DP0 pid=8139) self.model_executor.initialize_from_config(kv_cache_configs)
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 75, in initialize_from_config
(EngineCore_DP0 pid=8139) self.collective_rpc("compile_or_warm_up_model")
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc
(EngineCore_DP0 pid=8139) return [run_method(self.driver_worker, method, args, kwargs)]
(EngineCore_DP0 pid=8139) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/vllm/utils/init.py", line 3121, in run_method
(EngineCore_DP0 pid=8139) return func(*args, **kwargs)
(EngineCore_DP0 pid=8139) ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 344, in compile_or_warm_up_model
(EngineCore_DP0 pid=8139) cuda_graph_memory_bytes = self.model_runner.capture_model()
(EngineCore_DP0 pid=8139) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3520, in capture_model
(EngineCore_DP0 pid=8139) self._capture_cudagraphs(
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3597, in _capture_cudagraphs
(EngineCore_DP0 pid=8139) self._dummy_run(num_tokens,
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(EngineCore_DP0 pid=8139) return func(*args, **kwargs)
(EngineCore_DP0 pid=8139) ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3240, in dummy_run
(EngineCore_DP0 pid=8139) self.drafter.dummy_run(num_tokens,use_cudagraphs=cudagraph_runtime_mode!=CUDAGraphMode.NONE)
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/torch/utils/contextlib.py", line 120, in decorate_context
(EngineCore_DP0 pid=8139) return func(*args, **kwargs)
(EngineCore_DP0 pid=8139) ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/spec_decode/eagle.py", line 966, in dummy_run
(EngineCore_DP0 pid=8139) self.model(
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1773, in wrapped_call_impl
(EngineCore_DP0 pid=8139) return self.call_impl(*args, **kwargs)
(EngineCore_DP0 pid=8139) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1784, in call_impl
(EngineCore_DP0 pid=8139) return forward_call(*args, **kwargs)
(EngineCore_DP0 pid=8139) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen2_5_vl_eagle3.py", line 258, in forward
(EngineCore_DP0 pid=8139) return self.model(input_ids, positions, hidden_states, inputs_embeds)
(EngineCore_DP0 pid=8139) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/decorators.py", line 317, in call
(EngineCore_DP0 pid=8139) model_output = self.forward(*args, **kwargs)
(EngineCore_DP0 pid=8139) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen2_5_vl_eagle3.py", line 154, in forward
(EngineCore_DP0 pid=8139) def forward(
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/torch/dynamo/eval_frame.py", line 375, in call
(EngineCore_DP0 pid=8139) return super().call(*args, **kwargs)
(EngineCore_DP0 pid=8139) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1773, in wrapped_call_impl
(EngineCore_DP0 pid=8139) return self.call_impl(*args, **kwargs)
(EngineCore_DP0 pid=8139) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1784, in call_impl
(EngineCore_DP0 pid=8139) return forward_call(*args, **kwargs)
(EngineCore_DP0 pid=8139) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/torch/dynamo/eval_frame.py", line 929, in fn
(EngineCore_DP0 pid=8139) return fn(*args, **kwargs)
(EngineCore_DP0 pid=8139) ^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/torch/fx/graph_module.py", line 848, in call_wrapped
(EngineCore_DP0 pid=8139) return self.wrapped_call(self, *args, **kwargs)
(EngineCore_DP0 pid=8139) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/torch/fx/graph_module.py", line 424, in call
(EngineCore_DP0 pid=8139) raise e
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/torch/fx/graph_module.py", line 411, in call
(EngineCore_DP0 pid=8139) return super(self.cls, obj).call(*args, **kwargs) # type: ignore[misc]
(EngineCore_DP0 pid=8139) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1773, in wrapped_call_impl
(EngineCore_DP0 pid=8139) return self.call_impl(*args, **kwargs)
(EngineCore_DP0 pid=8139) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1784, in call_impl
(EngineCore_DP0 pid=8139) return forward_call(*args, **kwargs)
(EngineCore_DP0 pid=8139) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8139) File "<eval_with_key>.271", line 18, in forward
(EngineCore_DP0 pid=8139) submod_0 = self.submod_0(l_input_ids, s72, l_self_modules_embed_tokens_parameters_weight, l_self_modules_layers_modules_0_modules_input_layernorm_parameters_weight, l_hidden_states, l_self_modules_layers_modules_0_modules_hidden_norm_parameters_weight, l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_parameters_weight, l_self_modules_layers_modules_0_modules_self_attn_modules_rotary_emb_buffers_cos_sin_cache, l_positions, s18); l_input_ids = l_self_modules_embed_tokens_parameters_weight = l_self_modules_layers_modules_0_modules_input_layernorm_parameters_weight = l_self_modules_layers_modules_0_modules_hidden_norm_parameters_weight = l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_parameters_weight = l_self_modules_layers_modules_0_modules_self_attn_modules_rotary_emb_buffers_cos_sin_cache = l_positions = s18 = None
(EngineCore_DP0 pid=8139) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/cuda_graph.py", line 121, in call
(EngineCore_DP0 pid=8139) return self.runnable(*args, **kwargs)
(EngineCore_DP0 pid=8139) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/piecewise_backend.py", line 96, in call
(EngineCore_DP0 pid=8139) return self.compiled_graph_for_general_shape(*args)
(EngineCore_DP0 pid=8139) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/eval_frame.py", line 929, in _fn
(EngineCore_DP0 pid=8139) return fn(*args, **kwargs)
(EngineCore_DP0 pid=8139) ^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/torch/_functorch/aot_autograd.py", line 1241, in forward
(EngineCore_DP0 pid=8139) return compiled_fn(full_args)
(EngineCore_DP0 pid=8139) ^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 384, in runtime_wrapper
(EngineCore_DP0 pid=8139) all_outs = call_func_at_runtime_with_args(
(EngineCore_DP0 pid=8139) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/torch/_functorch/_aot_autograd/utils.py", line 126, in call_func_at_runtime_with_args
(EngineCore_DP0 pid=8139) out = normalize_as_list(f(args))
(EngineCore_DP0 pid=8139) ^^^^^^^
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 750, in inner_fn
(EngineCore_DP0 pid=8139) outs = compiled_fn(args)
(EngineCore_DP0 pid=8139) ^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 556, in wrapper
(EngineCore_DP0 pid=8139) return compiled_fn(runtime_args)
(EngineCore_DP0 pid=8139) ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/output_code.py", line 584, in call
(EngineCore_DP0 pid=8139) return self.current_callable(inputs)
(EngineCore_DP0 pid=8139) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8139) File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/utils.py", line 2716, in run
(EngineCore_DP0 pid=8139) out = model(new_inputs)
(EngineCore_DP0 pid=8139) ^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=8139) File "/root/.cache/vllm/torch_compile_cache/8304a885d2/rank_0_0/inductor_cache/2i/c2iggmzrq2sxtsm37f72hc22obqnk3swtexiwdxv5crnyca2zvan.py", line 622, in call
(EngineCore_DP0 pid=8139) assert_size_stride(arg8_1, (3, s18), (s18, 1))
(EngineCore_DP0 pid=8139) AssertionError: expected size 3==3, stride 2048==1536 at dim=0
(EngineCore_DP0 pid=8139) This error most often comes from a incorrect fake (aka meta) kernel for a custom op.

xuebwang-amd pushed a commit to xuebwang-amd/vllm that referenced this pull request Oct 10, 2025
@david6666666 @LJH-LBJ @xuebwang-amd
[Multimodal][Speculative Decoding]Eagle Eagle3 mm support, enablement…
6fb605a 
… on qwen2.5vl (vllm-project#22872)

Signed-off-by: Junhong <liujunhong11@huawei.com>
Signed-off-by: Junhong Liu <98734602+LJH-LBJ@users.noreply.github.com>
Co-authored-by: Junhong <liujunhong11@huawei.com>
Co-authored-by: LJH-LBJ <98734602+LJH-LBJ@users.noreply.github.com>
Signed-off-by: xuebwang-amd <xuebwang@amd.com>
sducouedic pushed a commit to sducouedic/vllm that referenced this pull request Oct 16, 2025
@david6666666 @LJH-LBJ
[Multimodal][Speculative Decoding]Eagle Eagle3 mm support, enablement…
cca0023 
… on qwen2.5vl (vllm-project#22872)

Signed-off-by: Junhong <liujunhong11@huawei.com>
Signed-off-by: Junhong Liu <98734602+LJH-LBJ@users.noreply.github.com>
Co-authored-by: Junhong <liujunhong11@huawei.com>
Co-authored-by: LJH-LBJ <98734602+LJH-LBJ@users.noreply.github.com>
lywa1998 pushed a commit to lywa1998/vllm that referenced this pull request Oct 20, 2025
@david6666666 @LJH-LBJ @lywa1998
[Multimodal][Speculative Decoding]Eagle Eagle3 mm support, enablement…
ed78704 
… on qwen2.5vl (vllm-project#22872)

Signed-off-by: Junhong <liujunhong11@huawei.com>
Signed-off-by: Junhong Liu <98734602+LJH-LBJ@users.noreply.github.com>
Co-authored-by: Junhong <liujunhong11@huawei.com>
Co-authored-by: LJH-LBJ <98734602+LJH-LBJ@users.noreply.github.com>
alhridoy pushed a commit to alhridoy/vllm that referenced this pull request Oct 24, 2025
@david6666666 @LJH-LBJ @alhridoy
[Multimodal][Speculative Decoding]Eagle Eagle3 mm support, enablement…
f55f28b 
… on qwen2.5vl (vllm-project#22872)

Signed-off-by: Junhong <liujunhong11@huawei.com>
Signed-off-by: Junhong Liu <98734602+LJH-LBJ@users.noreply.github.com>
Co-authored-by: Junhong <liujunhong11@huawei.com>
Co-authored-by: LJH-LBJ <98734602+LJH-LBJ@users.noreply.github.com>
xuebwang-amd pushed a commit to xuebwang-amd/vllm that referenced this pull request Oct 24, 2025
@david6666666 @LJH-LBJ @xuebwang-amd
[Multimodal][Speculative Decoding]Eagle Eagle3 mm support, enablement…
c1bab0f 
… on qwen2.5vl (vllm-project#22872)

Signed-off-by: Junhong <liujunhong11@huawei.com>
Signed-off-by: Junhong Liu <98734602+LJH-LBJ@users.noreply.github.com>
Co-authored-by: Junhong <liujunhong11@huawei.com>
Co-authored-by: LJH-LBJ <98734602+LJH-LBJ@users.noreply.github.com>
Signed-off-by: xuebwang-amd <xuebwang@amd.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Isotr0py Isotr0py Isotr0py left review comments

@DarkLight1337 DarkLight1337 DarkLight1337 approved these changes

@sighingnow sighingnow Awaiting requested review from sighingnow sighingnow is a code owner

@ywang96 ywang96 Awaiting requested review from ywang96 ywang96 is a code owner

@WoosukKwon WoosukKwon Awaiting requested review from WoosukKwon

@robertgshaw2-redhat robertgshaw2-redhat Awaiting requested review from robertgshaw2-redhat

@njhill njhill Awaiting requested review from njhill

@comaniac comaniac Awaiting requested review from comaniac

@alexm-redhat alexm-redhat Awaiting requested review from alexm-redhat

@benchislett benchislett Awaiting requested review from benchislett benchislett is a code owner

@luccafong luccafong Awaiting requested review from luccafong luccafong is a code owner

+3 more reviewers

@ggg-s ggg-s ggg-s left review comments

@LJH-LBJ LJH-LBJ LJH-LBJ left review comments

@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

llama Related to Llama models new-model Requests to new models performance Performance-related issues qwen Related to Qwen models ready ONLY add when PR is ready to merge/full CI is needed speculative-decoding v1

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

8 participants

@david6666666 @22quinn @LJH-LBJ @ggg-s @DarkLight1337 @taoyun951753 @benchislett @Isotr0py