[Bug]: EVS for Qwen 2_5 Errors out when using with video + image in this order

[sthakur2-sc]

🐛 Describe the bug

The new EVS pruner updates prompt place holders to pruned tokens. This is used to erroneously calculate the intial mrope positions. Example if request has 1 video followed by and image.

Prompt gets rewritten with pruned video token count (say 4 instead of 8)
_vl_get_input_positions_tensor is called
First iteration: finds video, uses video_grid_thw[0] which says 8 cells
Line 1015 advances: st = ed + 8 (full grid)
But the prompt only has 4 video tokens, so st overshoots
Second iteration: tries to find image token starting from overshot position
Search fails → both ed_image and ed_video become len(input_tokens) + 1
Condition ed_image < ed_video is FALSE
Goes to video branch → tries image_grid_thw[image_index] but should be looking at images!

While we recompute the mrope the code fails at _vl_get_input_positions_tensor never reaching the recompute_mrope

Your current environment

The output of python collect_env.py

```text ============================== System Info ============================== OS : Debian GNU/Linux 12 (bookworm) (x86_64) GCC version : (Debian 12.2.0-14+deb12u1) 12.2.0 Clang version : Could not collect CMake version : version 3.25.1 Libc version : glibc-2.36

==============================
PyTorch Info

PyTorch version : 2.8.0+cu128
Is debug build : False
CUDA used to build PyTorch : 12.8
ROCM used to build PyTorch : N/A

==============================
Python Environment

Python version : 3.11.2 (main, Apr 28 2025, 14:11:48) [GCC 12.2.0] (64-bit runtime)
Python platform : Linux-6.1.0-40-cloud-amd64-x86_64-with-glibc2.36

==============================
CUDA / GPU Info

Is CUDA available : True
CUDA runtime version : 13.0.48
CUDA_MODULE_LOADING set to : LAZY
GPU models and configuration : GPU 0: NVIDIA L4
Nvidia driver version : 580.65.06
cuDNN version : Could not collect
HIP runtime version : N/A
MIOpen runtime version : N/A
Is XNNPACK available : True

==============================
CPU Info

Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 46 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 32
On-line CPU(s) list: 0-31
Vendor ID: GenuineIntel
Model name: Intel(R) Xeon(R) CPU @ 2.20GHz
CPU family: 6
Model: 85
Thread(s) per core: 2
Core(s) per socket: 16
Socket(s): 1
Stepping: 7
BogoMIPS: 4400.38
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves arat avx512_vnni md_clear arch_capabilities
Hypervisor vendor: KVM
Virtualization type: full
L1d cache: 512 KiB (16 instances)
L1i cache: 512 KiB (16 instances)
L2 cache: 16 MiB (16 instances)
L3 cache: 38.5 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-31
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Mitigation; Aligned branch/return thunks
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Mitigation; Enhanced IBRS
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditional; PBRSB-eIBRS SW sequence; BHI SW loop, KVM SW loop
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
Vulnerability Vmscape: Not affected

==============================
Versions of relevant libraries

[pip3] numpy==2.2.6
[pip3] nvidia-cublas-cu12==12.8.4.1
[pip3] nvidia-cuda-cupti-cu12==12.8.90
[pip3] nvidia-cuda-nvrtc-cu12==12.8.93
[pip3] nvidia-cuda-runtime-cu12==12.8.90
[pip3] nvidia-cudnn-cu12==9.10.2.21
[pip3] nvidia-cufft-cu12==11.3.3.83
[pip3] nvidia-cufile-cu12==1.13.1.3
[pip3] nvidia-curand-cu12==10.3.9.90
[pip3] nvidia-cusolver-cu12==11.7.3.90
[pip3] nvidia-cusparse-cu12==12.5.8.93
[pip3] nvidia-cusparselt-cu12==0.7.1
[pip3] nvidia-nccl-cu12==2.27.3
[pip3] nvidia-nvjitlink-cu12==12.8.93
[pip3] nvidia-nvtx-cu12==12.8.90
[pip3] pyzmq==27.1.0
[pip3] torch==2.8.0
[pip3] torchaudio==2.8.0
[pip3] torchvision==0.23.0
[pip3] transformers==4.57.1
[pip3] triton==3.4.0
[conda] Could not collect

==============================
vLLM Info

ROCM Version : Could not collect
vLLM Version : 0.11.0
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled
GPU Topology:
GPU0 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X 0-31 0 N/A

Legend:

X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks

==============================
Environment Variables

LD_LIBRARY_PATH=/usr/local/cuda-13.0/lib64:/usr/local/cuda-13.0/lib64:
CUDA_HOME=/usr/local/cuda-13.0
CUDA_HOME=/usr/local/cuda-13.0
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
CUDA_MODULE_LOADING=LAZY

</details>



### Before submitting a new issue...

- [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.

[sthakur2-sc]

PR that added EVS : #22980
cc @BloodAxe please PTAL

[BloodAxe]

Would you please provide a code with call scenario which I can use to reproduce this?

[sthakur2-sc]

Just try any video followed by text and an image. The video should be long enough to consume num_tokens something like this

    messages = [
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Describe images in 10 words or less?"},
                {
                    "type": "video",
                    "video": ".data/synthetic_video/synth_01.mp4",
                },
                {"type": "text", "text": "middle tokens"},
                {
                    "type": "image",
                    "image": ".data/synthetic_images/synth_01.jpg",
                },
                {"type": "text", "text": "completed"},
            ],
        }
    ]

What happens is in vllm 0.11.0 MRotaryEmbedding._vl_get_input_positions_tensor which initially incorrectly calculates positions does not account for pruned tokens and when index has reached num_tokens it always sets ed_image and ed_video as num_tokens. But that's not a case covered in the function so it collides on if ed_image < ed_video and it always assumes video. but there is not more videos left so it errors out.

[sthakur2-sc]

@BloodAxe is the code under assumption that if there are images but server is deployed with video pruning it wont work ?

Reproduction code:

def prepare_inputs_for_vllm(messages, processor):
    text = processor.apply_chat_template(
        messages, tokenize=False, add_generation_prompt=True
    )
    # qwen_vl_utils 0.0.14+ reqired
    image_inputs, video_inputs, video_kwargs = process_vision_info(
        messages,
        image_patch_size=processor.image_processor.patch_size,
        return_video_kwargs=True,
        return_video_metadata=True,
    )
    print(f"video_kwargs: {video_kwargs}")

    mm_data = {}
    if image_inputs is not None:
        mm_data["image"] = image_inputs
    if video_inputs is not None:
        mm_data["video"] = video_inputs

    return {
        "prompt": text,
        "multi_modal_data": mm_data,
        "mm_processor_kwargs": video_kwargs,
    }


if __name__ == "__main__":

    processor = AutoProcessor.from_pretrained("Qwen/Qwen2.5-VL-7B-Instruct")
    messages = [
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Describe images in 10 words or less?"},
                {
                    "type": "image",
                    "image": ".data/test_video_1.jpg",
                },
                {
                    "type": "image",
                    "image": ".data/test_images_2.jpg",
                },
            ],
        }
    ]
    llm_engine = LLM(
        model="Qwen/Qwen2.5-VL-7B-Instruct",
        max_model_len=6000,
        gpu_memory_utilization=0.9,
        limit_mm_per_prompt={"image": 4, "video": 0, "audio": 0},
        mm_processor_kwargs={
            "max_pixels": 8 * 28 * 28,
            "min_pixels": 2 * 28 * 28,
            "pruner_strategy": "pooling",
        },
        max_num_batched_tokens=6000,
        max_num_seqs=128,
        trust_remote_code=True,
        disable_cascade_attn=True,
        disable_log_stats=False,
        skip_mm_profiling=True,
        enforce_eager=True,
        video_pruning_rate=0.5,
    )
    sampling_params = SamplingParams(
        max_tokens=32,
        logprobs=0,
        temperature=0.0,
    )
    inputs = [prepare_inputs_for_vllm(messages, processor)]
    outputs = llm_engine.generate(inputs, sampling_params=sampling_params)
    for output in outputs:
        print(output.outputs[0].text)`


Error : 
```(EngineCore_DP0 pid=1257599) Traceback (most recent call last):
(EngineCore_DP0 pid=1257599)   File "/usr/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore_DP0 pid=1257599)     self.run()
(EngineCore_DP0 pid=1257599)   File "/usr/lib/python3.11/multiprocessing/process.py", line 108, in run
(EngineCore_DP0 pid=1257599)     self._target(*self._args, **self._kwargs)
(EngineCore_DP0 pid=1257599)   File "/home/sthakur2/llm-inference/vllm_11/lib/python3.11/site-packages/vllm/v1/engine/core.py", line 712, in run_engine_core
(EngineCore_DP0 pid=1257599)     raise e
(EngineCore_DP0 pid=1257599)   File "/home/sthakur2/llm-inference/vllm_11/lib/python3.11/site-packages/vllm/v1/engine/core.py", line 701, in run_engine_core
(EngineCore_DP0 pid=1257599)     engine_core.run_busy_loop()
(EngineCore_DP0 pid=1257599)   File "/home/sthakur2/llm-inference/vllm_11/lib/python3.11/site-packages/vllm/v1/engine/core.py", line 728, in run_busy_loop
(EngineCore_DP0 pid=1257599)     self._process_engine_step()
(EngineCore_DP0 pid=1257599)   File "/home/sthakur2/llm-inference/vllm_11/lib/python3.11/site-packages/vllm/v1/engine/core.py", line 754, in _process_engine_step
(EngineCore_DP0 pid=1257599)     outputs, model_executed = self.step_fn()
(EngineCore_DP0 pid=1257599)                               ^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1257599)   File "/home/sthakur2/llm-inference/vllm_11/lib/python3.11/site-packages/vllm/v1/engine/core.py", line 284, in step
(EngineCore_DP0 pid=1257599)     model_output = self.execute_model_with_error_logging(
(EngineCore_DP0 pid=1257599)                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1257599)   File "/home/sthakur2/llm-inference/vllm_11/lib/python3.11/site-packages/vllm/v1/engine/core.py", line 270, in execute_model_with_error_logging
(EngineCore_DP0 pid=1257599)     raise err
(EngineCore_DP0 pid=1257599)   File "/home/sthakur2/llm-inference/vllm_11/lib/python3.11/site-packages/vllm/v1/engine/core.py", line 261, in execute_model_with_error_logging
(EngineCore_DP0 pid=1257599)     return model_fn(scheduler_output)
(EngineCore_DP0 pid=1257599)            ^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1257599)   File "/home/sthakur2/llm-inference/vllm_11/lib/python3.11/site-packages/vllm/v1/executor/abstract.py", line 103, in execute_model
(EngineCore_DP0 pid=1257599)     output = self.collective_rpc("execute_model",
(EngineCore_DP0 pid=1257599)              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1257599)   File "/home/sthakur2/llm-inference/vllm_11/lib/python3.11/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc
(EngineCore_DP0 pid=1257599)     return [run_method(self.driver_worker, method, args, kwargs)]
(EngineCore_DP0 pid=1257599)             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1257599)   File "/home/sthakur2/llm-inference/vllm_11/lib/python3.11/site-packages/vllm/utils/__init__.py", line 3122, in run_method
(EngineCore_DP0 pid=1257599)     return func(*args, **kwargs)
(EngineCore_DP0 pid=1257599)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1257599)   File "/home/sthakur2/llm-inference/vllm_11/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(EngineCore_DP0 pid=1257599)     return func(*args, **kwargs)
(EngineCore_DP0 pid=1257599)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1257599)   File "/home/sthakur2/llm-inference/vllm_11/lib/python3.11/site-packages/vllm/v1/worker/gpu_worker.py", line 447, in execute_model
(EngineCore_DP0 pid=1257599)     output = self.model_runner.execute_model(scheduler_output,
(EngineCore_DP0 pid=1257599)              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1257599)   File "/home/sthakur2/llm-inference/vllm_11/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(EngineCore_DP0 pid=1257599)     return func(*args, **kwargs)
(EngineCore_DP0 pid=1257599)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1257599)   File "/home/sthakur2/llm-inference/vllm_11/lib/python3.11/site-packages/vllm/v1/worker/gpu_model_runner.py", line 2239, in execute_model
(EngineCore_DP0 pid=1257599)     self._update_states(scheduler_output)
(EngineCore_DP0 pid=1257599)   File "/home/sthakur2/llm-inference/vllm_11/lib/python3.11/site-packages/vllm/v1/worker/gpu_model_runner.py", line 607, in _update_states
(EngineCore_DP0 pid=1257599)     self._init_mrope_positions(req_state)
(EngineCore_DP0 pid=1257599)   File "/home/sthakur2/llm-inference/vllm_11/lib/python3.11/site-packages/vllm/v1/worker/gpu_model_runner.py", line 764, in _init_mrope_positions
(EngineCore_DP0 pid=1257599)     MRotaryEmbedding.get_input_positions_tensor(
(EngineCore_DP0 pid=1257599)   File "/home/sthakur2/llm-inference/vllm_11/lib/python3.11/site-packages/vllm/model_executor/layers/rotary_embedding/mrope.py", line 481, in get_input_positions_tensor
(EngineCore_DP0 pid=1257599)     return cls._vl_get_input_positions_tensor(
(EngineCore_DP0 pid=1257599)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1257599)   File "/home/sthakur2/llm-inference/vllm_11/lib/python3.11/site-packages/vllm/model_executor/layers/rotary_embedding/mrope.py", line 985, in _vl_get_input_positions_tensor
(EngineCore_DP0 pid=1257599)     video_grid_thw[video_index][0],
(EngineCore_DP0 pid=1257599)     ~~~~~~~~~~~~~~^^^^^^^^^^^^^
(EngineCore_DP0 pid=1257599) IndexError: list index out of range

The error happens because the _vl_get_input_positions_tensor collides in the ed_image < ed_video comparison step as it doesnt account for both being the same due to pruning.

[BloodAxe]

I've tested EVS to work on image+video case and, if I recall correctly, on image only input..
IDK, maybe some logic had changed since then or I didn't test thoroughly.
In any case thanks for raising this issue. I don't have clear vision when I will be able to look into it but I hope it will happen somewhere this week.

[hebutliu2010]

The get_mrope_input_positions function has some issues in calculating st. Using vision_end_token_id for positioning is more accurate:
vision_end_token_id = hf_config.vision_end_token_id
...

st = ed + llm_grid_t * llm_grid_h * llm_grid_w

ed_end = input_tokens.index(vision_end_token_id, st + 1)
st = ed_end

[BloodAxe]

Looking at the current ToT code in main I do not observe _vl_get_input_positions_tensor existence at all.
I do see however that mrope positions computation logic has been moved to corresponding model classes. And code is now in qwen 2.5 vl class.
So I suggest you to try ToT VLLM just to see whether your issue still exists.

Secondly, a stacktrace you provided indicates you have a crash at the very start of the request (E.g even before any model's forward call) which cannot be attributed to EVS since no changes was done to initial mrope computation. EVS updates mrope positions inside forward(), but that's another story. Therefore I don't think you have EVS-caused issue in the first place. Have you tried setting video_pruning_rate to zero?

[sthakur2-sc]

@BloodAxe Actually even if you dint have direct changes in the function, your code modifies the placeholder offsets to pruned offtsets and _vl_get_input_positions_tensor function uses the same but old grid to calculate the intial positions. While it works well for videos only but when you have image plus videos it does not have the code to handle this edge case as it does not expect pruned offsets.

Looking at the current ToT code in main I do not observe _vl_get_input_positions_tensor existence at all.
Yes I am strictly using 0.11.0.

Btw,I wanna thank your implementation, I have used similar structure and implemented a generalized pruning for videos and images that allows any pruning class (not specifically evs to be init when vllm starts) and fixing these and some issue with evs specific recompute_mrope I get great results almost 2x gains.

[jojo23333]

Hi @sthakur2-sc , thank you for your issue, probably a bit unrelated to this particular issue, I'm recently looking into a generalized visual pruning module for VLLMs as well. I feel it's a nice feature to have. Right now, if I understand correctly, EVS enabled this by changing model_excutor directly, and they only support qwen_vl_2.5 and nemotron_vl. Is there a more elegant way to provide support for more models? Do you mind elaborating on your thoughts?

[sthakur2-sc]

Hi @jojo23333 I dont have a generic implementation for all models. My implementation is generic in the sense of pruner algorthim. It can use any pruning algortithm. I implement this via VLLM PLUGINS route. But if you need something generic. I would assumpe you would have to design a new pruner module that can be imported as layer past MultiModal encoder after the patch merger.

So the flow will be
Init deployment with pruner --> vllm starts --> Pruner layers are activated in the model

Changes will be required to : Scheduler, Mrope position calculations.
based on the pruner config , we also would need to update multimodal encoder compute budget calculations as it will now need to account for pruned tokens but also consider the memory required for vision encoder to go through the whole image (pre pruning). Same changes need to be applied to rope position calculations.

[jojo23333]

@sthakur2-sc Thank you for your valuable insight!