[Model] Enable DP for ViT in Qwen2-VL

[DarkLight1337]

Purpose

Part of #22743

Test Plan

Test Result

$ vllm bench serve \
    --backend openai-chat \
    --model Qwen/Qwen2-VL-7B-Instruct \
    --endpoint /v1/chat/completions \
    --dataset-name hf \
    --dataset-path lmarena-ai/VisionArena-Chat \
    --hf-split train \
    --num-prompts 500

$ vllm serve Qwen/Qwen2-VL-7B-Instruct -tp 2 --mm-encoder-tp-mode weights --limit_mm_per_prompt.image=1
============ Serving Benchmark Result ============
Successful requests:                     500       
Benchmark duration (s):                  159.99    
Total input tokens:                      34073     
Total generated tokens:                  49892     
Request throughput (req/s):              3.13      
Output token throughput (tok/s):         311.84    
Peak output token throughput (tok/s):    2314.00   
Peak concurrent requests:                500.00    
Total Token throughput (tok/s):          524.80    
---------------Time to First Token----------------
Mean TTFT (ms):                          62900.15  
Median TTFT (ms):                        52084.25  
P99 TTFT (ms):                           152029.84 
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          606.95    
Median TPOT (ms):                        711.70    
P99 TPOT (ms):                           791.46    
---------------Inter-token Latency----------------
Mean ITL (ms):                           584.34    
Median ITL (ms):                         716.03    
P99 ITL (ms):                            1033.28   
==================================================

$ vllm serve Qwen/Qwen2-VL-7B-Instruct -tp 2 --mm-encoder-tp-mode data --limit_mm_per_prompt.image=1
============ Serving Benchmark Result ============
Successful requests:                     500       
Benchmark duration (s):                  128.82    
Total input tokens:                      34073     
Total generated tokens:                  49656     
Request throughput (req/s):              3.88      
Output token throughput (tok/s):         385.46    
Peak output token throughput (tok/s):    2297.00   
Peak concurrent requests:                500.00    
Total Token throughput (tok/s):          649.95    
---------------Time to First Token----------------
Mean TTFT (ms):                          50187.68  
Median TTFT (ms):                        40427.38  
P99 TTFT (ms):                           120921.60 
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          488.14    
Median TPOT (ms):                        562.51    
P99 TPOT (ms):                           630.13    
---------------Inter-token Latency----------------
Mean ITL (ms):                           475.18    
Median ITL (ms):                         578.27    
P99 ITL (ms):                            728.98    
==================================================

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[gemini-code-assist]

Code Review

This pull request introduces data parallelism support for the Vision Transformer in Qwen2-VL. The changes are well-structured and primarily involve plumbing a use_data_parallel flag through the vision model components to conditionally disable tensor parallelism. The logic for handling data-parallel execution paths appears correct. Overall, the changes are sound and should enable the intended data parallelism functionality.

[Isotr0py]

LGTM

[gongshanchong]

I tested the DP strategy in two different hardware environments, using nccl-tests to test GPU communication in both environments. The results were (Avg bus bandwidth: 23.8703) for the L20 hardware environment and (Avg bus bandwidth: 282.133) for the H800 hardware environment. In the Qwen2.5-VL-7B-Instruct model I deployed, the TP strategy outperformed the DP strategy when the concurrency was below approximately 256. Why is this? Is it because communication is no longer a bottleneck at this point, and the computational benefits of TP outweigh the communication benefits of DP? When the concurrency reaches a certain level, communication in the Vit module becomes a bottleneck, and the communication benefits of DP outweigh the computational benefits of TP.