[Feature]: Generalized the DP feature for ViT and multimodal backbone for the benefit of all models

[tjtanaa]

🚀 The feature, motivation and pitch

The existing PRs

• [Model] enable data parallel for Llama4 vision encoder #18368
• [Model] Add option to run Step3VisionEncoder in DP #22697
• [FEAT] [Performance] Enable DP for ViT in Qwen2.5VL #22742

Has clearly shown that hybrid inferencing: (ViT is DP, and LLM is in TP), has greatly reduce the TTFT and improve the overall throughput significantly.

There are multiple reasons that we should have ViT implemented as a DP:

1. The ViT are small models, the TP all reduce incurred a larger overhead than the gain from accelerating through TP.
2. ViT are not captured in cuda graphs or torch compile graph, thus the kernel overhead and all reduce overhead will be higher.

Extending the support to more models:

• [Model] Support dp on ViT on GLM-4.5V #23168
• [Model] Support DP for ViT on MiniCPM-V-4 #23327
• [MM Encoder] Add Encoder DP to InternVL #23876
• [MM Encoder] Add Encoder DP to Kimi-VL #23878
• [Model] Enable encoder DP for MiniCPM-V #23948
• [MM Encoder] Apply DP ViT for Qwen3-VL model series #24955
• [Model] Enable DP for ViT in Qwen2-VL #25445

Alternatives

No response

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[noooop]

I also found that vit is too small for tp.

maybe we should have an async processor, further increase throughput

[david6666666]

I will contribute to GLM-4.5V DP for ViT

[666even666]

I am happy to contribute to enabling DP for InternVL ViT :)

[DarkLight1337]

Re-opening to track the implementation for more models

[ZhangCW-QwQ]

I will contribute to Intern-S1 DP for ViT

[LJH-LBJ]

I will contribute to Kimi-VL-A3B-Thinking-2506