[Model] EVS support for nano_nemotron_vl

[tomeras91]

Purpose

Add support for EVS (Efficient Video Sampling, introduced in #22980) for Nano Nemotron VL model.

Contrary to other multimodal models (for example, Qwen2.5-VL) Nano Nemotron VL uses text frame separators. These are not placeholders, and should not be replaced by video embeddings. For example, if a video has 2 frames, each with 5 tokens, the video text replacement will be:

Frame0: <img><image><image><image><image><image></img>Frame2: <img><image><image><image><image><image></img>

This poses a challenge to EVS, since it means we need to know the number of tokens per frame when creating the text replacement. In the standard vLLM flow, this is impossible since the replacement is created before the forward pass through the vision encoder, and EVS selects video tokens to prune based on the vision encoder outputs (which obviously are known only after the forward pass through the vision encoder).

We worked around this issue by signaling get_input_embeddings of the SupportsMultiModal protocol to not replace any of the video text replacement (done by setting embed_text=seq in get_video_repl when creating PromptUpdateDetails), and handle embedding merging ourselves in NemotronH_Nano_VL_V2.get_multimodal_embeddings, where we already know how many tokens are retained per frame. This also means the number of tokens per frame set in _get_prompt_updates's call to get_video_repl doesn't matter, as that text replacement is created only to set the correct number of tokens. Hence, all that matters in that call is the total number of retained video tokens, which is known at that stage based on video_pruning_rate.

Joint work with @BloodAxe

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[gemini-code-assist]

Code Review

This pull request adds support for Efficient Video Sampling (EVS) to the Nano Nemotron VL model. The implementation introduces a clever workaround to handle this model's specific use of text separators between video frames, which poses a challenge for the standard EVS workflow. The core of the solution involves manually merging video and text embeddings within the model's get_multimodal_embeddings method. Additionally, the EVS utility functions in vllm/multimodal/evs.py have been refactored for better generality, which is a positive change. The overall approach is sound, but I've identified a performance issue in the new code that could lead to unnecessary CPU-GPU synchronization.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

[DarkLight1337]

LGTM, thanks