[Model] EVS support for nano_nemotron_vl

[tomeras91]

Purpose

Add support for EVS (Efficient Video Sampling, introduced in #22980) for Nano Nemotron VL model.

Contrary to other multimodal models (for example, Qwen2.5-VL) Nano Nemotron VL uses text frame separators. These are not placeholders, and should not be replaced by video embeddings. For example, if a video has 2 frames, each with 5 tokens, the video text replacement will be:

Frame0: <img><image><image><image><image><image></img>Frame2: <img><image><image><image><image><image></img>

This poses a challenge to EVS, since it means we need to know the number of tokens per frame when creating the text replacement. In the standard vLLM flow, this is impossible since the replacement is created before the forward pass through the vision encoder, and EVS selects video tokens to prune based on the vision encoder outputs (which obviously are known only after the forward pass through the vision encoder).

We worked around this issue by signaling get_input_embeddings of the SupportsMultiModal protocol to not replace any of the video text replacement (done by setting embed_text=seq in get_video_repl when creating PromptUpdateDetails), and handle embedding merging ourselves in NemotronH_Nano_VL_V2.get_multimodal_embeddings, where we already know how many tokens are retained per frame. This also means the number of tokens per frame set in _get_prompt_updates's call to get_video_repl doesn't matter, as that text replacement is created only to set the correct number of tokens. Hence, all that matters in that call is the total number of retained video tokens, which is known at that stage based on video_pruning_rate.

Joint work with @BloodAxe

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[gemini-code-assist]

Code Review

This pull request adds support for Efficient Video Sampling (EVS) to the nano_nemotron_vl model. The implementation introduces a clever workaround to accommodate this model's unique video frame processing, which involves text separators, by handling embedding merging within get_multimodal_embeddings. The changes also include a beneficial refactoring of the EVS utility functions, improving their reusability and fixing a potential bug. The overall approach is sound and the implementation appears correct. I have one suggestion to improve robustness by replacing an assert statement with a more explicit ValueError for input validation in a critical path.

[gemini-code-assist]

Using assert for input validation in production code can be risky. Assertions can be disabled with Python's -O flag, and they raise a generic AssertionError. It's better to use an explicit if check and raise a ValueError with a descriptive message. This makes the code more robust against unexpected or malformed inputs and prevents potential server crashes.

            if single_video_embeddings.shape[0] % num_frames != 0:
                raise ValueError(
                    f"The number of video embeddings ({single_video_embeddings.shape[0]}) "
                    f"is not divisible by the number of frames ({num_frames})."
                )

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @tomeras91.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[tomeras91]

closed due to rebase havoc. Replaced by #26269