[Model] Define merge_by_field_config MM interface (U-Z)

[ayushsatyam146]

Part of #26149

[gemini-code-assist]

Code Review

This pull request refactors the Ultravox, Voxtral, and Whisper models to use the new merge_by_field_config multimodal interface. The changes involve setting this flag to True in the model classes and removing the flatten_bn utility, as its functionality is now handled by the new data processing pipeline. The code modifications are consistent and appear correct for this refactoring. I did not identify any issues of high or critical severity.

[chatgpt-codex-connector]

💡 Codex Review

vllm/vllm/model_executor/models/ultravox.py

Lines 551 to 574 in a7615f9

	# Audio lens and token_len are already in the correct shape
	audio_lens = audio_input["lens"]
	audio_token_len = audio_input["token_len"]
	embeddings = self._audio_features_to_embeddings(audio_features, audio_lens)
	# We should flatten and concatenate embeddings based on token lengths
	# For example, with token_len = [4, 2, 3], flattened_embeddings will be
	# concat(embeddings[0][:4], embeddings[1][:2], embeddings[2][:3])
	# Create a mask of valid indices based on token lengths
	max_len = embeddings.shape[1]
	indices = torch.arange(max_len, device=embeddings.device).expand(
	embeddings.shape[0], -1
	)
	mask = indices < audio_token_len[:, None]
	# Apply mask and flatten
	flattened_embeddings = embeddings[mask]
	# Return one tensor per input audio
	embed_lens = [
	token_len_item.sum().item() for token_len_item in audio_input["token_len"]
	]
	return flattened_embeddings.split(embed_lens)

Splitting Ultravox embeddings per chunk instead of per audio

In _process_audio_input the lens and token lengths are now taken as-is (audio_token_len = audio_input["token_len"]) because the new merge_by_field_config path already flattens the B×N dimensions. However, embed_lens is still built by iterating over audio_input["token_len"] and summing each element. With the flattened inputs this loop now produces one entry per chunk rather than one entry per audio item, so flattened_embeddings.split(embed_lens) returns a list of tensors per chunk while _get_prompt_updates still emits only one PromptReplacement per audio based on audio_num_chunks. The number and ordering of multimodal embeddings therefore no longer matches the placeholders inserted into the prompt, which will misalign audio embeddings with their token replacements (and can raise when counts differ). Consider rebuilding embed_lens using the original per-audio grouping (e.g. via audio_num_chunks) so the function continues to return one tensor per audio item.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

[DarkLight1337]

Have you tested each model using the example script?

[ayushsatyam146]

@DarkLight1337 I have done the required changes. But, I am sorry since I couldn't run the example scripts due to GPU constraints

[DarkLight1337]

Voxtral and Whisper are both failing locally, I have fixed Voxtral and will fix Whisper later

[DarkLight1337]

Fixed Whisper as well, let's merge this