[Core][Bugfix] Fix Offline MM Beam Search

[alex-jw-brooks]

Fixes passing the multimodal data for offline beam search.

FIX #16240
FIX #15694

Snippet to verify:

import vllm
from vllm import LLM
from vllm.sampling_params import BeamSearchParams, SamplingParams
from vllm.assets.audio import AudioAsset

model_name = "Qwen/Qwen2-Audio-7B-Instruct"

wav = AudioAsset("mary_had_lamb").audio_and_sample_rate

question = "What is recited in the audio?"
audio_in_prompt = "Audio 1: <|audio_bos|><|AUDIO|><|audio_eos|>\n"
prompt = ("<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n"
            "<|im_start|>user\n"
            f"{audio_in_prompt}{question}<|im_end|>\n"
            "<|im_start|>assistant\n")

inputs = [
    {
        "prompt": prompt,
        "multi_modal_data": {"audio": wav},
    },
]


llm = LLM(
    model=model_name,
    max_model_len=4096,
    max_num_seqs=5,
    limit_mm_per_prompt={"audio": 1}, 
)

outputs = llm.beam_search(inputs, BeamSearchParams(beam_width=1, max_tokens=50, temperature=0))
for output in outputs:
    generated_text = output.sequences[0].text
    print(f"Generated text from beam search (single beam 0 temp): {generated_text!r}")

outputs = llm.generate(inputs, SamplingParams(max_tokens=50, temperature=0))
for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"From greedy decoding (generate): {generated_text!r}")

Now considers the mm data in the beam search result.

Generated text from beam search (single beam 0 temp): "<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\nAudio 1: <|audio_bos|><|AUDIO|><|audio_eos|>\nWhat is recited in the audio?<|im_end|>\n<|im_start|>assistant\nThe recited content in the audio is: 'First words I spoke in the original cornograph A little piece of practical poetry Mary had a little lamb Its fleece was white as snow And everywhere that Mary went The lamb was sure to go.'<|im_end|>"
Processed prompts: 100%|██████████████████████| 1/1 [00:01<00:00,  1.54s/it, est. speed input: 283.70 toks/s, output: 31.88 toks/s]
From greedy decoding (generate): "The recited content in the audio is: 'First words I spoke in the original cornograph A little piece of practical poetry Mary had a little lamb Its fleece was white as snow And everywhere that Mary went The lamb was sure to go.'"

CC @DarkLight1337 🙂

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[DarkLight1337]

Can we add a simple test similar to tests/samplers/test_beam_search.py?

[alex-jw-brooks]

Seems like we currently don't use this, so removed it for now to keep things nice looking and consistent the other helpers / get_inputs. Happy to put it back later on if it ends up being needed though

[DarkLight1337]

Thanks, this should be good assuming that the tests pass

[alex-jw-brooks]

Awesome, thanks a lot for the fast reviews Cyrus! 😄

[alex-jw-brooks]

Looks like it failed in the CI, I'll try to fix it later today 🤞

[alex-jw-brooks]

Hey @DarkLight1337, things are passing now (reduced beams to n=2 to avoid the example that diverges + added some output sanitization for the eos). Can you please take a look?