[Benchmark] Add plot utility for parameter sweep

[DarkLight1337]

Purpose

Follow-up to #27085

• Split up the old script into multiple files in a new directory vllm/benchmarks/sweep, abstracting away common code.
• Add a separate plotting CLI vllm/benchmarks/sweep/plot.py.

cc @lengrongfu

Test Plan

Test Result

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[mergify]

Documentation preview: https://vllm--27168.org.readthedocs.build/en/27168/

[ProExpertProg]

Better code organization than our mainline code! 😂

A couple of thoughts:

• Should we move SLA into a separate entrypoint serve_sla.py? To me it's different enough than the serve sweep, and we're already reusing utils. But up to you the current approach is fine
• Could you post an example plot command with its output plot? Hard to imagine what the plots look like just from the code
• I'll try the script when I can and let you know if I have more thoughts, we can merge before that though

[DarkLight1337]

Should we move SLA into a separate entrypoint serve_sla.py? To me it's different enough than the serve sweep, and we're already reusing utils. But up to you the current approach is fine

Moved

Could you post an example plot command with its output plot? Hard to imagine what the plots look like just from the code

python -m vllm.benchmarks.sweep.plot benchmarks/results/20251019_101029 \
    --fig-dir throughput_vs_concurrency \
    --var-x max_concurrency \
    --var-y request_throughput \
    --col-by api_server_count \
    --curve-by max_num_batched_tokens \
    --filter-by 'max_concurrency<=1024'

[lengrongfu]

If we use a vllm serve api address, we should how to config?

Maybe we should add a --serve-host param, user can set a vllm online server, then this --serve-params param can be invalid.

[DarkLight1337]

You can set the server's host via --serve-cmd. And for resetting the server cache after each benchmark run, you can use --after-bench-cmd.

[DarkLight1337]

If you mean that the benchmark should not be responsible for launching the server, you can just use a dummy command that sleeps infinitely and adjust --bench-cmd to access the real server. Of course, you should also set --after-bench-cmd in this case.

[lengrongfu]

I see, maybe i not need set --serve-cmd param, use --bench-cmd param to set vllm bench serve --model meta-llama/Llama-2-7b-chat-hf --backend openai is enough.

[lengrongfu]

The generated file name has single quotes above it, which is weird, not sure if it's just my environment.

[DarkLight1337]

The generated file name has single quotes above it, which is weird, not sure if it's just my environment.

Fixed now