Enable --profile in 'vllm bench throughput'

[tomasruizt]

Purpose

The goal is easily profile a throughput workload. Concretely the command vllm bench throughput --profile ... will generate a profiling file, which is useful to debug performance gaps using the https://ui.perfetto.dev/ UI.

Test Plan

The vllm bench command can take 4 different paths. Here is how I tested them.
First I set env vars to enable profiling and also enable CUDA blocking to make long GPU ops show long runtime (optional).

export VLLM_USE_V1=1
export VLLM_TORCH_PROFILER_DIR=./profiles/
export CUDA_LAUNCH_BLOCKING=1

1. The run_vllm() path

vllm bench throughput     --model=Qwen/Qwen3-1.7B     --dataset-name=hf     --dataset-path=likaixin/InstructCoder     --max-num-seqs=100     --num-prompts=10     --input-len=1000     --output-len=10     --max-model-len=2048     --gpu-memory-utilization=0.6     --profile --enforce-eager

2. The run_vllm_async() path

vllm bench throughput     --model=Qwen/Qwen3-1.7B     --dataset-name=hf     --dataset-path=likaixin/InstructCoder     --max-num-seqs=100     --num-prompts=10     --input-len=1000     --output-len=10     --max-model-len=2048     --gpu-memory-utilization=0.6     --profile --enforce-eager --async-engine

3. the backend=vllm-chat path for multimodal models

vllm bench throughput     --model=Qwen/Qwen2.5-VL-3B-Instruct     --dataset-name=hf --dataset-path=lmarena-ai/VisionArena-Chat     --max-num-seqs=100     --num-prompts=10     --input-len=1000     --output-len=10     --max-model-len=2048     --gpu-memory-utilization=0.6     --profile --enforce-eager  --backend=vllm-chat

4. The backend=hf path

vllm bench throughput     --model=Qwen/Qwen3-1.7B     --dataset-name=sharegpt --max-num-seqs=100     --num-prompts=10 --input-len=1000 --output-len=10 --profile --enforce-eager --backend=hf  --hf-max-batch-size=10

Test Result

All paths except backend=hf generate a profile file, like this one: file.pt.trace.json.gz. As mentioned, they can be opened with https://ui.perfetto.dev/.

The reason is that the class AutoModelForCausalLM used in backend=hf does not implement .start_profiler() and stop_profiler(). Therefore, I raise a NotImplementedError, so the user knows to remove the --profile flag.

[gemini-code-assist]

Code Review

This pull request adds support for profiling throughput benchmarks using the --profile flag. The implementation correctly wraps the model execution calls with start_profile() and stop_profile(). My main feedback is to ensure the profiler is always stopped, even if an error occurs during model execution. This can be achieved by using try...finally blocks, which will make the profiling logic more robust. I've added specific suggestions for this improvement.

[tomasruizt]

I don't agree with Geminis review. One problem with its suggestion that I've seen in real life is that the generation fails, the profiler is graciously stopped, and it creates a profile result (which is very short obviously). Therefore, it appears as if the generation had been successful and super fast based on this short profile.

That is why, if the generation fails, the profiler should not dump a profiling result IMO. It should not be graciously stopped.