[Feature] Add visionarena offline support for benchmark_throughput

[JenZhao]

This is a follow-up PR for #14036 to support vision arena in the benchmark_throughput script.

With the vllm-chat backend, the prompt token count will now include image token counts, as we will be using the request output to perform the calculation.

CLOSES #14600

---

hf: lmarena-ai/vision-arena-bench-v0.1

MODEL_NAME="Qwen/Qwen2-VL-7B-Instruct"
NUM_PROMPTS=10
DATASET_NAME="hf"
DATASET_PATH="lmarena-ai/vision-arena-bench-v0.1"
DATASET_SPLIT="train"

python3 benchmarks/benchmark_throughput.py \
  --model "${MODEL_NAME}" \
  --backend "vllm-chat" \
  --dataset-name "${DATASET_NAME}" \
  --dataset-path "${DATASET_PATH}" \
  --num-prompts "${NUM_PROMPTS}" \
  --hf-split "${DATASET_SPLIT}"

Throughput: 1.96 requests/s, 3099.89 total tokens/s, 251.02 output tokens/s
Total num prompt tokens:  14527
Total num output tokens:  1280

Try another model

MODEL_NAME="llava-hf/llava-v1.6-mistral-7b-hf"
# rest is the same as above

Throughput: 2.27 requests/s, 7409.71 total tokens/s, 290.59 output tokens/s
Total num prompt tokens:  31359
Total num output tokens:  1280

---

hf: lmms-lab/LLaVA-OneVision-Data

MODEL_NAME="llava-hf/llava-v1.6-mistral-7b-hf"
NUM_PROMPTS=10
DATASET_NAME="hf"
DATASET_PATH="lmms-lab/LLaVA-OneVision-Data"
DATASET_SPLIT="train"

python3 vllm/benchmarks/benchmark_throughput.py \
  --model "${MODEL_NAME}" \
  --backend "vllm-chat" \
  --dataset-name "${DATASET_NAME}" \
  --dataset-path "${DATASET_PATH}" \
  --num-prompts "${NUM_PROMPTS}" \
  --hf-split "${DATASET_SPLIT}" \
  --hf-subset "chart2text(cauldron)"

Throughput: 4.25 requests/s, 9978.81 total tokens/s, 307.99 output tokens/s
Total num prompt tokens:  22765
Total num output tokens:  725

---

since we changed the token calculation for vllm and vllm-chat, here we are using sharegpt dataset to verify the change

This Branch

Dataset	Backend	Model	Processed Prompts	Total Prompt Tokens	Total Output Tokens	Requests/s	Total Tokens/s	Output Tokens/s
sharegpt	vllm-chat	NousResearch/Hermes-3-Llama-3.1-8B	1000	235196	198343	27.14	11764.43	5382.20
sharegpt	vllm	NousResearch/Hermes-3-Llama-3.1-8B	1000	215196	198343	27.58	11404.09	5469.67

python3 vllm/benchmarks/benchmark_throughput.py --model NousResearch/Hermes-3-Llama-3.1-8B --backend vllm-chat  --dataset-name sharegpt --dataset-path ~/data/vllm_benchmark_datasets/ShareGPT_V3_unfiltered_cleaned_split.json --num-prompts 1000

python3 vllm/benchmarks/benchmark_throughput.py --model NousResearch/Hermes-3-Llama-3.1-8B --backend vllm --dataset-name sharegpt --dataset-path ~/data/vllm_benchmark_datasets/ShareGPT_V3_unfiltered_cleaned_split.json --num-prompts 1000

Main Branch
vllm-chat is not support on the main branch

Dataset	Backend	Model	Processed Prompts	Total Prompt Tokens	Total Output Tokens	Requests/s	Total Tokens/s	Output Tokens/s
sharegpt	vllm	NousResearch/Hermes-3-Llama-3.1-8B	1000	215196	198343	27.51	11375.09	5455.76

python3 vllm/benchmarks/benchmark_throughput.py --model NousResearch/Hermes-3-Llama-3.1-8B --backend vllm --dataset-name sharegpt --dataset-path ~/data/vllm_benchmark_datasets/ShareGPT_V3_unfiltered_cleaned_split.json --num-prompts 1000

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[ywang96]

I left a few comments - thank you for your work!

Could you please also do a comparison run between main and this PR when running vLLM with ShareGPT? I'd like to see the differences between the two if there's any.

[JenZhao]

I left a few comments - thank you for your work!

Could you please also do a comparison run between main and this PR when running vLLM with ShareGPT? I'd like to see the differences between the two if there's any.

test results here

https://github.com/JenZhao/vllm/pull/5#issue-2918317590

[ywang96]

LGTM! I left some final comments.