[Usage]: How to use the image datasets sharegpt4v provided in benchmark_serving?

[DK-DARKmatter]

Your current environment

The output of `python collect_env.py`

How would you like to use vllm

I want to run multi modal benchmark with vllm, first with image inputs. I download the dataset sharegpt4v as provide in README, but I found if I run benchmark_serving with:

dataset-name sharegpt

the program actually read the text prompt but no image input. How can we use benchmark_serving with the given dataset, to test the basic performance of dealing with images and prompt input?

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[DK-DARKmatter]

OK，so the sharegpt4v only support for bench_throughput
Do we have any suggestions for multi-modal test with bench_serving?

[ywang96]

@DK-DARKmatter Hey - feel free to take a look at the example in the description of #12389.

We're also trying to consolidate these datasets for different benchmarks in #14036

[DK-DARKmatter]

@ywang96 Hi, thanks for the quick reply! So use the vision-arena-bench dataset could solve this. The README here is a little bit misleading. I wonder if we could use some other datasets which has the same format as vision-arena-bench? Because vision-arena-bench-v0.1 now only has 500 images and some times we may want to have more test cases.

[Potabk]

I wonder if we are now have sufficient accuracy using multimodal input for the serving benchmark, and is there any plan to show on dashboard with multi modal benchmark

[github-actions]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!

[github-actions]

This issue has been automatically closed due to inactivity. Please feel free to reopen if you feel it is still relevant. Thank you!