Add support for the /rerank endpoint in vllm bench serve

[maxdebayser]

The /rerank API can be support both by embedding models and native reranker models. However, with reranker models the query is concatenated with each document with a separator token in between. Therefore the amount of tokens that passes through the model has to be accounted for differently in each case.

Because of these details, this PR, adds a specialized random dataset to generates requests which send the expected amount of tokens. So when the use sets random-input-len, num-prompts and random-batch-size, in both cases we will generate requests such that the total amount of tokens is promptsinput-len in batches of size batch-sizeinput-len.

Here is an example of how this works. With the server running an reranker or embedding model

vllm serve BAAI/bge-reranker-v2-m3

run a benchmark using the vllm-rerank backend and the random-rerank dataset:

vllm bench serve --model BAAI/bge-reranker-v2-m3 --tokenizer BAAI/bge-reranker-v2-m3 \
  --backend vllm-rerank --endpoint /v1/rerank --dataset-name random-rerank \
  --random-input-len 512 --num-prompts 10 --random-batch-size 5

[mergify]

Documentation preview: https://vllm--26602.org.readthedocs.build/en/26602/

[maxdebayser]

cc: @noooop , @DarkLight1337 , @ZJY0516

[gemini-code-assist]

Code Review

This pull request adds valuable support for benchmarking the /rerank endpoint, including a new specialized random dataset and documentation. The implementation is well-structured, refactoring existing embedding benchmark logic into a more general _run_pooling_request function to accommodate both embeddings and reranking. However, I've identified a critical issue that can cause the benchmark to crash under specific default conditions. Please see the detailed comment for the fix.

[maxdebayser]

Related to #21796

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @maxdebayser.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork