[Hardware] add platform-specific request validation api

[joerunde]

This PR adds a generic validate_request api to the platform interface. This allows platforms to implement any runtime checks on each request to ensure that all the requested features are supported before scheduling it. There is already one existing check in this category, supports_structured_output, and I'd like to avoid a proliferation of more platform apis for individual features like this.

Currently, the spyre plugin needs to implement some extra validation around the shape of inputs, as we have more constraints on valid prompt lengths and max token requests. This new api would let us do that without needing to hack around rejecting requests from the scheduler.

FIX vllm-project/vllm-spyre#77

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[njhill]

cc @NickLucche this is was we were discussing a couple of days ago...

[njhill]

Thanks @joerunde!

[njhill]

Wondering whether we should remove the call to supports_structured_output and have the default impl of validate_request call that instead. Actually maybe we could remove the supports_structured_output interface method and have validate_request only call it if it exists in the same class?

[joerunde]

Sure. Maybe the simplest thing to do is to just add an impl for the TPU backend and have it reject structured output requests?

[joerunde]

@njhill I went with the 🔥🔥🔥 option- WDYT?
The only difference in behavior now should be that all out-of-tree platforms will need to explicitly reject structured output in validate_request instead of inheriting the default impl of supports_structured_output

[njhill]

Do we need to include lora_request here? Wouldn't a platform either support lora or not, and if not isn't this something that could be checked at startup time?

[joerunde]

Ah, yeah that's true. I was thinking there might be a case where something about the adapter needs validation but I think you're right that anything about supporting lora would be checked either at boot time, or at adapter load time

[njhill]

Thanks @joerunde looks great ... would be good for @NickLucche to take a look too!

[NickLucche]

fine work @joerunde thanks!
I already have another check for TPU here #16172 so the structured output exception will feel less lonely.

[joerunde]

the structured output exception will feel less lonely

Ah nice, everybody needs friends!

[yarongmu-google]

This PR has broken the benchmark_serving.py command; can we please rollback, or fix?

Traceback (most recent call last):
File "/workspace/vllm/benchmarks/benchmark_serving.py", line 1083, in
main(args)
File "/workspace/vllm/benchmarks/benchmark_serving.py", line 684, in main
benchmark_result = asyncio.run(
File "/usr/local/lib/python3.10/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/usr/local/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
return future.result()
File "/workspace/vllm/benchmarks/benchmark_serving.py", line 297, in benchmark
raise ValueError(
ValueError: Initial test run failed - Please make sure benchmark arguments are correctly specified. Error: Never received a valid chunk to calculate TTFT.This response will be marked as failed!

Repro:
(this) cb391d8-> failed
(one before) fee5b8d-> good

[mgoin]

@yarongmu-google could you share the benchmark_serving.py command that failed? I tried a simple command and it worked

vllm serve meta-llama/Llama-3.1-8B-Instruct --port 9000

python benchmarks/benchmark_serving.py --backend vllm --model meta-llama/Llama-3.1-8B-Instruct --num-prompts 1 --dataset-name random --random-input 1024 --random-output 512 --port 9000
============ Serving Benchmark Result ============
Successful requests:                     1         
Benchmark duration (s):                  6.83      
Total input tokens:                      1024      
Total generated tokens:                  512       
Request throughput (req/s):              0.15      
Output token throughput (tok/s):         74.92     
Total Token throughput (tok/s):          224.77    
---------------Time to First Token----------------
Mean TTFT (ms):                          21.53     
Median TTFT (ms):                        21.53     
P99 TTFT (ms):                           21.53     
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          13.33     
Median TPOT (ms):                        13.33     
P99 TPOT (ms):                           13.33     
---------------Inter-token Latency----------------
Mean ITL (ms):                           13.33     
Median ITL (ms):                         10.36     
P99 ITL (ms):                            29.87     
==================================================

[yarongmu-google]

@mgoin

python benchmarks/benchmark_serving.py
--backend vllm
--model $MODEL
--dataset-name sonnet
--dataset-path benchmarks/sonnet_4x.txt
--sonnet-input-len 1800
--sonnet-output-len 128
--ignore-eos

where MODEL is llama3 70B. Note that this is run on a clean machine created only for perf benchamrks.

@yaochengji also saw the breakage. Chengji what's your command?

[yarongmu-google]

Note that the breakage is on TPU

[yaochengji]

@yaochengji also saw the breakage. Chengji what's your command?

I only saw this breakage in CI test, my benchmarking command on llama-8B model is good.

[yarongmu-google]

Hmmm .. maybe it's fixed somehow later?? Let's give it a bit more time. Sorry for flooding this PR :)

[yaochengji]

maybe it's fixed somehow later

I don't think so. It's the latest commit at the moment.

[mgoin]

I have posted a fix here, it is specific to TPU V1 #16369

[joerunde]

Shoot, sorry for the breakage!

I had wrongly assumed that TPU tests would catch that case during the CI for this PR, before merging to main :(