[Core/DBO][2/N] Dual-Batch Overlap add DeepEP High Throughput support and Prefill support

[LucasWilkinson]

Purpose

• Add support for having prefill requests inside ubatches; this is done by allowing a request to be split across ubatches with the second batch effectively chunked-prefill continuing on from the the first (since it always runs after)
• Add support for the DeepEP high-throughput kernels with SM control for DeepGEMM and DeepEP (shout-out to @yewentao256 for the HT support)
• General cleanups

Test Plan

lm_eval

Test Result

export VLLM_ALL2ALL_BACKEND=deepep_high_throughput

lm_eval --model local-completions --model_args "base_url=http://0.0.0.0:8000/v1/completions,model=deepseek-ai/DeepSeek-R1,num_concurrent=256" --tasks gsm8k
....
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.9522|±  |0.0059|
|     |       |strict-match    |     5|exact_match|↑  |0.9500|±  |0.0060|

export VLLM_ALL2ALL_BACKEND=deepep_low_latency

lm_eval --model local-completions --model_args "base_url=http://0.0.0.0:8000/v1/completions,model=deepseek-ai/DeepSeek-R1,num_concurrent=256" --tasks gsm8k
....
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.9522|±  |0.0059|
|     |       |strict-match    |     5|exact_match|↑  |0.9507|±  |0.0060|

HT Overlap Trace (2x8xH100)

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @LucasWilkinson.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @LucasWilkinson.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[voipmonitor]

@LucasWilkinson this commit introduces weired behaviour - the first http request with larger context is working normally but the subsequent requests are signifficantly slower. I have verified that it is this commit: cc1dc7e which should be this PR. a903669 is working normally

CUDA_VISIBLE_DEVICES=4,5 NCCL_DEBUG=INFO vllm serve /mnt/gpt-oss-120b --async-scheduling --tensor-parallel-size 2 --port 4999 --host 0.0.0.0 --config  GPT-OSS_Blackwell.yaml --served-model-name default --gpu-memory-utilization 0.8 --max-num-seqs 1024 --max-num-batched-tokens 32000

cat GPT-OSS_Blackwell.yaml
compilation-config: '{"pass_config":{"enable_fi_allreduce_fusion":false,"enable_noop":true},"custom_ops":["+rms_norm"],"cudagraph_mode":"FULL_AND_PIECEWISE"}'
async-scheduling: true
cuda-graph-sizes: 2048
max-num-batched-tokens: 32000

[varun-sundar-rabindranath]

removing this doesn't make the padding happen.