SYNC RHOAI 2.24 to midstream v0.10.0.1 release #175

heyselbi · 2025-08-08T06:27:06Z

Includes:

Upstream's v0.10.0 release
Cherry picks:
- Fix CUDA permute/unpermute for use with DeepGemm Moe vllm-project/vllm#17934 --> resolves MOE models on ROCM
- [Bugfix][ROCm] Fix for warp_size uses on host vllm-project/vllm#21205 --> resolves MOE models on ROCM
- heyselbi@297ea4e --> resolves XET issue not present anymore
Remove VLLM_ALLOW_LONG_MAX_MODEL_LEN=1
Remove hf[xet] requirement
Updates to Dockerfile.ubi and Dockerfile.rocm.ubi
- upgrade tgis_adapter to 0.8.0
- nccl version update for [Bug]: CUDA Illegal Access Error when running 2x Blackwell for tensor or pipeline parallel. vllm-project/vllm#19166
- from @dtrifiro 👍
  - Dockerfile.ubi: we're still using flashinfer 0.2.5 (https://storage.googleapis.com/neuralmagic-public-pypi/dist/flashinfer_python-0.2.5-cp38-abi3-linux_x86_64.whl)
  - Dockerfile.rocm.ubi: ARG FA_BRANCH="3cea2fb" (v2.6.3-21-g3cea2fb6), we've been using the same revision that upstream has, which is 1a7f4dfa (v2.7.4.post1-26-g1a7f4dfa)
  - Dockerfile.rocm still has libsodium
  - We could probably remove the VLLM_USE_V1 env vars since V0 has been removed in v0.10.0
  - removed libsodium from rocm contianer
  - bump flashinfer to 0.2.8 (https://storage.googleapis.com/nm-public-pypi/dist/flashinfer_python-0.2.8-cp39-abi3-linux_x86_64.whl)
  - bump ROCm flash-attention to v2.7.4.post1-26-g1a7f4dfa
  - remove ROCm VLLM_USE_V1 flag

Signed-off-by: Yong Hoon Shin <yhshin@meta.com>

…Seek TP (vllm-project#20769) Signed-off-by: Alexander Matveev <amatveev@redhat.com>

…ites from the cache (vllm-project#20942) Signed-off-by: Richard Zou <zou3519@gmail.com>

… DeepGEMM (vllm-project#20933) Signed-off-by: mgoin <mgoin64@gmail.com>

…ct#20945) Signed-off-by: mgoin <mgoin64@gmail.com>

Signed-off-by: Xiongfei Wei <isaacwxf23@gmail.com>

Signed-off-by: Ricardo Decal <rdecal@anyscale.com>

…t#20763) Signed-off-by: reidliu41 <reid201711@gmail.com>

Signed-off-by: NickLucche <nlucches@redhat.com>

Signed-off-by: Ruheena Suhani Shaik <rsshaik@habana.ai>

vllm-project#20934) Signed-off-by: Pavani Majety <pmajety@nvidia.com>

…oject#20926) Signed-off-by: Isotr0py <2037008807@qq.com>

…ject#20940) Signed-off-by: Boyuan Feng <boyuan@meta.com>

…#20534) Signed-off-by: Jennifer He <islandhe@gmail.com> Signed-off-by: <islandhe@gmail.com> Signed-off-by: Jen H <islandhe@gmail.com>

Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>

…0918) Signed-off-by: ilmarkov <imarkov@redhat.com> Co-authored-by: ilmarkov <imarkov@redhat.com>

…ject#20961) Signed-off-by: reidliu41 <reid201711@gmail.com>

Signed-off-by: Ricardo Decal <rdecal@anyscale.com>

…roject#20206) Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>

…llm-project#20602) Signed-off-by: Ricardo Decal <rdecal@anyscale.com>

Signed-off-by: Yifei Teng <tengyifei88@gmail.com>

…li (vllm-project#20840) Signed-off-by: Thomas Parnell <tpa@zurich.ibm.com>

Signed-off-by: wangli <wangli858794774@gmail.com>

Signed-off-by: Rui Qiao <ruisearch42@gmail.com>

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Signed-off-by: Christian Pinto <christian.pinto@ibm.com>

Signed-off-by: Patrick von Platen <patrick.v.platen@gmail.com> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>

…roject#20994) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

…m-project#20979) Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

heyselbi · 2025-08-08T06:32:09Z

/ok-to-test

heyselbi · 2025-08-08T06:33:19Z

/build-rocm

heyselbi · 2025-08-08T06:33:24Z

/build-cuda

ckhordiasma · 2025-08-08T13:01:47Z

/build-cuda

ckhordiasma · 2025-08-08T13:14:24Z

/build-cuda

heyselbi · 2025-08-08T16:42:23Z

/build-cuda

heyselbi · 2025-08-09T10:59:05Z

/build-cuda

ckhordiasma · 2025-08-09T13:44:22Z

/build-cuda

dtrifiro · 2025-08-09T16:00:06Z

@ckhordiasma I pushed v0.10.0.0 tags to this repo and to vllm-rocm. Can you make sure that versioning (tags) is (are) correct when the branches are ready? Thanks!

heyselbi · 2025-08-10T05:49:03Z

/build-cuda

heyselbi · 2025-08-11T09:15:56Z

/build-cuda

heyselbi · 2025-08-11T10:01:05Z

/build-cuda

heyselbi · 2025-08-11T10:17:11Z

/build-cuda

heyselbi · 2025-08-11T15:24:34Z

/build-cuda

heyselbi · 2025-08-11T15:33:53Z

/build-cuda

heyselbi · 2025-08-11T19:32:51Z

/build-cuda

heyselbi · 2025-08-15T19:14:32Z

/build-cuda

heyselbi · 2025-08-18T17:51:57Z

Validation is passed on Midstream side. PSAP team is currently running some final benchmarks. As no apparent blockers are expected, merging it for now to make it on time to release tomorrow.

sarckk and others added 30 commits July 14, 2025 23:11

Fall back if flashinfer comm module not found (vllm-project#20936)

61e2082

Signed-off-by: Yong Hoon Shin <yhshin@meta.com>

SM100 Cutlass MLA decode with unrestricted num_heads (< 128) for Deep…

8cdc371

…Seek TP (vllm-project#20769) Signed-off-by: Alexander Matveev <amatveev@redhat.com>

[BugFix] VLLM_DISABLE_COMPILE_CACHE=1 should disable all reads and wr…

ba8c300

…ites from the cache (vllm-project#20942) Signed-off-by: Richard Zou <zou3519@gmail.com>

[Bugfix] Fix incorrect dispatch for CutlassBlockScaledGroupedGemm and…

bcdfb2a

… DeepGEMM (vllm-project#20933) Signed-off-by: mgoin <mgoin64@gmail.com>

[CI/Build] Split Entrypoints Test into LLM and API Server (vllm-proje…

946aadb

…ct#20945) Signed-off-by: mgoin <mgoin64@gmail.com>

Use w8a8 quantized matmul Pallas kernel (vllm-project#19170)

d4170fa

Signed-off-by: Xiongfei Wei <isaacwxf23@gmail.com>

[Docs] Add Kuberay to deployment integrations (vllm-project#20592)

054c865

Signed-off-by: Ricardo Decal <rdecal@anyscale.com>

feat: add image zoom to improve image viewing experience (vllm-projec…

37e2eca

…t#20763) Signed-off-by: reidliu41 <reid201711@gmail.com>

[CI] Fix flaky test_streaming_response test (vllm-project#20913)

80305c1

Signed-off-by: NickLucche <nlucches@redhat.com>

Enabled BnB NF4 inference on Gaudi (vllm-project#20172)

016b8d1

Signed-off-by: Ruheena Suhani Shaik <rsshaik@habana.ai>

[Bugfix] Switch bailout logic for kv-cache-dtype with SM100 Flashinfer (

9ad0a45

vllm-project#20934) Signed-off-by: Pavani Majety <pmajety@nvidia.com>

[Doc] Clearer mistral3 and pixtral model support description (vllm-pr…

fc01791

…oject#20926) Signed-off-by: Isotr0py <2037008807@qq.com>

[cold start] replace VLLM_COMPILE_DEPYF with debug_dump_dir (vllm-pro…

91b3d19

…ject#20940) Signed-off-by: Boyuan Feng <boyuan@meta.com>

[Model] Add AutoWeightsLoader support for BERT, RoBERTa (vllm-project…

85bd659

…#20534) Signed-off-by: Jennifer He <islandhe@gmail.com> Signed-off-by: <islandhe@gmail.com> Signed-off-by: Jen H <islandhe@gmail.com>

Implement Async Scheduling (vllm-project#19970)

d4d3094

Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>

[Misc] Refactor AllReduceFusionPass. Remove parameter (vllm-project#2…

37a7d5d

…0918) Signed-off-by: ilmarkov <imarkov@redhat.com> Co-authored-by: ilmarkov <imarkov@redhat.com>

[frontend] Add --help=page option for paginated help output (vllm-pro…

68d28e3

…ject#20961) Signed-off-by: reidliu41 <reid201711@gmail.com>

[Docs] Improve documentation for RLHF example (vllm-project#20598)

235bfd5

Signed-off-by: Ricardo Decal <rdecal@anyscale.com>

[frontend] Refactor CLI Args for a better modular integration (vllm-p…

f148c44

…roject#20206) Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>

[Docs] Improve documentation for ray cluster launcher helper script (v…

33d5600

…llm-project#20602) Signed-off-by: Ricardo Decal <rdecal@anyscale.com>

[TPU] Optimize kv cache update kernel (vllm-project#20415)

c586b55

Signed-off-by: Yifei Teng <tengyifei88@gmail.com>

[V1] [Hybrid] Refactor mamba state shape calculation; enable V1 via c…

3534c39

…li (vllm-project#20840) Signed-off-by: Thomas Parnell <tpa@zurich.ibm.com>

[MISC] Add init files for python package (vllm-project#20908)

20149d8

Signed-off-by: wangli <wangli858794774@gmail.com>

[doc] Add more details for Ray-based DP (vllm-project#20948)

d912781

Signed-off-by: Rui Qiao <ruisearch42@gmail.com>

[Deprecation] Remove TokenizerPoolConfig (vllm-project#20968)

56fe4be

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

[v1][core] Support for attention free models (vllm-project#20811)

4ffd963

Signed-off-by: Christian Pinto <christian.pinto@ibm.com>

Voxtral (vllm-project#20970)

e7e3e6d

Signed-off-by: Patrick von Platen <patrick.v.platen@gmail.com> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>

[CI/Build] Fix wrong path in Transformers Nightly Models Test (vllm-p…

c847e34

…roject#20994) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

[Deprecation] Remove everything scheduled for removal in v0.10.0 (vll…

313ae8c

…m-project#20979) Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Configure Gemini (vllm-project#20971)

5bac613

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

heyselbi added 2 commits August 8, 2025 11:04

Update tgis adapter to 0.8.0

01e9d83

remove VLLM_ALLOW_LONG_MAX_MODEL_LEN=1

d3548b7

heyselbi mentioned this pull request Aug 8, 2025

Sync v0.10.0 rhoai rocm red-hat-data-services/vllm-rocm#52

Merged

update vllm version

7053645

dtrifiro approved these changes Aug 11, 2025

View reviewed changes

[TEMP] Make openai entrypoint default and add template files

9d8b4d6

Correct chown

f777d8d

update template to examples directory

73b8071

install blobfile for Kimi model support

efe58e0

fix the typo

32bb10d

add nccl upgrade to tgis adapter

6acccf0

heyselbi merged commit 9f294e4 into red-hat-data-services:main Aug 18, 2025
4 checks passed

SYNC RHOAI 2.24 to midstream v0.10.0.1 release #175

SYNC RHOAI 2.24 to midstream v0.10.0.1 release #175

Uh oh!

Conversation

heyselbi commented Aug 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

heyselbi commented Aug 8, 2025

Uh oh!

heyselbi commented Aug 8, 2025

Uh oh!

heyselbi commented Aug 8, 2025

Uh oh!

ckhordiasma commented Aug 8, 2025

Uh oh!

ckhordiasma commented Aug 8, 2025

Uh oh!

heyselbi commented Aug 8, 2025

Uh oh!

heyselbi commented Aug 9, 2025

Uh oh!

ckhordiasma commented Aug 9, 2025

Uh oh!

dtrifiro commented Aug 9, 2025

Uh oh!

heyselbi commented Aug 10, 2025

Uh oh!

heyselbi commented Aug 11, 2025

Uh oh!

heyselbi commented Aug 11, 2025

Uh oh!

heyselbi commented Aug 11, 2025

Uh oh!

heyselbi commented Aug 11, 2025

Uh oh!

heyselbi commented Aug 11, 2025

Uh oh!

heyselbi commented Aug 11, 2025

Uh oh!

heyselbi commented Aug 15, 2025

Uh oh!

heyselbi commented Aug 18, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

121 participants

heyselbi commented Aug 8, 2025 •

edited

Loading