AArch64 CPU Docker pipeline #26931

ghost · 2025-10-15T18:36:19Z

Purpose

Adding workflow for building CPU AArch64 wheels.

Aims to fix #26017.

Test Plan

The same workflow was gracefully ran locally, on an AArch64 machine.

Test Result

Essential Elements of an Effective PR Description Checklist

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan, such as providing test command.
The test results, such as pasting the results comparison before and after, or e2e results
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
(Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

gemini-code-assist

Code Review

This pull request adds a new Buildkite pipeline step to build and publish an AArch64 CPU Docker image. The change correctly uses a multi-platform Dockerfile and a dedicated arm64 agent. However, there is a critical issue with the Docker image tagging strategy. The new arm64 build uses the same tags as the existing x86 build, which will cause a race condition and result in one image overwriting the other. My review includes a suggestion to use architecture-specific tags to resolve this, which is a standard practice for multi-architecture image releases.

.buildkite/release-pipeline.yaml

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

.buildkite/release-pipeline.yaml

mgoin · 2025-10-16T13:30:50Z

.buildkite/release-pipeline.yaml

+      - "DOCKER_BUILDKIT=1 docker build --build-arg max_jobs=16 --build-arg USE_SCCACHE=1 --build-arg GIT_REPO_CHECK=1 --tag vllm-ci:build-image --target vllm-openai --progress plain -f docker/Dockerfile.cpu ."
+      - "mkdir artifacts"
+      - "docker run --rm -v $(pwd)/artifacts:/artifacts_host vllm-ci:build-image bash -c 'cp -r dist /artifacts_host && chmod -R a+rw /artifacts_host'"
+      - "bash .buildkite/scripts/upload-wheels.sh"


Do you know what the final wheel name will be and if it might clobber the existing arm CUDA wheel? I'm not sure what we do for x86 cpu

hi! the CUDA wheel should mention "+cu..." in its name, while the CPU only builds leave that out. I've tested the workflow locally and it omits the CUDA identifier and uses ".cpu-" instead.

cfRod · 2025-10-16T13:40:06Z

.buildkite/release-pipeline.yaml

      DOCKER_BUILDKIT: "1"

+  # aarch64 build.
+  - label: "Build arm64 CPU wheel"


How are we differentiating arm64 CUDA wheel from non-CUDA wheel?
What naming convention are you using?

I think on pypi I see
vllm-0.11.0-cp38-abi3-manylinux2014_aarch64.whl. for the one that has CUDA dependency

you should see a cpu somewhere in the wheel name if it got built on CPU.
locally, I see: vllm-0.11.0rc2.dev481+g74704d455.cpu-cp310-cp310-linux_aarch64.whl

by the latest release, having "+cu..." part of the wheel name should indicate CUDA dependencies. I've tested the patched workflow locally, dropping sccache mentions, since that only seems to be supported in CI
, and got something similar to what Fadi mentioned.

Signed-off-by: Ioana Ghiban <ioana.ghiban@arm.com>

ghost · 2025-10-16T14:34:52Z

docker/Dockerfile.cpu

+ENV MAX_JOBS=${max_jobs}
+
+ARG USE_SCCACHE
+ARG SCCACHE_DOWNLOAD_URL=https://github.com/mozilla/sccache/releases/download/v0.8.1/sccache-v0.8.1-x86_64-unknown-linux-musl.tar.gz


for CUDA dependent AArch64 wheel build we're using this cache as well

fadara01 · 2025-10-20T11:07:30Z

@mgoin do you have any other feedback w.r.t this PR? Are we happy with it as is?

mgoin · 2025-10-20T11:09:06Z

I'm just a little hesitant with respect to not messing up the arm CUDA wheel, but if cpu shows up in the wheel name it should be fine. I think we can merge and validate. Thanks for the reminder ping

Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>

Signed-off-by: 0xrushi <6279035+0xrushi@users.noreply.github.com>

mergify bot added the ci/build label Oct 15, 2025

gemini-code-assist bot reviewed Oct 15, 2025

View reviewed changes

.buildkite/release-pipeline.yaml Outdated Show resolved Hide resolved

chatgpt-codex-connector bot reviewed Oct 15, 2025

View reviewed changes

.buildkite/release-pipeline.yaml Outdated Show resolved Hide resolved

ghost force-pushed the arm64-cpu-docker-pipeline branch from bd22e4e to 355b559 Compare October 15, 2025 18:45

fadara01 reviewed Oct 16, 2025

View reviewed changes

.buildkite/release-pipeline.yaml Show resolved Hide resolved

ghost force-pushed the arm64-cpu-docker-pipeline branch from 45d1999 to 6bcf6c2 Compare October 16, 2025 10:19

mgoin reviewed Oct 16, 2025

View reviewed changes

cfRod reviewed Oct 16, 2025

View reviewed changes

mgoin added the ready ONLY add when PR is ready to merge/full CI is needed label Oct 16, 2025

[CI][Release][Arm64] Add Arm64 CPU docker pipeline

947c644

Signed-off-by: Ioana Ghiban <ioana.ghiban@arm.com>

ghost force-pushed the arm64-cpu-docker-pipeline branch from 6bcf6c2 to 947c644 Compare October 16, 2025 14:33

ghost requested a review from bigPYJ1151 as a code owner October 16, 2025 14:33

ghost commented Oct 16, 2025

View reviewed changes

mgoin approved these changes Oct 20, 2025

View reviewed changes

mgoin merged commit 1c691f4 into vllm-project:main Oct 20, 2025
17 checks passed

adabeyta pushed a commit to adabeyta/vllm that referenced this pull request Oct 20, 2025

AArch64 CPU Docker pipeline (vllm-project#26931)

c42c738

faaany pushed a commit to faaany/vllm that referenced this pull request Oct 21, 2025

AArch64 CPU Docker pipeline (vllm-project#26931)

777a369

Zhuul pushed a commit to Zhuul/vllm that referenced this pull request Oct 21, 2025

AArch64 CPU Docker pipeline (vllm-project#26931)

2c80d2c

Ther-LF pushed a commit to Ther-LF/vllm that referenced this pull request Oct 22, 2025

AArch64 CPU Docker pipeline (vllm-project#26931)

30a6bfc

albertoperdomo2 pushed a commit to albertoperdomo2/vllm that referenced this pull request Oct 23, 2025

AArch64 CPU Docker pipeline (vllm-project#26931)

1e9db44

Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>

0xrushi pushed a commit to 0xrushi/vllm that referenced this pull request Oct 26, 2025

AArch64 CPU Docker pipeline (vllm-project#26931)

76e06a5

Signed-off-by: 0xrushi <6279035+0xrushi@users.noreply.github.com>

0xrushi pushed a commit to 0xrushi/vllm that referenced this pull request Oct 26, 2025

AArch64 CPU Docker pipeline (vllm-project#26931)

d3f9ad9

Signed-off-by: 0xrushi <6279035+0xrushi@users.noreply.github.com>

fadara01 mentioned this pull request Nov 4, 2025

[Bug]: Missing libarm_compute.so in Arm CPU pip installed wheels #28056

Open

1 task

ilmarkov pushed a commit to neuralmagic/vllm that referenced this pull request Nov 7, 2025

AArch64 CPU Docker pipeline (vllm-project#26931)

60ffe2b

Uh oh!

AArch64 CPU Docker pipeline #26931

AArch64 CPU Docker pipeline #26931

Uh oh!

Conversation

ghost commented Oct 15, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Test Plan

Test Result

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

mgoin Oct 16, 2025

Choose a reason for hiding this comment

Uh oh!

ghost Oct 17, 2025 • edited by ghost Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cfRod Oct 16, 2025

Choose a reason for hiding this comment

Uh oh!

cfRod Oct 16, 2025

Choose a reason for hiding this comment

Uh oh!

fadara01 Oct 16, 2025

Choose a reason for hiding this comment

Uh oh!

ghost Oct 17, 2025

Choose a reason for hiding this comment

Uh oh!

ghost Oct 16, 2025

Choose a reason for hiding this comment

Uh oh!

fadara01 commented Oct 20, 2025

Uh oh!

mgoin commented Oct 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ghost commented Oct 15, 2025 •

edited by github-actions bot

Loading

ghost Oct 17, 2025 •

edited by ghost

Loading

mgoin commented Oct 20, 2025 •

edited

Loading