Skip to content

Support S3 Sharded loading with RunAI Model Streamer #16317

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
Apr 22, 2025
Merged

Support S3 Sharded loading with RunAI Model Streamer #16317

merged 3 commits into from
Apr 22, 2025

Conversation

@omer-dayan
Copy link
Contributor

@omer-dayan omer-dayan commented Apr 9, 2025 •
edited by github-actions bot
Loading

In this PR we add support for using RunAI Model Streamer with sharded model files.

It allows the user to upload the sharded model files to S3:
image

and run this model using the following command:
vllm serve s3://core-llm/Llama-3-8b-4/ --tensor-parallel-size 4 --load-format runai_streamer_sharded

@github-actions
Copy link

github-actions bot commented Apr 9, 2025

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

@omer-dayan omer-dayan changed the title Support Sharded loading with RunAI Model Streamer Support S3 Sharded loading with RunAI Model Streamer Apr 9, 2025
@omer-dayan
Support Sharded loading with RunAI Model Streamer
c447ec3 
Signed-off-by: Omer Dayan (SW-GPU) <omer@run.ai>
@omer-dayan omer-dayan mentioned this pull request Apr 9, 2025
Closed
1 task
DarkLight1337
DarkLight1337 reviewed Apr 10, 2025
View reviewed changes
@omer-dayan @DarkLight1337
Update vllm/model_executor/model_loader/loader.py
78264e4 
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
Signed-off-by: Omer Dayan (SW-GPU) <omer@run.ai>
@DarkLight1337 DarkLight1337 added the ready ONLY add when PR is ready to merge/full CI is needed label Apr 20, 2025
@DarkLight1337
Copy link
Member

I can't seem to get docker build to pass, can you merge from main again?

DarkLight1337
DarkLight1337 approved these changes Apr 22, 2025
View reviewed changes
Copy link
Member

@DarkLight1337 DarkLight1337 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test failures look unrelated. Thanks for expanding this support!

@vllm-bot vllm-bot merged commit 71ce440 into vllm-project:main Apr 22, 2025
40 of 43 checks passed
@DarkLight1337 DarkLight1337 mentioned this pull request Apr 22, 2025
Merged
frieda-huang pushed a commit to frieda-huang/vllm that referenced this pull request Apr 23, 2025
@omer-dayan @DarkLight1337 @frieda-huang
Support S3 Sharded loading with RunAI Model Streamer (vllm-project#16317
a0ac262 
)

Signed-off-by: Omer Dayan (SW-GPU) <omer@run.ai>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
Signed-off-by: Frieda (Jingying) Huang <jingyingfhuang@gmail.com>
@omer-dayan omer-dayan mentioned this pull request Apr 24, 2025
Merged
jikunshang pushed a commit to jikunshang/vllm that referenced this pull request Apr 29, 2025
@omer-dayan @DarkLight1337 @dbyoung18
Support S3 Sharded loading with RunAI Model Streamer (vllm-project#16317
b2e15fe 
)

Signed-off-by: Omer Dayan (SW-GPU) <omer@run.ai>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
lk-chen pushed a commit to lk-chen/vllm that referenced this pull request Apr 29, 2025
@omer-dayan @DarkLight1337 @lk-chen
Support S3 Sharded loading with RunAI Model Streamer (vllm-project#16317
c6652ac 
)

Signed-off-by: Omer Dayan (SW-GPU) <omer@run.ai>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
adobrzyn pushed a commit to HabanaAI/vllm-fork that referenced this pull request Apr 30, 2025
@omer-dayan @DarkLight1337 @adobrzyn
Support S3 Sharded loading with RunAI Model Streamer (vllm-project#16317
f3aff39 
)

Signed-off-by: Omer Dayan (SW-GPU) <omer@run.ai>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
Signed-off-by: Agata Dobrzyniewicz <adobrzyniewicz@habana.ai>
RichardoMrMu pushed a commit to RichardoMrMu/vllm that referenced this pull request May 12, 2025
@omer-dayan @DarkLight1337
Support S3 Sharded loading with RunAI Model Streamer (vllm-project#16317
27f6095 
)

Signed-off-by: Omer Dayan (SW-GPU) <omer@run.ai>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
Signed-off-by: Mu Huai <tianbowen.tbw@antgroup.com>
@ckhordiasma ckhordiasma mentioned this pull request May 14, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@DarkLight1337 DarkLight1337 DarkLight1337 approved these changes

@WoosukKwon WoosukKwon Awaiting requested review from WoosukKwon

Assignees

No one assigned

Labels

ready ONLY add when PR is ready to merge/full CI is needed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@omer-dayan @DarkLight1337 @vllm-bot