Skip to content

[Misc] Use envs.VLLM_USE_RAY_COMPILED_DAG_CHANNEL_TYPE #15831

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 7 commits into from
Apr 1, 2025
Merged

[Misc] Use envs.VLLM_USE_RAY_COMPILED_DAG_CHANNEL_TYPE #15831

merged 7 commits into from
Apr 1, 2025

Conversation

ruisearch42
Copy link
Collaborator

@ruisearch42 ruisearch42 commented Mar 31, 2025 •
edited by github-actions bot
Loading

Use envs.VLLM_USE_RAY_COMPILED_DAG_CHANNEL_TYPE instead of envs.VLLM_USE_RAY_COMPILED_DAG_NCCL_CHANNEL.

Since ray 2.42 , the behavior of using with_tensor_transport(transport='auto') is that Compiled Graph will use NCCL if both the sender and receiver are using GPU (instead of CPU), which is almost always the case for vLLM. This PR allows overwriting the env var to use shared memory instead, which will be useful for debugging purposes. This also opens up opportunities to support additional channel types in future, say for different hardware.

The original VLLM_USE_RAY_COMPILED_DAG_NCCL_CHANNEL is rarely changed, so backward compatibility should not be a concern.

@ruisearch42
[Misc] Use env.VLLM_USE_RAY_COMPILED_DAG_CHANNEL_TYPE
982a152 
Signed-off-by: Rui Qiao <ruisearch42@gmail.com>
@github-actions GitHub Actions
Copy link

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

simon-mo
simon-mo approved these changes Mar 31, 2025
View reviewed changes
@ruisearch42
up
24520cd 
Signed-off-by: Rui Qiao <ruisearch42@gmail.com>
@ruisearch42
up
e29c0c9 
Signed-off-by: Rui Qiao <ruisearch42@gmail.com>
@ruisearch42
up
e241be3 
Signed-off-by: Rui Qiao <ruisearch42@gmail.com>
comaniac
comaniac approved these changes Mar 31, 2025
View reviewed changes
vllm/envs.py
# - "auto": use the default channel type
# - "nccl": use NCCL for communication
# - "shm": use shared memory and gRPC for communication
# This flag is ignored if VLLM_USE_RAY_COMPILED_DAG is not set.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIRC we enable CG in v1 by default? If so we should make it clear here.

Copy link
Collaborator Author

@ruisearch42 ruisearch42 Mar 31, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, this statement is technically correct :) In V1 with ray backend, we set VLLM_USE_RAY_COMPILED_DAG to 1.
I added a comment at the definition of VLLM_USE_RAY_COMPILED_DAG.

ruisearch42 and others added 2 commits March 31, 2025 22:51
@ruisearch42 @comaniac
Update vllm/executor/ray_distributed_executor.py
0f17c58 
Co-authored-by: Cody Yu <hao.yu.cody@gmail.com>
Signed-off-by: Rui Qiao <ruisearch42@gmail.com>
@ruisearch42
up
89c563e 
Signed-off-by: Rui Qiao <ruisearch42@gmail.com>
@comaniac comaniac added the ready ONLY add when PR is ready to merge/full CI is needed label Mar 31, 2025
@vllm-bot vllm-bot merged commit 8dd41d6 into vllm-project:main Apr 1, 2025
31 of 34 checks passed
Alex4210987 pushed a commit to LeiWang1999/vllm-bitblas that referenced this pull request Apr 5, 2025
@ruisearch42 @comaniac
@DarkLight1337 @Alex4210987
[Misc] Use envs.VLLM_USE_RAY_COMPILED_DAG_CHANNEL_TYPE (vllm-project#…
a8c6f83 
…15831)

Signed-off-by: Rui Qiao <ruisearch42@gmail.com>
Co-authored-by: Cody Yu <hao.yu.cody@gmail.com>
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: xinyuxiao <xinyuxiao2024@gmail.com>
lulmer pushed a commit to lulmer/vllm that referenced this pull request Apr 7, 2025
@ruisearch42 @comaniac
@DarkLight1337 @lulmer
[Misc] Use envs.VLLM_USE_RAY_COMPILED_DAG_CHANNEL_TYPE (vllm-project#…
ebc2914 
…15831)

Signed-off-by: Rui Qiao <ruisearch42@gmail.com>
Co-authored-by: Cody Yu <hao.yu.cody@gmail.com>
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: Louis Ulmer <ulmerlouis@gmail.com>
@ckhordiasma ckhordiasma mentioned this pull request Apr 17, 2025
Closed
lk-chen pushed a commit to lk-chen/vllm that referenced this pull request Apr 29, 2025
@ruisearch42 @comaniac
@DarkLight1337 @lk-chen
[Misc] Use envs.VLLM_USE_RAY_COMPILED_DAG_CHANNEL_TYPE (vllm-project#…
2003d17 
…15831)

Signed-off-by: Rui Qiao <ruisearch42@gmail.com>
Co-authored-by: Cody Yu <hao.yu.cody@gmail.com>
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
shreyankg pushed a commit to shreyankg/vllm that referenced this pull request May 3, 2025
@ruisearch42 @comaniac
@DarkLight1337 @shreyankg
[Misc] Use envs.VLLM_USE_RAY_COMPILED_DAG_CHANNEL_TYPE (vllm-project#…
cdef46d 
…15831)

Signed-off-by: Rui Qiao <ruisearch42@gmail.com>
Co-authored-by: Cody Yu <hao.yu.cody@gmail.com>
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
RichardoMrMu pushed a commit to RichardoMrMu/vllm that referenced this pull request May 12, 2025
@ruisearch42 @comaniac @DarkLight1337
[Misc] Use envs.VLLM_USE_RAY_COMPILED_DAG_CHANNEL_TYPE (vllm-project#…
f5139da 
…15831)

Signed-off-by: Rui Qiao <ruisearch42@gmail.com>
Co-authored-by: Cody Yu <hao.yu.cody@gmail.com>
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: Mu Huai <tianbowen.tbw@antgroup.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@comaniac comaniac comaniac approved these changes

@simon-mo simon-mo simon-mo approved these changes

Assignees

@simon-mo simon-mo

@youkaichao youkaichao

Labels

ready ONLY add when PR is ready to merge/full CI is needed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@ruisearch42 @comaniac @simon-mo @youkaichao @vllm-bot @DarkLight1337