[Example] add one-click runnable example for P2P NCCL XpYd #20246
Conversation
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels. Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can either: Add 🚀 |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Summary of Changes
Hello @KuntaiDu, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
This pull request introduces a new, self-contained example that allows users to quickly deploy and test the P2P NCCL XpYd disaggregated prefill feature within vLLM. It automates the setup of multiple vLLM instances and includes a basic benchmark to demonstrate the functionality.
Highlights
- New Example Script: I've added a new one-click runnable example script,
disagg_example_p2p_nccl_xpyd.sh, located inexamples/online_serving/disaggregated_serving_p2p_nccl_xpyd/. This script demonstrates how to set up and run vLLM with P2P NCCL XpYd disaggregated prefill. - Disaggregated Serving Setup: The example script configures and launches one vLLM prefiller instance (acting as a KV producer) and three vLLM decoder instances (acting as KV consumers), all communicating via
P2pNcclConnectorfor key-value transfer. - Automated Benchmarking: After successfully launching and verifying the vLLM servers, the script automatically executes a serving benchmark using
benchmark_serving.pyto simulate a workload against the disaggregated setup. - Prerequisite Checks: The script includes checks for necessary prerequisites such as the Hugging Face token, the availability of at least two GPUs, and the installation of required Python libraries (pandas, datasets, vllm, quart) to ensure a smooth execution experience.
Using Gemini Code Assist
The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.
Invoking Gemini
You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.
| Feature | Command | Description |
|---|---|---|
| Code Review | /gemini review |
Performs a code review for the current pull request in its current state. |
| Pull Request Summary | /gemini summary |
Provides a summary of the current pull request in its current state. |
| Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in issue comments and review comments. |
| Help | /gemini help |
Displays a list of available commands. |
Customization
To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.
Limitations & Feedback
Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.
You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.
Footnotes
-
Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩
There was a problem hiding this comment.
Code Review
This pull request adds a new example script for P2P NCCL disaggregated serving. The script is well-structured, but I've found a few issues related to correctness and robustness that should be addressed. My main concerns are an incorrect GPU count check which could lead to runtime failures, and missing shell script best practices for error handling. I've also pointed out a minor issue with signal trapping and a redundant environment variable definition.
Signed-off-by: KuntaiDu <kuntai@uchicago.edu>
Signed-off-by: KuntaiDu <kuntai@uchicago.edu>
…ents. Signed-off-by: KuntaiDu <kuntai@uchicago.edu>
github-actions
bot
added
the
ready
|
Did a force push to resolve DCO issue. |
…used for XpYd Signed-off-by: KuntaiDu <kuntai@uchicago.edu>
|
I'm probably doing something wrong, in which case please ignore this, but just in case it looks like it could be a bug: I tried DeepSeek using 1p1d based on this example, on a 8xH200 (i.e. 2x4xH200), and got this error on the decode server: Reproduction:# Install vllm:
cd /root
git clone https://github.com/vllm-project/vllm.git
cd vllm
# Pinned to ~most recent commit as of writing:
git reset --hard ecad851cbd0a2a6f9922fc9f3e94bde3f8220176
VLLM_USE_PRECOMPILED=1 pip install --editable .# Proxy server:
tmux new -s proxy
pip install quart==0.20.0
wget -O /root/disagg_proxy_p2p_nccl_xpyd.py "https://raw.githubusercontent.com/vllm-project/vllm/0e96cc9b7e473afda794fa3c32e83c391d9a3d27/examples/online_serving/disaggregated_serving_p2p_nccl_xpyd/disagg_proxy_p2p_nccl_xpyd.py"
python3 /root/disagg_proxy_p2p_nccl_xpyd.py# Prefill server:
tmux new -s prefill
CUDA_VISIBLE_DEVICES=0,1,2,3 VLLM_USE_V1=1 VLLM_WORKER_MULTIPROC_METHOD=spawn VLLM_MARLIN_USE_ATOMIC_ADD=1 vllm serve RedHatAI/DeepSeek-R1-0528-quantized.w4a16 --enforce-eager --host 0.0.0.0 --port 20003 --tensor-parallel-size 4 --max-model-len 8192 --max-seq-len-to-capture 8192 --enable-chunked-prefill --enable-prefix-caching --trust-remote-code --disable-log-requests --gpu-memory-utilization 0.95 --served-model-name deepseek-chat --compilation-config '{"full_cuda_graph": true}' --kv-transfer-config '{"kv_connector":"P2pNcclConnector","kv_role":"kv_producer","kv_buffer_size":"1e1","kv_port":"21001","kv_connector_extra_config":{"proxy_ip":"0.0.0.0","proxy_port":"30001","http_port":"20003","send_type":"PUT_ASYNC","nccl_num_channels":"16"}}'# Decode server:
tmux new -s decode
CUDA_VISIBLE_DEVICES=4,5,6,7 VLLM_USE_V1=1 VLLM_WORKER_MULTIPROC_METHOD=spawn VLLM_MARLIN_USE_ATOMIC_ADD=1 vllm serve RedHatAI/DeepSeek-R1-0528-quantized.w4a16 --enforce-eager --host 0.0.0.0 --port 20005 --tensor-parallel-size 4 --max-model-len 8192 --max-seq-len-to-capture 8192 --enable-chunked-prefill --enable-prefix-caching --trust-remote-code --disable-log-requests --gpu-memory-utilization 0.95 --served-model-name deepseek-chat --compilation-config '{"full_cuda_graph": true}' --kv-transfer-config '{"kv_connector":"P2pNcclConnector","kv_role":"kv_consumer","kv_buffer_size":"8e9","kv_port":"22001","kv_connector_extra_config":{"proxy_ip":"0.0.0.0","proxy_port":"30001","http_port":"20005","send_type":"PUT_ASYNC","nccl_num_channels":"16"}}'Then I sent a request at the proxy server, and got the error on the decode server. I tried removing all of these: |
|
@Abatom Would be nice if you could take a look |
Sure, I'll take a look. |
|
same issue @Abatom
|
Okay, I will reproduce DeepSeek-R1-0528. |
|
@josephrocca @renwuli Can the non-quantized version of DeepSeek-R1-0528 work normally on H200? |
|
@josephrocca @renwuli I don't have access to an H200. Could you help print out the type of |
|
@Abatom I don't have access to H200 either, but I can access to 2 H100s whose memory is 640GB in total, for the unquantized version of DeepSeek-R1 671B, do you have any suggestion in parallel config? |
The current P2P NCCL solution doesn't support pipeline parallelism (PP). We need one machine capable of running DeepSeek-R1 671B, and for a 1P1D setup, two machines are required. By the way, would it be possible for us to communicate via Slack? |
…ect#20246) Signed-off-by: KuntaiDu <kuntai@uchicago.edu> Signed-off-by: Jinzhen Lin <linjinzhen@hotmail.com>
4 participants
Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model.Purpose
Add a one-click runnable example for P2P NCCL XpYd introduced by #18242
(Optional) Documentation Update