[Misc] support multi-node data parallel #15863

zxfan-cpu · 2025-04-01T07:37:17Z

ray-based support for multi-node dp and reduce start_dp_msg communication in DPAsyncMPClient class.

github-actions · 2025-04-01T07:37:25Z

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

tlrmchlsmth · 2025-04-01T18:09:26Z

vllm/model_executor/layers/fused_moe/layer.py

        vllm_config = get_current_vllm_config()
        use_ep = (vllm_config.parallel_config.enable_expert_parallel
-                  and self.tp_size > 1)
+                  and self.tp_size * self.dp_size > 1)


This fix is good. Without it, everything will work OK but the MoE layers will run with TP instead of EP (pretty sure that's true at least). Probably worth putting into its own PR

Thanks for catching this!

njhill · 2025-04-03T03:48:49Z

Hi @zxfan-cpu, thanks for this. I'm working on non-ray multi-node support first, which should be ready within the next day or so. It would be great to add ray support after that.

Could you open a separate PR with the fix that @tlrmchlsmth commented on above?

I will also look closer at the start_dp_msg changes, though I'm not sure about whether they would make a measurable difference performance-wise.

mergify · 2025-04-04T19:57:23Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @zxfan-cpu.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

ruisearch42

Thanks for the PR! It's great to have support for DP with Ray.

I left some comments, please take a look. Also, are there any testing done so far?

ruisearch42 · 2025-04-25T16:43:03Z

vllm/executor/ray_utils.py

+
+        head_ip = get_ip()
+
+        def sort_by_driver_then_worker_ip(ip_and_id):


nit: ip_and_id is not accurate, as the tuple is (ip, node_id, int(num_gpus))

Very good advice, thank you.

ruisearch42 · 2025-04-25T16:45:02Z

vllm/executor/ray_utils.py

+                    num_gpus = node_info["Resources"].get("GPU", 0)
+                    node_gpu_mapping.append((ip, node_id, int(num_gpus)))
+            node_gpu_mapping = sorted(node_gpu_mapping, key=sort_by_driver_then_worker_ip)
+            return node_gpu_mapping


nit: the return value is not a mapping, but a list. Also consider update method name.

Very good advice, thank you.

ruisearch42 · 2025-04-25T16:50:19Z

vllm/executor/ray_utils.py

+                if dp_rank * world_size < accumulated_gpus + num_gpus:
+                    gpu_index = dp_rank * world_size - accumulated_gpus
+                    return (ip, node_id, gpu_index)


Question: what happens if a DP rank spans across multiple nodes?

The world size here is identical to tensor_parallel_size (for V1 do not support pipeline parallel), therefore, it's OK if a DP rank spans across multiple nodes.

hmm, V1 does support pipeline parallel. Also, can you elaborate a bit "it's OK if a DP rank spans across multiple nodes"? How are nodes assigned to each DP rank?

ruisearch42 · 2025-04-25T16:54:28Z

vllm/executor/ray_utils.py

+            return (0 if ip == head_ip else 1, ip)
+
+        def _get_gpu_mapping():
+            nodes = ray.nodes()


Currently we use a placement group to define the resources for vLLM. This PR directly uses node resources from the cluster, without respecting the placement group. I think we will need to make things consistent and have a well-defined resource allocation protocol.

Oneal65 · 2025-04-28T07:53:17Z

Hi @zxfan-cpu, thanks for this. I'm working on non-ray multi-node support first, which should be ready within the next day or so. It would be great to add ray support after that.

Could you open a separate PR with the fix that @tlrmchlsmth commented on above?

I will also look closer at the start_dp_msg changes, though I'm not sure about whether they would make a measurable difference performance-wise.

Hi, Is the non-ray multi-node support version ready?

ruisearch42 · 2025-04-30T01:32:20Z

vllm/executor/ray_utils.py

+                if dp_rank * world_size < accumulated_gpus + num_gpus:
+                    gpu_index = dp_rank * world_size - accumulated_gpus
+                    return (ip, node_id, gpu_index)


hmm, V1 does support pipeline parallel. Also, can you elaborate a bit "it's OK if a DP rank spans across multiple nodes"? How are nodes assigned to each DP rank?

ruisearch42 · 2025-04-30T01:32:39Z

vllm/executor/ray_utils.py

+
+        node_gpu_mapping = _get_gpu_mapping()
+        current_dp_rank = parallel_config.data_parallel_rank
+        selected_node_ip, selected_node_id, gpu_index = _find_target_gpu(node_gpu_mapping, current_dp_rank)


gpu_index is not actually used?

njhill · 2025-04-30T03:13:44Z

Hi @zxfan-cpu, thanks for this. I'm working on non-ray multi-node support first, which should be ready within the next day or so. It would be great to add ray support after that.
Could you open a separate PR with the fix that @tlrmchlsmth commented on above?
I will also look closer at the start_dp_msg changes, though I'm not sure about whether they would make a measurable difference performance-wise.

Hi, Is the non-ray multi-node support version ready?

@Oneal65 yes, in PR #15977, should hopefully be merged soon. I am working on a follow-on to this to support multiple API server processes, which is needed to avoid the front-end process becoming a bottleneck when there are multiple DP engines.

github-actions · 2025-07-31T02:39:31Z

This pull request has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this pull request should remain open. Thank you!

github-actions · 2025-08-31T02:10:36Z

This pull request has been automatically closed due to inactivity. Please feel free to reopen if you intend to continue working on it. Thank you!

support multi-node dp

f02c649

zxfan-cpu requested review from WoosukKwon, alexm-redhat, comaniac, njhill, robertgshaw2-redhat and ywang96 as code owners April 1, 2025 07:37

mergify bot added the v1 label Apr 1, 2025

tlrmchlsmth reviewed Apr 1, 2025

View reviewed changes

zxfan-cpu mentioned this pull request Apr 3, 2025

[V1] DP scale-out (2/N): Decouple engine process management and comms #15977

Merged

4 tasks

mergify bot added the needs-rebase label Apr 4, 2025

ruisearch42 reviewed Apr 25, 2025

View reviewed changes

ruisearch42 reviewed Apr 30, 2025

View reviewed changes

github-actions bot added the stale Over 90 days of inactivity label Jul 31, 2025

github-actions bot closed this Aug 31, 2025


		head_ip = get_ip()

		def sort_by_driver_then_worker_ip(ip_and_id):

Uh oh!

[Misc] support multi-node data parallel #15863

[Misc] support multi-node data parallel #15863

Uh oh!

Conversation

zxfan-cpu commented Apr 1, 2025

Uh oh!

github-actions bot commented Apr 1, 2025

Uh oh!

tlrmchlsmth Apr 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

njhill commented Apr 3, 2025

Uh oh!

mergify bot commented Apr 4, 2025

Uh oh!

ruisearch42 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Oneal65 commented Apr 28, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

njhill commented Apr 30, 2025

Uh oh!

github-actions bot commented Jul 31, 2025

Uh oh!

github-actions bot commented Aug 31, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

tlrmchlsmth Apr 1, 2025 •

edited

Loading