[Reshard] Support reshard s to s on same placement #57210

LiYuRio · 2023-09-12T02:45:56Z

PR types

New features

PR changes

Others

Description

Pcard-73145

在一维mesh上shard到shard状态的转换，工作原理：

以形状为(a, b, c)的tensor在有n个进程的一维mesh上做切分转换为例。

输入dims_mapping为[-1, -1, 0]，输出dims_mapping为[0, -1, -1]，每个进程上需要从(a, b, c/n)变成(a/n, b, c)。
- 首先，做all_to_all，shape不变，值进行交换。这时，0维的值已经和其他进程交换过了。
- 其次，做reshape，形状会从(a, b, c/n)变成(n, a/n, b, c/n)
- 然后，做transpose，形状会从(n, a/n, b, c/n)变成(a/n, b, n, c/n)
- 最后，做reshape，形状会从(a/n, b, n, c/n)变成(a/n, b, c)
输入dims_mapping为[0, -1, -1]，输出dims_mapping为[-1, -1, 0]，每个进程上需要从(a/n, b, c)变成(a, b, c/n)。
- 首先，做reshape，形状会从(a/n, b, c)变成(a/n, b, n, c/n)
- 其次，做transpose，形状会从(a/n, b, n, c/n)变成(n, a/n, b, c/n)
- 然后，做reshape，形状会从(n, a/n, b, c/n)变成(a, b, c/n)
- 最后，做all_to_all，shape不变，值进行交换。这时，0维的值已经和其他进程交换过了，补全了0维缺失的数据

TODO：

对非均匀切分做硬性检查
all_to_all目前不支持cpu，需要写对应的cpu kernel

paddle-bot · 2023-09-12T02:46:00Z

你的PR提交成功，感谢你对开源项目的贡献!
请关注后续CI自动化测试结果，详情请参考Paddle-CI手册。
Your PR has been submitted. Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

chenwhql · 2023-09-13T08:02:46Z

paddle/phi/core/distributed/auto_parallel/s_to_s_reshard_function.cc

+
+#include "paddle/phi/core/distributed/auto_parallel/s_to_s_reshard_function.h"
+
+#include "glog/logging.h"


这个头文件好像没有使用

chenwhql · 2023-09-13T08:10:22Z

paddle/phi/core/distributed/auto_parallel/s_to_s_reshard_function.cc

+  const auto& logical_ddim = in.dims();
+  int64_t nranks = in_process_ids.size();
+  int64_t in_split_axis =
+      GetSplitAxisWithDimsMapping(in.dist_attr().dims_mapping()).begin()->first;


会不会第一个之后还有非-1的维度

在这个场景下，是没有的，因为限制了mesh的维度是1，也就是说这里只会出现一次shard axis。

chenwhql · 2023-09-13T08:12:10Z

paddle/phi/core/distributed/auto_parallel/s_to_s_reshard_function.cc

+
+    DenseTensor out_reshape1;
+    RESHARD_FUNCTOR(
+        dev_ctx, Reshape, dtype, in.value(), pre_shape_vec, &out_reshape1);


这里reshape有没有可能引入不必要的拷贝，浅拷贝一个输入传进去会不会好一点

reshape会修改值的排列顺序，暂时先避免可能对input底层DenseTensor产生的修改，后续有需要可以进一步优化

chenwhql · 2023-09-13T08:15:05Z

paddle/phi/core/distributed/auto_parallel/s_to_s_reshard_function.cc

+        dev_ctx, Reshape, dtype, in.value(), pre_shape_vec, &out_reshape1);
+
+    // 1.2 calc the the desire axis and transpose
+    std::vector<int> axis;


int64_t到int极端情况下是不是有可能截断，要不直接用vector int64？

因为axis不会超过32位的限制，已经全部改成vector int

chenwhql

LGTM

XieYunshen

LGTM for set_tests_properties(test_reshard_s_to_s PROPERTIES LABELS "RUN_TYPE=EXCLUSIVE" TIMEOUT 100)

* support reshard s to s * refine, remove useless code * reduce changable file

LiYuRio force-pushed the dev_s_to_s branch 2 times, most recently from 17f53e7 to 0316a9c Compare September 12, 2023 03:49

support reshard s to s

2a2b727

LiYuRio force-pushed the dev_s_to_s branch from 0316a9c to 2a2b727 Compare September 12, 2023 06:41

chenwhql reviewed Sep 13, 2023

View reviewed changes

refine, remove useless code

Verified

This commit was signed with the committer’s verified signature.

suemto suem

GPG key ID: 8E2A4F179C6D8B52

Verified
Learn about vigilant mode

544b039

LiYuRio force-pushed the dev_s_to_s branch from 36e8ad5 to 544b039 Compare September 14, 2023 03:09

reduce changable file

75df74b

chenwhql approved these changes Sep 15, 2023

View reviewed changes

XieYunshen approved these changes Sep 15, 2023

View reviewed changes

LiYuRio merged commit da98117 into PaddlePaddle:develop Sep 15, 2023

LiYuRio deleted the dev_s_to_s branch September 15, 2023 03:05

danleifeng pushed a commit to danleifeng/Paddle that referenced this pull request Nov 14, 2023

[Reshard] Support reshard s to s on same placement (PaddlePaddle#57210)

6a9f90e

* support reshard s to s * refine, remove useless code * reduce changable file

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Reshard] Support reshard s to s on same placement #57210

[Reshard] Support reshard s to s on same placement #57210

LiYuRio commented Sep 12, 2023 •

edited

Loading

paddle-bot bot commented Sep 12, 2023

chenwhql Sep 13, 2023

chenwhql Sep 13, 2023

LiYuRio Sep 13, 2023

chenwhql Sep 13, 2023

LiYuRio Sep 15, 2023

chenwhql Sep 13, 2023

LiYuRio Sep 15, 2023

chenwhql left a comment

XieYunshen left a comment


		#include "paddle/phi/core/distributed/auto_parallel/s_to_s_reshard_function.h"

		#include "glog/logging.h"

[Reshard] Support reshard s to s on same placement #57210

[Reshard] Support reshard s to s on same placement #57210

Conversation

LiYuRio commented Sep 12, 2023 • edited Loading

PR types

PR changes

Description

paddle-bot bot commented Sep 12, 2023

chenwhql Sep 13, 2023

Choose a reason for hiding this comment

chenwhql Sep 13, 2023

Choose a reason for hiding this comment

LiYuRio Sep 13, 2023

Choose a reason for hiding this comment

chenwhql Sep 13, 2023

Choose a reason for hiding this comment

LiYuRio Sep 15, 2023

Choose a reason for hiding this comment

chenwhql Sep 13, 2023

Choose a reason for hiding this comment

LiYuRio Sep 15, 2023

Choose a reason for hiding this comment

chenwhql left a comment

Choose a reason for hiding this comment

XieYunshen left a comment

Choose a reason for hiding this comment

LiYuRio commented Sep 12, 2023 •

edited

Loading