From 452aae47ae1d2ce6dee6a389f5383dbe7ca888cf Mon Sep 17 00:00:00 2001 From: smile2game <2426827419@qq.com> Date: Tue, 8 Oct 2024 08:25:02 +0000 Subject: [PATCH 1/4] smile2game add 2024.09.09-2024.09.23 WeeklyReport --- .../[WeeklyReport]2024.09.09~2024.09.23.md | 62 +++++++++++++++++++ 1 file changed, 62 insertions(+) create mode 100644 WeeklyReports/Hackathon_7th/07_smile2game/[WeeklyReport]2024.09.09~2024.09.23.md diff --git a/WeeklyReports/Hackathon_7th/07_smile2game/[WeeklyReport]2024.09.09~2024.09.23.md b/WeeklyReports/Hackathon_7th/07_smile2game/[WeeklyReport]2024.09.09~2024.09.23.md new file mode 100644 index 00000000..3bdb6112 --- /dev/null +++ b/WeeklyReports/Hackathon_7th/07_smile2game/[WeeklyReport]2024.09.09~2024.09.23.md @@ -0,0 +1,62 @@ +### 姓名 +刘卉杰 + +### 实习项目 +自动并行张量切分机制预研&建设 + +### 本周工作 + +1. 学习jeff提交的 s_to_r case下的不均匀切分的pr，并调研其余cases + +参考链接 [67756pr](https://github.com/PaddlePaddle/Paddle/pull/67756) + + * 阅读 base_reshard_func.py,了解ReshardFunction基本类,需要我们实现is_suitable和reshard的 + * 阅读 s_to_r_reshard_func.py: + 1. SToRReshardFunction.reshard() + 1.1. padding 不均匀需要先padding + 1.2. reshard_s_to_r_with_padding 具体怎么通信得到复制，并去除padding + 2. SToRReshardFunctionCrossMesh.reshard() + 1.1 先用 SameStatusReshardFunction() + 1.2 在用 SToRReshardFunction.reshard() + * 源码编译之后，跑一下test + + +2. 进行code-reading和学习，参考链接 [code-reading](https://github.com/PaddlePaddle/community/tree/master/pfcc/paddle-code-reading) + + * 学习PIR机制的概念 + * 了解多层级Dialect + * 了解Pass体系 + * 学习自动并行机制 + * 了解动手、静手、自动并行方式（切分推导-切分转换） + * 了解分布式张量的三种 placements 和 ProcessMesh + * 了解了reshard的含义和多种case + * 了解了自动并行和分布式策略：数据并行，张量并行，流水线并行，3D混合并行 + + + +3. **问题疑惑与解答** + + * s-to-r的情况,是说shard状态是被不均匀切分了吗？这里的s-to-r的src value本身就是不均匀的,那这是哪里来的？ + 答：xxx + * 目前s-to-r的不均匀切分,Processmesh的dimNum还是1，多重切分指的是后面的 crossmesh吗？ + 答：xxx + * 目前是不是只有 s-to-r实现了这个不均匀切分，其余情况p-r,p-s,r-p,r-s,是否也需要实现不均匀切分？ + 答：xxx + * tmp_src_type = paddle.base.libpaddle.pir.cvt_to_dist_type不太明白为什么需要这样做？ + 答：好像是为了解决静态图的问题，以及后面的op_dist_attribute。还需要再学一下pir以及get_defining_op() + * 这些程序，执行时候就和cuda并行一样，每个device上获得相同的代码，根据rank来实现不同的操作吗？ + 答：xxx + * 问题出现的动机是，不均匀时候没法直接用all_gather吗？ + 答：xxx + + + + +### 下周工作 + +1. 尝试 r-to-s的不均匀切分实现 +2. 尝试 s-to-r的多重切分实现 + + +### 导师点评 +请联系导师填写 From ae157b3b11ffdfbe84319a663236eadda08afcef Mon Sep 17 00:00:00 2001 From: smile2game <2426827419@qq.com> Date: Tue, 5 Nov 2024 07:57:43 +0000 Subject: [PATCH 2/4] 10.25 weekly report --- .../[WeeklyReport]2024.10.08~2024.10.25.md | 37 +++++++++++++++++++ 1 file changed, 37 insertions(+) create mode 100644 WeeklyReports/Hackathon_7th/07_smile2game/[WeeklyReport]2024.10.08~2024.10.25.md diff --git a/WeeklyReports/Hackathon_7th/07_smile2game/[WeeklyReport]2024.10.08~2024.10.25.md b/WeeklyReports/Hackathon_7th/07_smile2game/[WeeklyReport]2024.10.08~2024.10.25.md new file mode 100644 index 00000000..81eccdb5 --- /dev/null +++ b/WeeklyReports/Hackathon_7th/07_smile2game/[WeeklyReport]2024.10.08~2024.10.25.md @@ -0,0 +1,37 @@ +### 姓名 +刘卉杰 + +### 实习项目 +自动并行张量切分机制预研&建设 + +### 本周工作 + +1. 开发 `p_to_s_reshard_func.py` + 创建了静态图下的p_to_s_reshard_func.py,并在其中参考着s_to_r_reshard_func.py编写了逻辑为 + if balanced: + reduce_scatter + else: + padding + reduce_scatter + split + + 目前存在的问题： + (1) reduce_scatter返回尺寸不对，如 我希望4 x 7的partial张量，在 0维度，变成 2 x 7 - 2 x 7 的shard张量(global shape还是4 x 7)，但是目前返回 global shape是 2 x 7的张量 + +2. 参考s_to_r的单测，编写p_to_s单测`p_to_s_unittest.py`和`reshard_p_to_s.py`并能够跑通 + +代码暂时都存放在我fork下的github仓库，链接为[个人fork下仓库](https://github.com/smile2game/Paddle/tree/note/unittest) + + +### 存在的问题 +reduce_scatter返回尺寸不对，如 我希望4 x 7的partial张量，在 0维度，变成 2 x 7 - 2 x 7 的shard张量(global shape还是4 x 7)，但是目前返回 global shape是 2 x 7的张量 + +### 下周工作 + +1. 修复目前 p_to_s_reshard_func.py存在的问题 +2. 开始 r_to_s_reshard_func.py实现 + + + +### 导师点评 +请联系导师填写 From b1282623db032324f89ca77e1062144b636f298f Mon Sep 17 00:00:00 2001 From: smile2game <2426827419@qq.com> Date: Wed, 6 Nov 2024 13:52:23 +0000 Subject: [PATCH 3/4] WeeklyReport 10.08-10.25 --- .../07_smile2game/[WeeklyReport]2024.09.09~2024.09.23.md | 2 +- .../07_smile2game/[WeeklyReport]2024.10.08~2024.10.25.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/WeeklyReports/Hackathon_7th/07_smile2game/[WeeklyReport]2024.09.09~2024.09.23.md b/WeeklyReports/Hackathon_7th/07_smile2game/[WeeklyReport]2024.09.09~2024.09.23.md index 3bdb6112..b2815614 100644 --- a/WeeklyReports/Hackathon_7th/07_smile2game/[WeeklyReport]2024.09.09~2024.09.23.md +++ b/WeeklyReports/Hackathon_7th/07_smile2game/[WeeklyReport]2024.09.09~2024.09.23.md @@ -59,4 +59,4 @@ ### 导师点评 -请联系导师填写 +通过 diff --git a/WeeklyReports/Hackathon_7th/07_smile2game/[WeeklyReport]2024.10.08~2024.10.25.md b/WeeklyReports/Hackathon_7th/07_smile2game/[WeeklyReport]2024.10.08~2024.10.25.md index 81eccdb5..8840ca44 100644 --- a/WeeklyReports/Hackathon_7th/07_smile2game/[WeeklyReport]2024.10.08~2024.10.25.md +++ b/WeeklyReports/Hackathon_7th/07_smile2game/[WeeklyReport]2024.10.08~2024.10.25.md @@ -34,4 +34,4 @@ reduce_scatter返回尺寸不对，如 我希望4 x 7的partial张量，在 0维 ### 导师点评 -请联系导师填写 +通过 From 50e0d1124e1f3a3b381890f04bdc2e0673d3aa4f Mon Sep 17 00:00:00 2001 From: smile2game <2426827419@qq.com> Date: Sat, 23 Nov 2024 09:30:18 +0000 Subject: [PATCH 4/4] smile2game add 2024.10.28~2024.11.22 WeeklyReport --- .../[WeeklyReport]2024.10.28~2024.11.22.md | 73 +++++++++++++++++++ 1 file changed, 73 insertions(+) create mode 100644 WeeklyReports/Hackathon_7th/07_smile2game/[WeeklyReport]2024.10.28~2024.11.22.md diff --git a/WeeklyReports/Hackathon_7th/07_smile2game/[WeeklyReport]2024.10.28~2024.11.22.md b/WeeklyReports/Hackathon_7th/07_smile2game/[WeeklyReport]2024.10.28~2024.11.22.md new file mode 100644 index 00000000..91ef527f --- /dev/null +++ b/WeeklyReports/Hackathon_7th/07_smile2game/[WeeklyReport]2024.10.28~2024.11.22.md @@ -0,0 +1,73 @@ +### 姓名 +刘卉杰 + +### 实习项目 +自动并行张量切分机制预研&建设 + +### 本周工作 + +1. 开发 `p_to_s_reshard_func.py`,代码全局尺寸修正： + +reduce_scatter: +```python +dst_value = paddle._C_ops.reduce_scatter( + src_value, group.id, len(src_dist_attr.process_mesh.process_ids) + ) + out_global_shape = dst_value.shape + out_global_shape[split_axis] = ( + padding_num + out_global_shape[split_axis] + ) + out_global_shape[split_axis] = ( + padding_num + out_global_shape[split_axis] + ) + dst_tmp_type = paddle.pir.create_shaped_type( + dst_value.type(), out_global_shape + ) + dst_tmp_type = paddle.base.libpaddle.pir.cvt_to_dist_type( + dst_tmp_type, dst_dist_attr + ) +``` +split: +```python +dst_value = paddle._C_ops.split( + dst_value, + [ + dst_value.shape[split_axis] - padding_num, + padding_num, + ], + 0, + )[0] +``` + +2. pr描述修正 + +代码暂时都存放在我fork下的github仓库，链接为[个人fork下仓库](https://github.com/smile2game/Paddle/tree/note/unittest) + +3. 代码风格修正 + +``` +pip isntall pre-commit +pre-commit run --files python/paddle/distributed/auto_parallel/static/reshard_funcs/p_to_s_reshard_func.py \ + test/auto_parallel/pir/pir_reshard_p_to_s.py \ + test/auto_parallel/pir/test_pir_reshard_p_to_s.py +``` +4.提交pr +``` +git add . +git commit --amend --no-edit +git push --force +``` +pr链接:https://github.com/PaddlePaddle/Paddle/pull/69265 + + + +### 下周工作 + +1. 确保 p_to_s的pr能成功合入 +2. 开始 r_to_s_reshard_func.py实现 +3. 找老师咨询多重切分相关pr进行学习和开发 + + + +### 导师点评 +通过