[AutoParallel] Add paddle.distributed.dtensor_from_fn api #56565

yangxiaoyu14 · 2023-08-23T07:12:00Z

PR types

New features

PR changes

APIs

Description

Pcard-73145

[AutoParallel] Add paddle.distributed.dtensor_from_fn api

… deve dtensor_from_fn first edition

… add_dtensor_from_fn_api

CLAassistant · 2023-08-23T07:12:06Z

All committers have signed the CLA.

… add_dtensor_from_fn_api delete /Paddle/build/test/auto_parallel/test_dist_tensor.py

… add_dtensor_from_fn_api delete build/test/auto_parallel/test_dist_tensor.py

chenwhql · 2023-08-24T02:50:05Z

python/paddle/distributed/auto_parallel/api.py

@@ -24,7 +24,7 @@
 class DistAttr(core.TensorDistAttr):
    """
    DistAttr specifies how tensors are distributed or sliced on ProcessMesh.
-
+        


多余的空格缩进可以移除

chenwhql · 2023-08-24T02:53:22Z

python/paddle/distributed/auto_parallel/api.py

+        fn, dist_attr, *args, **kwargs
+):
+    """
+    Construct a Distributed Tensor from a function of arguments.


这里是不是强调下是paddle api funciton，不是任意的function都可以

chenwhql · 2023-08-24T02:55:42Z

python/paddle/distributed/auto_parallel/api.py

+    Construct a Distributed Tensor from a function of arguments.
+
+    Args:
+        fn (callable): A callable function that takes arguments of Distributed Tensor and returns tensor.


这句解释不太准确，翻译过来是：“一个可调用的函数，它接受分布式张量作为参数，并返回张量”，应该不是接收分布式张量作为参数吧

改为：fn (callable): A paddle api function that takes arguments of *args, **kwargs and returns tensor.

这里修改了吗？

chenwhql · 2023-08-24T02:56:21Z

python/paddle/distributed/auto_parallel/api.py

+        import paddle
+        import paddle.distribute as dist
+
+        def generate_tensor():


这个函数有什么作用？是不是应该移除

chenwhql · 2023-08-24T02:57:25Z

test/auto_parallel/test_dist_tensor.py

+class TestDistributedTensor(unittest.TestCase):
+    def test_dtensor_from_fn(self):
+        # Define a function for generating a tensor
+        def generate_tensor_ones():


这几个函数好像都没有用到，可以移除

chenwhql · 2023-08-24T02:58:20Z

test/auto_parallel/test_dist_tensor.py

+        result_random = dist.dtensor_from_fn(paddle.rand, dist_attr=dist_attr, shape=[2, 3])
+        self.assertIsInstance(result_random, paddle.Tensor)
+        self.assertEqual(result_random.shape, [2, 3])
+        self.assertEqual(result_random.dist_attr, dist_attr)


是不是需要加几个异常case，测试一下报错的情况

是不是也测试下静态图？

… add_dtensor_from_fn_api resolve issues raised by code review

sunzhongkai588 · 2023-08-29T08:04:20Z

python/paddle/distributed/auto_parallel/api.py

+        import paddle
+        import paddle.distribute as dist
+
+        # Create a distributed attribute
+        mesh = dist.ProcessMesh([[2, 4, 5], [0, 1, 3]], dim_names=["x", "y"])
+        dist_attr = dist.DistAttr(mesh=mesh, sharding_specs=['x', 'y'])
+
+        # Call the function dtensor_from_fn with dist_attr parameter
+        d_tensor = dist.dtensor_from_fn(paddle.ones, dist_attr=dist_attr, shape=[2, 3])
+
+        print(d_tensor)


代码示例部分请采用google style样式，参考
https://www.paddlepaddle.org.cn/documentation/docs/zh/develop/dev_guides/style_guide_and_references/code_example_writing_specification_cn.html

参考学习后重新提交，感谢

… add_dtensor_from_fn_api change sample codes " >>>" to ">>> "

chenwhql · 2023-08-30T03:10:14Z

python/paddle/distributed/auto_parallel/api.py

+
+    Args:
+        fn (callable): A paddle api function that takes arguments of *args, **kwargs and returns tensor.
+        dist_attr(paddle.distributed.DistAttr): Specify how tensors are distributed or sliced on ProcessMesh.


fn和(callable之间有空格，dist_attr和括号之间也建议增加空格

chenwhql · 2023-08-30T03:10:59Z

python/paddle/distributed/auto_parallel/api.py

+    Args:
+        fn (callable): A paddle api function that takes arguments of *args, **kwargs and returns tensor.
+        dist_attr(paddle.distributed.DistAttr): Specify how tensors are distributed or sliced on ProcessMesh.
+        *args: A list of arguments to be passed to the ``fn`` function.


建议也增加括号统一格式

*args (tuple): **kwargs (dict):

这里两个参数一个是tuple，一个是dict，不是list，建议区分一下

chenwhql · 2023-08-30T03:12:26Z

python/paddle/distributed/auto_parallel/api.py

+
+    .. code-block:: python
+
+    >>> import paddle


这里代码是不是需要缩进4个空格

chenwhql · 2023-08-30T03:13:38Z

test/auto_parallel/test_dist_tensor.py

+        mesh = dist.ProcessMesh([[2, 4, 5], [0, 1, 3]], dim_names=["x", "y"])
+        dist_attr = dist.DistAttr(mesh=mesh, sharding_specs=['x', 'y'])
+
+        # Test with generate_tensor_ones()


这个注释对应的函数已经没有了，注释建议修改一下，和下面的代码关联不明确，例如改为Test with paddle.ones

决定删除了不对应的注释

chenwhql · 2023-08-30T03:16:15Z

test/auto_parallel/test_dist_tensor.py

@@ -53,6 +53,56 @@ def test_dist_tensor_creation(self):
        self.assertEqual(dist_tensor_with_tensor.dist_attr, dist_attr)


+class TestDistributedTensor(unittest.TestCase):


测试类名要和测试内容对应，改为TestDistTensorFromFn？

chenwhql · 2023-08-30T03:17:11Z

test/auto_parallel/test_dist_tensor.py

@@ -53,6 +53,56 @@ def test_dist_tensor_creation(self):
        self.assertEqual(dist_tensor_with_tensor.dist_attr, dist_attr)


+class TestDistributedTensor(unittest.TestCase):
+    def test_dtensor_from_fn(self):


目前还是没有测试静态图，需要再补充一下

补充了静态图enable_static()状态下的测试

… add_dtensor_from_fn_api Get the latest code

sunzhongkai588 · 2023-09-01T09:37:01Z

python/paddle/distributed/auto_parallel/api.py

+    Examples:
+
+    .. code-block:: python
+
+        >>> import paddle
+        >>> import paddle.distribute as dist
+        >>> # Create a distributed attribute
+        >>> mesh = dist.ProcessMesh([0, 1], dim_names=["x"])
+        >>> dist_attr = dist.DistAttr(mesh=mesh, sharding_specs=[None])
+        >>> # Call the function dtensor_from_fn with dist_attr parameter
+        >>> d_tensor = dist.dtensor_from_fn(paddle.ones, dist_attr=dist_attr, shape=[1])
+        >>> print(d_tensor)


code-block部分及以下整体往右加4个缩进

同时看ci检查，好像没有 paddle.distribute，应该改成 paddle.distributed 吧？务必注意代码示例要能跑通

Suggested change

Examples:

.. code-block:: python

>>> import paddle

>>> import paddle.distribute as dist

>>> # Create a distributed attribute

>>> mesh = dist.ProcessMesh([0, 1], dim_names=["x"])

>>> dist_attr = dist.DistAttr(mesh=mesh, sharding_specs=[None])

>>> # Call the function dtensor_from_fn with dist_attr parameter

>>> d_tensor = dist.dtensor_from_fn(paddle.ones, dist_attr=dist_attr, shape=[1])

>>> print(d_tensor)

Examples:

.. code-block:: python

>>> import paddle

>>> import paddle.distributed as dist

>>> # Create a distributed attribute

>>> mesh = dist.ProcessMesh([0, 1], dim_names=["x"])

>>> dist_attr = dist.DistAttr(mesh=mesh, sharding_specs=[None])

>>> # Call the function dtensor_from_fn with dist_attr parameter

>>> d_tensor = dist.dtensor_from_fn(paddle.ones, dist_attr=dist_attr, shape=[1])

>>> print(d_tensor)

chenwhql · 2023-09-01T12:28:06Z

test/auto_parallel/test_dist_tensor.py

+            self.assertIsInstance(result, paddle.fluid.framework.Variable)
+            self.assertEqual(result.shape, (1,))
+
+        # Test with generate_tensor_zeros()


这里的注释还需要删除一下

chenwhql · 2023-09-01T12:28:50Z

test/auto_parallel/test_dist_tensor.py

+            self.assertIsInstance(result, paddle.fluid.framework.Variable)
+            self.assertEqual(result.shape, (1,))
+
+        # Test with generate_tensor_random()


这里的注释的也是

chenwhql

LGTM

zhiqiu · 2023-09-07T03:25:08Z

python/paddle/distributed/auto_parallel/api.py

+    Construct a Distributed Tensor from a function of arguments.
+
+    Args:
+        fn (callable): A callable function that takes arguments of Distributed Tensor and returns tensor.


这里修改了吗？

zhiqiu · 2023-09-07T03:27:03Z

python/paddle/distributed/auto_parallel/api.py

+    Examples:
+
+        .. code-block:: python
+            >>> import paddle


示例代码一般不加 >>> 吧？

咨询过钟凯，目前的新要求是要加

zhiqiu · 2023-09-07T03:28:03Z

test/auto_parallel/test_dist_tensor.py

+    def run_dtensor_from_fn(self):
+        # Create a distributed attribute
+        mesh = dist.ProcessMesh([0, 1], dim_names=["x"])
+        dist_attr = dist.DistAttr(mesh=mesh, sharding_specs=[None])


是不是增加测试一个sharding_specs不为None的版本？

这里本来是有sharding_specs不为None的版本的，但是后来雨芮建议改成None，我就改了

sunzhongkai588 · 2023-09-07T06:28:34Z

python/paddle/distributed/auto_parallel/api.py

+        .. code-block:: python
+            >>> import paddle


Suggested change

.. code-block:: python

>>> import paddle

.. code-block:: python

>>> import paddle

code-block 下加个空行，否则无法正常预览代码块

done， thx

jeff41404

LGTM for API

… add_dtensor_from_fn_api Pull the latest code to resolve conflicts 20230911

…licts

sunzhongkai588

LGTM for docs

…kward.parsed.yaml and paddle/fluid/ir/dialect/paddle_dialect/ir/generated/pd_ops.parsed.yaml

sunzhongkai588

LGTM for docs

zhiqiu

LGTM

…le#56565) * def dtensor_from_fn first edition * dtensor_from_fn first edition * Delete file /home/Paddle/build/test/auto_parallel/test_dist_tensor.py * polish code format * fix sample code formatting issues * change sample codes ' >>>' to '>>> ' * Add static image single measurement * modify the Indent of Sample Code * complete the sample code modification according to ZhongKai's suggestion * modify according to the review * change fluid.Variable to static.Variable * modify according to zhongkai's review * According to Yifan's suggestion, pull the latest code to resolve conflicts * remove paddle/fluid/ir/dialect/paddle_dialect/ir/generated/pd_ops_backward.parsed.yaml and paddle/fluid/ir/dialect/paddle_dialect/ir/generated/pd_ops.parsed.yaml

yangxiaoyu14 added 4 commits August 23, 2023 02:43

def dtensor_from_fn first edition

bfca775

dtensor_from_fn first edition

d20a032

Merge branch 'develop' of https://github.com/yangxiaoyu14/Paddle into…

9070579

… deve dtensor_from_fn first edition

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

03f62dc

… add_dtensor_from_fn_api

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

95015fa

… add_dtensor_from_fn_api delete /Paddle/build/test/auto_parallel/test_dist_tensor.py

paddle-bot bot added the contributor External developers label Aug 23, 2023

yangxiaoyu14 added 2 commits August 24, 2023 02:28

Delete file /home/Paddle/build/test/auto_parallel/test_dist_tensor.py

3902f7d

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

672ba4d

… add_dtensor_from_fn_api delete build/test/auto_parallel/test_dist_tensor.py

chenwhql reviewed Aug 24, 2023

View reviewed changes

luotao1 removed the contributor External developers label Aug 24, 2023

yangxiaoyu14 added 2 commits August 25, 2023 08:22

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

2d3fc3a

… add_dtensor_from_fn_api resolve issues raised by code review

polish code format

50fd5a9

sunzhongkai588 reviewed Aug 29, 2023

View reviewed changes

yangxiaoyu14 added 3 commits August 29, 2023 08:20

fix sample code formatting issues

3953a5e

change sample codes ' >>>' to '>>> '

1d510ea

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

11e76c7

… add_dtensor_from_fn_api change sample codes " >>>" to ">>> "

chenwhql reviewed Aug 30, 2023

View reviewed changes

yangxiaoyu14 added 3 commits August 30, 2023 05:16

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

221c03b

… add_dtensor_from_fn_api Get the latest code

Add static image single measurement

47564d2

modify the Indent of Sample Code

dd6c04f

sunzhongkai588 reviewed Sep 1, 2023

View reviewed changes

complete the sample code modification according to ZhongKai's suggestion

ff9928d

chenwhql reviewed Sep 1, 2023

View reviewed changes

yangxiaoyu14 added 2 commits September 4, 2023 11:32

modify according to the review

a9dd254

change fluid.Variable to static.Variable

832d36b

chenwhql previously approved these changes Sep 6, 2023

View reviewed changes

zhiqiu reviewed Sep 7, 2023

View reviewed changes

sunzhongkai588 reviewed Sep 7, 2023

View reviewed changes

modify according to zhongkai's review

bf0598b

yangxiaoyu14 dismissed chenwhql’s stale review via bf0598b September 7, 2023 07:03

jeff41404 previously approved these changes Sep 7, 2023

View reviewed changes

yangxiaoyu14 added 2 commits September 11, 2023 03:01

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

71ea731

… add_dtensor_from_fn_api Pull the latest code to resolve conflicts 20230911

According to Yifan's suggestion, pull the latest code to resolve conf…

d545aed

…licts

yangxiaoyu14 dismissed jeff41404’s stale review via d545aed September 11, 2023 03:10

sunzhongkai588 previously approved these changes Sep 11, 2023

View reviewed changes

remove paddle/fluid/ir/dialect/paddle_dialect/ir/generated/pd_ops_bac…

f064d24

…kward.parsed.yaml and paddle/fluid/ir/dialect/paddle_dialect/ir/generated/pd_ops.parsed.yaml

yangxiaoyu14 dismissed sunzhongkai588’s stale review via f064d24 September 11, 2023 08:41

tianshuo78520a approved these changes Sep 12, 2023

View reviewed changes

sunzhongkai588 approved these changes Sep 12, 2023

View reviewed changes

jeff41404 approved these changes Sep 12, 2023

View reviewed changes

chenwhql approved these changes Sep 12, 2023

View reviewed changes

zhiqiu approved these changes Sep 12, 2023

View reviewed changes

chenwhql merged commit 85be34f into PaddlePaddle:develop Sep 12, 2023

yangxiaoyu14 mentioned this pull request Sep 13, 2023

add_dtensor_from_fn_cn_doc PaddlePaddle/docs#6130

Merged

		@@ -53,6 +53,56 @@ def test_dist_tensor_creation(self):
		self.assertEqual(dist_tensor_with_tensor.dist_attr, dist_attr)


		class TestDistributedTensor(unittest.TestCase):

[AutoParallel] Add paddle.distributed.dtensor_from_fn api #56565

[AutoParallel] Add paddle.distributed.dtensor_from_fn api #56565

Conversation

yangxiaoyu14 commented Aug 23, 2023

PR types

PR changes

Description

CLAassistant commented Aug 23, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chenwhql left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jeff41404 left a comment

Choose a reason for hiding this comment

sunzhongkai588 left a comment

Choose a reason for hiding this comment

sunzhongkai588 left a comment

Choose a reason for hiding this comment

zhiqiu left a comment

Choose a reason for hiding this comment

CLAassistant commented Aug 23, 2023 •

edited

Loading