[Add] Paddle 代码 CI 中引入 xdoctest 检查 #55295

megemini · 2023-07-10T12:23:13Z

PR types

New features

PR changes

Others

Description

中国软件开源创新大赛：飞桨框架任务挑战赛
赛题五：将 xdoctest 引入到飞桨框架工作流中

RFC [Add]将 xdoctest 引入到飞桨框架工作流中v1
ISSUE 赛题五：将 xdoctest 引入到飞桨框架工作流中 Tracking Issue

第一阶段
任务：Paddle 代码 CI 中引入 xdoctest 检查

sampcd_processor_readme.md : Xdoctester 的详细设计文档
sampcd_processor_utils.py : 代码检查的相关工具
sampcd_processor_xdoctest.py : Xdoctester 的相关实现
sampcd_processor.py : 原代码检查工具
test_sampcd_processor.py : 原代码检查工具单元测试
test_sampcd_processor_xdoctest.py : Xdoctester 单元测试
requirements.txt : 添加了 xdoctest
beta.py : 修改了其中的示例代码样式，用于检查新工具 Paddle CI 的正确性。

建议先 review sampcd_processor_readme.md : Xdoctester 的详细设计文档，看看还有哪些需要修改的设计～

另外，xdoctest 的 requirement 看看放在 python 目录下是否合适？

@SigureMo @luotao1 @jzhang533 @sunzhongkai588

请评审，谢谢！：）

… add_xdoctest

paddle-bot · 2023-07-10T12:23:18Z

你的PR提交成功，感谢你对开源项目的贡献!
请关注后续CI自动化测试结果，详情请参考Paddle-CI手册。
Your PR has been submitted. Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

SigureMo · 2023-07-10T15:27:17Z

python/paddle/distribution/beta.py

+            >>> beta = paddle.distribution.Beta(alpha=0.5, beta=0.5)
+            >>> print(beta.mean)
+            Tensor(shape=[1], dtype=float32, place=Place(cpu), stop_gradient=True,
+            [0.50000000])


这里是特意用的错误例子还是？0D Tensor 应该就是 shape=[] 的，2.5 应该就已经支持 0D Tensor 了，你那里用的是 2.4 嘛？

忘了更新 Paddle 了 ...........................................

SigureMo · 2023-07-10T15:35:48Z

python/requirements.txt

@@ -7,3 +7,4 @@ decorator
 astor
 paddle_bfloat==0.1.7
 opt_einsum==3.3.0
+xdoctest


xdoctest 的 requirement 看看放在 python 目录下是否合适？

不合适，requirements.txt 会在 setup.py 里用到，作为 paddle 的「依赖项」，用户在安装时候会自动安装

相比于「依赖项」，xdoctest 更类似于「开发依赖项」，不过我们目前是没有一个统一的地方来添加这个的（大多数社区规范是写一个 requirements-dev.txt，但我们没有），我们只有 python/paddle/unittest_py/requirements.txt（即「测试依赖」），但放在里面也是明显不合适的

目前可以添加到流水线中，或许以后可以直接放在 Docker 里

刚和 @luotao1 讨论了一下，这个还是放在 python/paddle/unittest_py/requirements.txt 吧，PR-CI-Static-Check 是会自动安装这个的

OK～只是 Paddle build 的时候可能也会调用这个代码检查，感觉放哪儿都不太合适～

刚和 @luotao1 讨论了一下，这个还是放在 python/paddle/unittest_py/requirements.txt 吧，PR-CI-Static-Check 是会自动安装这个的

是 Paddle/python/unittest_py/requirements.txt 吧？

😂 嗯 python/unittest_py/requirements.txt

https://github.com/PaddlePaddle/Paddle/blob/develop/python/unittest_py/requirements.txt

… add_xdoctest

SigureMo · 2023-07-16T06:50:40Z

辛苦，我看这个 PR 最近还在修改一些文档进行测试，是否已经准备好 review 了呢？准备好的时候可以 @ 一下我

另外我们也测试一下这边和 docs 那边能否正常 copy 这边修改过的源码吧，这边在 commit 里加上 test=docs_preview 就可以在 PR-CI-Paddle-Doc-Preview 流水线看到英文文档了

中文文档那边需要新建一个 PR，然后在 PR 描述里添加 PADDLEPADDLE_PR=55295 即可

megemini · 2023-07-16T07:08:28Z

一直在等 review ......... 一直没消息，所以这几天就改了几个 codeblock，暂时也没发现什么问题～

中文文档那边需要新建一个 PR

这个是指？

SigureMo · 2023-07-16T07:09:31Z

这个是指？

docs repo

megemini · 2023-07-16T07:10:59Z

这个是指？

docs repo

关于什么的 PR？

SigureMo · 2023-07-16T07:11:58Z

docs 端才能触发中文文档 preview 呀，随便创建即可

megemini · 2023-07-16T07:25:55Z

docs 端才能触发中文文档 preview 呀，随便创建即可

@SigureMo 建好了～

PaddlePaddle/docs#6031

SigureMo · 2023-07-16T07:49:03Z

tools/sampcd_processor_readme.md

+
+## Paddle docs 需要注意
+
+目前 Paddle docs 对于 `>>> ` 代码的处理是，strip 掉此提示符，然后交给原有代码检查工具进行检测。这种方法在大部分情况下没什么问题，但是，如果代码中有 `requires` 项，则可能检查失败。所以，后续需要修改 Paddle docs 的检查逻辑，建议对于 `>>> ` 直接跳过，与当前 Paddle 的 `sampcd_processor.py` 一致。最后收尾的时候，移除掉 Paddle docs 的代码检查。


最后收尾的时候，移除掉 Paddle docs 的代码检查。

COPY-FROM 任务结束的时候应该就可以移除掉了

SigureMo · 2023-07-16T07:50:45Z

tools/sampcd_processor_readme.md

整体上看起来没有问题，但是是否能补充一些原来的文档链接呢？比如 directive 原文档是如何描述的呢？

SigureMo · 2023-07-16T07:50:50Z

tools/sampcd_processor_readme.md

整体上看起来没有问题，但是是否能补充一些原来的文档链接呢？比如 directive 原文档是如何描述的呢？

SigureMo · 2023-07-16T07:54:37Z

tools/sampcd_processor_xdoctest.py

@@ -0,0 +1,269 @@
+# Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.


SigureMo · 2023-07-16T07:55:36Z

tools/sampcd_processor_xdoctest.py

+
+        self.directive_pattern = re.compile(
+            r"""
+                                            (?<=(\#\s{1}))  # positive lookbehind, directive begins


这里前面加这么多空格是有必要的吗？

megemini added 2 commits July 10, 2023 19:44

[Add]Add Xdoctester

688a0ef

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

9a8ebdf

… add_xdoctest

paddle-bot bot added contributor External developers status: proposed labels Jul 10, 2023

megemini mentioned this pull request Jul 10, 2023

赛题五：将 xdoctest 引入到飞桨框架工作流中 Tracking Issue #54705

Closed

13 tasks

SigureMo reviewed Jul 10, 2023

View reviewed changes

luotao1 self-assigned this Jul 11, 2023

megemini added 5 commits July 11, 2023 13:37

[Fix]fix beta docstring

7a943f1

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

a25b27a

… add_xdoctest

[Doctest]change dirichlet docstring

eaa65d4

[Doctest]change gumbel docstring

f6fd132

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

f394a4d

… add_xdoctest

luotao1 assigned SigureMo Jul 13, 2023

megemini added 7 commits July 14, 2023 12:51

[Doctest]change bernoulli docstring

0629f84

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

961b9a3

… add_xdoctest

[Doctest]change categorical docstring

9507ece

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

e5242af

… add_xdoctest

[Doctest]change ops.py docstring

a22ea53

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

2d2c6d3

… add_xdoctest

[Doctest]change conv docstring

99c1c32

[Doctest]change distance docstring, test=docs_preview

bbd3e01

megemini mentioned this pull request Jul 16, 2023

[Change]just for test PaddlePaddle/docs#6031

Closed

SigureMo reviewed Jul 16, 2023

View reviewed changes

This was referenced Sep 9, 2023

[xdoctest][task 274] reformat example code with google style in python/paddle/static/nn/sequence_lod.py #57146

Merged

[xdoctest][task 337] reformat example code with google style in python/paddle/base/framework.py #57151

Merged

This was referenced Sep 11, 2023

[xdoctest][task 237] reformat example code with google style in python/paddle/distributed/utils/moe_utils.py #57202

Merged

[xdoctest][task 236] reformat example code with google style in python/paddle/distributed/utils/launch_utils.py #57203

Merged

yuchen202 mentioned this pull request Sep 18, 2023

[xdoctest] reformat example code with google style in No.356 #57437

Merged

PommesPeter mentioned this pull request Sep 18, 2023

[xdoctest] reformat example code with google style No.354 #57440

Merged

enkilee mentioned this pull request Sep 18, 2023

[xdoctest] reformat example code with google style in No.92 #57460

Merged

yuchen202 mentioned this pull request Sep 19, 2023

[xdoctest] reformat example code with google style in No.138 and 140 #57479

Closed

This was referenced Sep 21, 2023

[xdoctest][task 232-235] reformat example code with google style in python/paddle/distributed/* #57591

Merged

[xdoctest][task 219-223] reformat example code with google style in python/paddle/distributed/fleet/* #57601

Merged

megemini mentioned this pull request Sep 24, 2023

[xdoctest][task cleanup 1-8] reformat example code with google style #57670

Merged

yuchen202 mentioned this pull request Sep 26, 2023

[xdoctest] reformat example code with google style in No.366 #57786

Closed

jinyouzhi mentioned this pull request Sep 27, 2023

[xdoctest][task 226-230] reformat example code with google style in python/paddle/distributed/fleet/* #57787

Merged

This was referenced Sep 27, 2023

[xdoctest] reformat example code with google style in No.371~374 #57825

Merged

[xdoctest][task cleanup 376~381] reformat example code with google style #57853

Closed

ooooo-create mentioned this pull request Oct 17, 2023

[xdoctest] reformat example code with google style in No. 261-263 #58168

Merged

DrRyanHuang mentioned this pull request Oct 18, 2023

[xdoctest][task 355] reformat example code with google style in pir.cc #58177

Merged

This was referenced Oct 18, 2023

[xdoctest][task 365-370] reformat example code with google style in paddle/incubate/nn/functional/ ,paddle/incubate/optimizer/ #58178

Merged

[xdoctest][task 138 and 140] reformat example code with google style in audio/* #58189

Merged

yuchen202 mentioned this pull request Oct 18, 2023

[xdoctest] reformat example code with google style in No.365~370 #58213

Closed

This was referenced Nov 2, 2023

[xdoctest] reformat example code with google style in No.365~370 #58610

Closed

[xdoctest] reformat example code with google style in remaining files #58617

Merged

megemini mentioned this pull request Dec 9, 2023

WAVE SUMMIT+2023下半年飞桨开源之星评选-信息征集 PaddlePaddle/community#765

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Add] Paddle 代码 CI 中引入 xdoctest 检查 #55295

[Add] Paddle 代码 CI 中引入 xdoctest 检查 #55295

megemini commented Jul 10, 2023 •

edited

Loading

paddle-bot bot commented Jul 10, 2023

SigureMo Jul 10, 2023

megemini Jul 11, 2023

SigureMo Jul 10, 2023

SigureMo Jul 11, 2023

megemini Jul 11, 2023

megemini Jul 11, 2023

SigureMo Jul 11, 2023 •

edited

Loading

SigureMo commented Jul 16, 2023

megemini commented Jul 16, 2023

SigureMo commented Jul 16, 2023

megemini commented Jul 16, 2023

SigureMo commented Jul 16, 2023 •

edited

Loading

megemini commented Jul 16, 2023

SigureMo Jul 16, 2023

SigureMo Jul 16, 2023

SigureMo Jul 16, 2023

SigureMo Jul 16, 2023

SigureMo Jul 16, 2023


		## Paddle docs 需要注意

		目前 Paddle docs 对于 `>>> ` 代码的处理是，strip 掉此提示符，然后交给原有代码检查工具进行检测。这种方法在大部分情况下没什么问题，但是，如果代码中有 `requires` 项，则可能检查失败。所以，后续需要修改 Paddle docs 的检查逻辑，建议对于 `>>> ` 直接跳过，与当前 Paddle 的 `sampcd_processor.py` 一致。最后收尾的时候，移除掉 Paddle docs 的代码检查。

		@@ -0,0 +1,269 @@
		# Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved.

[Add] Paddle 代码 CI 中引入 xdoctest 检查 #55295

[Add] Paddle 代码 CI 中引入 xdoctest 检查 #55295

Conversation

megemini commented Jul 10, 2023 • edited Loading

PR types

PR changes

Description

paddle-bot bot commented Jul 10, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SigureMo Jul 11, 2023 • edited Loading

Choose a reason for hiding this comment

SigureMo commented Jul 16, 2023

megemini commented Jul 16, 2023

SigureMo commented Jul 16, 2023

megemini commented Jul 16, 2023

SigureMo commented Jul 16, 2023 • edited Loading

megemini commented Jul 16, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

megemini commented Jul 10, 2023 •

edited

Loading

SigureMo Jul 11, 2023 •

edited

Loading

SigureMo commented Jul 16, 2023 •

edited

Loading