Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

动转静单测机制推全任务列表 #58356

Closed
gouzil opened this issue Oct 24, 2023 · 0 comments
Closed

动转静单测机制推全任务列表 #58356

gouzil opened this issue Oct 24, 2023 · 0 comments
Assignees
@SigureMo @gouzil
Labels
PFCC Paddle Framework Contributor Club，https://github.com/PaddlePaddle/community/tree/master/pfcc status/close 已关闭

Comments

@gouzil
Copy link
Member

gouzil commented Oct 24, 2023 •
edited by SigureMo
Loading

Note

SOT 开发小组对组合方式进行了修改，请查看: #58630 、#58633

Motivation

单测在项目开发中是很重要的一部分，我们动转静相关的单测主要放置在 test/dygraph_to_static，也一直保障着动转静功能的正确性。

而随着近期 SOT 动转静新模式和 PIR 底层 IR 的开发，两者都使用了动转静单测作为功能正确性的验证方式，为了能够同时跑多种模式，SOT 和 PIR 开发小组在 test/dygraph_to_static/dygraph_to_static_util.py 分别添加了相关的装饰器，用于选择具体跑在什么模式下。

但 SOT 和 PIR 的单测扩增机制非常不同，这将会影响之后单测的扩展（比如之后即将开展的 PIR 动转静（理想态）单测推全验证），为了能够方便、统一地对单测进行管理，#57824 中在 test/dygraph_to_static/dygraph_to_static_utils_new.py 添加了一套统一的动转静单测机制，方便灵活地启用各种单测 case。但 #57824 的主要目标只是将 SOT 合入 Paddle，为了确保动转静单测机制的一致性，我们需要尽快将新机制推全～

Design

新的机制利用了 Metaclass，将会在生成新的 class 的时候统一生成新的单测 case，比如对于如下单测代码：

class MyTest(Dy2StTestBase):
    @set_to_static_mode(
        ToStaticMode.LEGACY_AST | ToStaticMode.SOT
    )
    @set_ir_mode(IrMode.LEGACY_PROGRAM | IrMode.PIR)
    def test_case1(self):
        ...

在装饰器的执行阶段，会在函数 MyTest.test_case1 上 patch 两个属性，分别是 test_case1.to_static_mode = ToStaticMode.LEGACY_AST | ToStaticMode.SOT 和 test_case1.ir_mode = IrMode.LEGACY_PROGRAM | IrMode.PIR。而从 Metaclass Dy2StTestMeta 生成 MyTest 的时候，我们会根据 patch 的属性来生成新的单测 case，这里对于 test_case1，根据其 to_static_mode 和 ir_mode，我们共有 2x2 = 4 种组合，因此我们会生成如下四个单测 case（伪码描述，实际实现会略有不同）：

class MyTest(unittest.TestCase):
    def test_case1__legacy_ast_legacy_program(self):
        fn = MyTest.test_case1
        fn = to_legacy_program_test(fn)
        fn = to_legacy_ast_test(fn)
        fn(self)

    def test_case1__legacy_ast_pir(self):
        fn = MyTest.test_case1
        fn = to_legacy_pir(fn)
        fn = to_legacy_ast_test(fn)
        fn(self)

    def test_case1__sot_legacy_program(self):
        fn = MyTest.test_case1
        fn = to_legacy_program_test(fn)
        fn = to_sot_test(fn)
        fn(self)

    def test_case1__sot_pir(self):
        fn = MyTest.test_case1
        fn = to_legacy_pir(fn)
        fn = to_sot_test(fn)
        fn(self)

Important

  • 新的机制一律使用 Dy2StTestBase 作为基类，如果不设置基类，根据新机制的原理，仅仅会在函数上 patch 一个属性，而不会跑两次，也就是没有任何效果，注意不仅仅是原来使用 dy2static_unittest 装饰的单测 case，只要使用了 dygraph_to_static_util 下装饰器的 case 都需要修改
  • 新的机制只能装饰 test 开头的 test case，非 test case 使用相关装饰器是没有效果的（相关装饰器一律以 test_ 作为前缀）
  • 我们保留了 compare_legacy_with_pir 这一个特殊的装饰器，因为它大多数情况是需要装饰在非 test case 上面的，也就是 test case 所调用的内部函数，根据我们新机制的原理，我们没有办法利用新机制实现这一点，因此保留了该装饰器，该装饰器仍然是旧的一次跑两种 case 的机制

Tasks

  • 推全动转静单测新机制，使新机制覆盖全部动转静单测
  • 确保新机制的正确性，测试生成的 case 数量符合预期，迭代完善单测新机制
  • 移除单测旧机制代码

Details

序号 文件 pr
1 test/dygraph_to_static/test_assert.py #58316
2 test/dygraph_to_static/test_ast_util.py #58316
3 test/dygraph_to_static/test_backward_without_params.py #58316
4 test/dygraph_to_static/test_basic_api_transformation.py #58389
5 test/dygraph_to_static/test_bert.py #58389
6 test/dygraph_to_static/test_bmn.py #58389
7 test/dygraph_to_static/test_break_continue.py #58389
8 test/dygraph_to_static/test_build_strategy.py #58389
9 test/dygraph_to_static/test_cache_program.py
10 test/dygraph_to_static/test_cast.py #58316
11 test/dygraph_to_static/test_cinn_prim_gelu.py #58389
12 test/dygraph_to_static/test_cinn_prim_layer_norm.py #58389
13 test/dygraph_to_static/test_cinn_prim_mean.py #58389
14 test/dygraph_to_static/test_cinn_prim.py #58389
15 test/dygraph_to_static/test_cinn.py #58389
16 test/dygraph_to_static/test_closure_analysis.py #58316
17 test/dygraph_to_static/test_container.py #58509
18 test/dygraph_to_static/test_convert_call_generator.py #58389
19 test/dygraph_to_static/test_convert_call.py #58389
20 test/dygraph_to_static/test_convert_operators.py #58389
21 test/dygraph_to_static/test_cpu_cuda_to_tensor.py #58389
22 test/dygraph_to_static/test_cycle_gan.py #58389
23 test/dygraph_to_static/test_declarative.py #58316
24 test/dygraph_to_static/test_decorator_transform.py #58316
25 test/dygraph_to_static/test_deepcopy.py #58316
26 test/dygraph_to_static/test_dict.py #58389
27 test/dygraph_to_static/test_drop_path.py #58389
28 test/dygraph_to_static/test_duplicate_output.py #58389
29 test/dygraph_to_static/test_error.py
30 test/dygraph_to_static/test_eval_frame.py
31 test/dygraph_to_static/test_fallback.py #58458
32 test/dygraph_to_static/test_fetch_feed.py #58458
33 test/dygraph_to_static/test_for_enumerate.py #58458
34 test/dygraph_to_static/test_full_name_usage.py #58458
35 test/dygraph_to_static/test_function_spec.py
36 test/dygraph_to_static/test_grad.py #58458
37 test/dygraph_to_static/test_gradient_aggregation.py #58458
38 test/dygraph_to_static/test_gradname_parse.py #58458
39 test/dygraph_to_static/test_grid_generator.py #58389
40 test/dygraph_to_static/test_ifelse.py #58458
41 test/dygraph_to_static/test_ignore_module.py
42 test/dygraph_to_static/test_inplace_assign.py #58464
43 test/dygraph_to_static/test_isinstance.py #58458
44 test/dygraph_to_static/test_jit_property_save.py #58458
45 test/dygraph_to_static/test_jit_setitem.py #58458
46 test/dygraph_to_static/test_lac.py #58509
47 test/dygraph_to_static/test_lambda.py #58458
48 test/dygraph_to_static/test_layer_hook.py #58458
49 test/dygraph_to_static/test_len.py #58458
50 test/dygraph_to_static/test_list.py #58458
51 test/dygraph_to_static/test_load_transformer.py #58316
52 test/dygraph_to_static/test_logging_utils.py
53 test/dygraph_to_static/test_logical.py #58464
54 test/dygraph_to_static/test_loop.py #58464
55 test/dygraph_to_static/test_lstm.py #58464
56 test/dygraph_to_static/test_mnist_amp.py(test_mnist.py) #58464
57 test/dygraph_to_static/test_mnist_pure_fp16.py(test_mnist.py) #58464
58 test/dygraph_to_static/test_mnist.py #58464
59 test/dygraph_to_static/test_mobile_net.py #58464
60 test/dygraph_to_static/test_multi_forward.py #58464
61 test/dygraph_to_static/test_new_ir_selectedrows.py #58464
62 test/dygraph_to_static/test_op_attr.py #58464
63 test/dygraph_to_static/test_origin_info.py #58464
64 test/dygraph_to_static/test_param_guard.py #58464
65 test/dygraph_to_static/test_params_no_grad.py #58464
66 test/dygraph_to_static/test_partial_program_hook.py #58465
67 test/dygraph_to_static/test_partial_program.py #58316
68 test/dygraph_to_static/test_place.py #58465
69 test/dygraph_to_static/test_print.py #58465
70 test/dygraph_to_static/test_program_translator.py #58465
71 test/dygraph_to_static/test_ptb_lm_v2.py #58465
72 test/dygraph_to_static/test_ptb_lm.py #58465
73 test/dygraph_to_static/test_pylayer.py #58465
74 test/dygraph_to_static/test_reinforcement_learning.py #58465
75 test/dygraph_to_static/test_resnet_amp.py #58465
76 test/dygraph_to_static/test_resnet_pure_fp16.py #58465
77 test/dygraph_to_static/test_resnet_v2.py #58465
78 test/dygraph_to_static/test_resnet.py #58465
79 test/dygraph_to_static/test_return.py #58465
80 test/dygraph_to_static/test_rollback.py #58465
81 test/dygraph_to_static/test_save_inference_model.py #58465
82 test/dygraph_to_static/test_save_load.py #58490
83 test/dygraph_to_static/test_se_resnet.py #58490
84 test/dygraph_to_static/test_sentiment.py #58509
85 test/dygraph_to_static/test_seq2seq.py #58490
86 test/dygraph_to_static/test_set_dynamic_shape.py #58316
87 test/dygraph_to_static/test_setter_helper.py
88 test/dygraph_to_static/test_simnet_v2.py #58490
89 test/dygraph_to_static/test_simnet.py #58490
90 test/dygraph_to_static/test_slice.py #58490
91 test/dygraph_to_static/test_spec_names.py #58316
92 test/dygraph_to_static/test_static_analysis.py
93 test/dygraph_to_static/test_tensor_hook.py #58490
94 test/dygraph_to_static/test_tensor_memcpy_on_cpu.py #58490
95 test/dygraph_to_static/test_tensor_memcpy_on_gpu.py #58490
96 test/dygraph_to_static/test_tensor_methods.py #58490
97 test/dygraph_to_static/test_tensor_shape.py #58316
98 test/dygraph_to_static/test_to_tensor.py #58490
99 test/dygraph_to_static/test_train_step.py #58490
100 test/dygraph_to_static/test_transformer.py #58490
101 test/dygraph_to_static/test_tsm.py #58490
102 test/dygraph_to_static/test_typehint.py #58490
103 test/dygraph_to_static/test_typing.py #58499
104 test/dygraph_to_static/test_unuseful_inputs.py #58499
105 test/dygraph_to_static/test_utils.py #58499
106 test/dygraph_to_static/test_variable_trans_func.py #58499
107 test/dygraph_to_static/test_warning.py #58499
108 test/dygraph_to_static/test_word2vec.py #58499
109 test/dygraph_to_static/test_write_python_container.py #58509
110 test/dygraph_to_static/test_yolov3.py #58499
111 test/legacy_test/test_cond.py #58509
112 test/legacy_test/test_while_loop_op.py #58509
113 test/legacy_test/test_while_op.py #58509
114 test/dygraph_to_static/test_no_gradient.py #58509
@gouzil gouzil added status/new-issue 新建 type/others 其他问题 labels Oct 24, 2023
@paddle-bot paddle-bot bot added the PFCC Paddle Framework Contributor Club，https://github.com/PaddlePaddle/community/tree/master/pfcc label Oct 24, 2023
@gouzil gouzil mentioned this issue Oct 25, 2023
Merged
@Ligoml Ligoml removed status/new-issue 新建 type/others 其他问题 labels Oct 27, 2023
This was referenced Oct 27, 2023
Merged
Merged
Merged
Merged
Merged
Merged
@SigureMo SigureMo closed this as completed Nov 1, 2023
@paddle-bot paddle-bot bot added the status/close 已关闭 label Nov 2, 2023
@SigureMo SigureMo mentioned this issue Nov 6, 2023
Closed
@ooooo-create ooooo-create mentioned this issue Nov 24, 2023
Closed
@gouzil gouzil mentioned this issue Dec 1, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@SigureMo SigureMo

@gouzil gouzil

Labels
PFCC Paddle Framework Contributor Club，https://github.com/PaddlePaddle/community/tree/master/pfcc status/close 已关闭
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@SigureMo @Ligoml @gouzil and others