Add python YAML runner script to runs test over chip-tool and the placeholder apps #25155

vivien-apple · 2023-02-17T14:51:36Z

Problem

While matter_yamltests has been designed to runs over multiple adapters, it currently only runs the chip-repl adapter.

This PR adds both chip-tool and placeholder adapters as well as a runner for both.

With the chip-tool adapter the full yaml tests suites can be runned with the following command:

# Mac
$ ./scripts/tests/run_test_suite.py --runner chip_tool_python --target-skip-glob '{Test_TC_DGTHREAD_2_1,Test_TC_DGTHREAD_2_2,Test_TC_DGTHREAD_2_3,Test_TC_DGTHREAD_2_4}' --log-level info run --iterations 1 --test-timeout-seconds 120

# Linux
$ ./scripts/tests/run_test_suite.py --runner chip_tool_python  --log-level info run --iterations 1 --test-timeout-seconds 120

The target-skip-glob targets on Mac are unrelated with the chip-tooladapter, this is what is runned in CI for the current chip-tool due to the lack of thread supports.

For running a particular yaml test for chip-tool, the command is:

$ ./scripts/tests/yaml/runner.py Test_TC_OO_1_1 chiptool

For running a particular yaml test for placeholder app1, the command is:

$ ./scripts/tests/yaml/runner.py Test_TC_BINFO_2_3_Simulated app1

For parsing the whole YAML test suite running on the CI and validating at parse time that there is no errors at parse time, the command is:

$ ./scripts/tests/yaml/runner.py all parse

For parsing a particular YAML test, the command is:

$ ./scripts/tests/yaml/runner.py Test_TC_OO_1_1 parse

For parsing the whole YAML test suite of manual test and validating at parse time that there is no errors at parse time, the command is:

$ ./scripts/tests/yaml/runner.py --configuration_name manualTests all parse

For doing a dry-run of a given test, for example if you want to visually see which steps will be runned given your PICS config:

$ ./scripts/tests/yaml/runner.py Test_TC_OO_1_1 dry-run

The PR also contains a drop-in replacement for chip-tool as I would like to make it as easy as possible for our Test Harness friends to move along by adopting the new runner - which would let me remove the previous runner and starts adding new features without having to develop them for the codegen version too.

The drop-in replacement can be used with the same set of parameters than chip-tool, for example:

$ ./scripts/tests/yaml/chiptool.py onoff toggle 0x12344321 1 --commissioner-nodeid 5

The difference is that when the tests command is used is will be re-routed to the new python version instead of the codegen version.

While this PR contains a chip-tool adapter that can runs all tests, and all tests are runned in .github/workflows/tests.yaml it seems to me that we would need to do something since those tasks are going to be particularly long during the transition.

vivien-apple · 2023-02-17T14:57:02Z

As a note for reviewers, I am sorry for the size of the PR.
I have tried with all the previous patches to matter_yamltests to strip it down as much as I can but sadly none of what is in this PR makes sense without the other parts. So I have kept piling things up for a while :(

github-actions · 2023-02-22T19:32:33Z

PR #25155: Size comparison from c6cadc8 to 3f658a8

Decreases (1 build for cc32xx)

platform	target	config	section	`c6cadc8`	`3f658a8`	change	% change
cc32xx	lock	CC3235SF_LAUNCHXL	.debug_info	20254045	20254044	-1	-0.0

Full report (1 build for cc32xx)

platform	target	config	section	`c6cadc8`	`3f658a8`	change	% change
cc32xx	lock	CC3235SF_LAUNCHXL		0	0	0	0.0
			(read only)	642545	642545	0	0.0
			(read/write)	203664	203664	0	0.0
			.ARM.attributes	44	44	0	0.0
			.ARM.exidx	8	8	0	0.0
			.bss	197064	197064	0	0.0
			.comment	194	194	0	0.0
			.data	1480	1480	0	0.0
			.debug_abbrev	930168	930168	0	0.0
			.debug_aranges	87272	87272	0	0.0
			.debug_frame	299752	299752	0	0.0
			.debug_info	20254045	20254044	-1	-0.0
			.debug_line	2655138	2655138	0	0.0
			.debug_loc	2795342	2795342	0	0.0
			.debug_ranges	281592	281592	0	0.0
			.debug_str	3017877	3017877	0	0.0
			.ramVecs	780	780	0	0.0
			.resetVecs	64	64	0	0.0
			.rodata	105817	105817	0	0.0
			.shstrtab	232	232	0	0.0
			.stab	204	204	0	0.0
			.stabstr	441	441	0	0.0
			.stack	2048	2048	0	0.0
			.strtab	377630	377630	0	0.0
			.symtab	256192	256192	0	0.0
			.text	534604	534604	0	0.0

.github/workflows/tests.yaml

examples/chip-tool/commands/common/CHIPCommand.h

examples/chip-tool/commands/discover/DiscoverCommissionablesCommand.cpp

scripts/py_matter_yamltests/matter_yamltests/parser_builder.py

scripts/py_matter_yamltests/matter_yamltests/runner.py

scripts/tests/yaml/runner.py

github-actions · 2023-02-23T19:33:15Z

PR #25155: Size comparison from 84c77b6 to 3b1db94

Full report (3 builds for cc32xx, qpg)

platform	target	config	section	`84c77b6`	`3b1db94`
cc32xx	lock	CC3235SF_LAUNCHXL		0	0
			(read only)	642681	642681
			(read/write)	203672	203672
			.ARM.attributes	44	44
			.ARM.exidx	8	8
			.bss	197072	197072
			.comment	194	194
			.data	1480	1480
			.debug_abbrev	930133	930133
			.debug_aranges	87272	87272
			.debug_frame	299776	299776
			.debug_info	20253400	20253400
			.debug_line	`2655882`	`2655882`
			.debug_loc	2797015	2797015
			.debug_ranges	281760	281760
			.debug_str	`3018765`	`3018765`
			.ramVecs	780	780
			.resetVecs	64	64
			.rodata	105769	105769
			.shstrtab	232	232
			.stab	204	204
			.stabstr	441	441
			.stack	2048	2048
			.strtab	377825	377825
			.symtab	256256	256256
			.text	534788	534788
qpg	lighting-app	qpg6105+debug	(read/write)	`1151340`	`1151340`
			.bss	99804	99804
			.data	852	852
			.text	598436	598436
	lock-app	qpg6105+debug	(read/write)	1118540	1118540
			.bss	96292	96292
			.data	864	864
			.text	565640	565640

vivien-apple · 2023-02-24T11:50:36Z

Ah the CI linter is complaining, it seems like a992882 has landed this morning and is finding things...

vivien-apple · 2023-02-24T12:26:46Z

Ah the CI linter is complaining, it seems like a992882 has landed this morning and is finding things...

The linter should be happier now.

github-actions · 2023-02-24T12:42:42Z

PR #25155: Size comparison from 61cf6f2 to 8a00d47

Increases (1 build for cc32xx)

platform	target	config	section	`61cf6f2`	`8a00d47`	change	% change
cc32xx	lock	CC3235SF_LAUNCHXL	.debug_info	20257009	20257010	1	0.0

Full report (1 build for cc32xx)

platform	target	config	section	`61cf6f2`	`8a00d47`	change
cc32xx	lock	CC3235SF_LAUNCHXL		0	0	0
			(read only)	642681	642681	0
			(read/write)	203672	203672	0
			.ARM.attributes	44	44	0
			.ARM.exidx	8	8	0
			.bss	197072	197072	0
			.comment	194	194	0
			.data	1480	1480	0
			.debug_abbrev	930133	930133	0
			.debug_aranges	87304	87304	0
			.debug_frame	299872	299872	0
			.debug_info	20257009	20257010	1
			.debug_line	2656183	2656183	0
			.debug_loc	`2797535`	`2797535`	0
			.debug_ranges	281792	281792	0
			.debug_str	3023459	3023459	0
			.ramVecs	780	780	0
			.resetVecs	64	64	0
			.rodata	105769	105769	0
			.shstrtab	232	232	0
			.stab	204	204	0
			.stabstr	441	441	0
			.stack	2048	2048	0
			.strtab	377825	377825	0
			.symtab	256256	256256	0
			.text	534788	534788	0

examples/chip-tool/commands/discover/DiscoverCommissionablesCommand.cpp

examples/chip-tool/py_matter_chip_tool_adapter/matter_chip_tool_adapter/decoder.py

…ests

… matter_yamltests

…ceholder apps

…e when cleaning up parameters

… as ErrorStr use a single static buffer to compute the error string and in interactive mode the stack continue to do some work and it may race

github-actions · 2023-02-24T16:41:51Z

PR #25155: Size comparison from ad5dc95 to a4a0933

Increases (1 build for cc32xx)

platform	target	config	section	`ad5dc95`	`a4a0933`	change	% change
cc32xx	lock	CC3235SF_LAUNCHXL	.debug_info	20256906	20256907	1	0.0

Full report (1 build for cc32xx)

platform	target	config	section	`ad5dc95`	`a4a0933`	change
cc32xx	lock	CC3235SF_LAUNCHXL		0	0	0
			(read only)	642681	642681	0
			(read/write)	203672	203672	0
			.ARM.attributes	44	44	0
			.ARM.exidx	8	8	0
			.bss	197072	197072	0
			.comment	194	194	0
			.data	1480	1480	0
			.debug_abbrev	930077	930077	0
			.debug_aranges	87280	87280	0
			.debug_frame	299816	299816	0
			.debug_info	20256906	20256907	1
			.debug_line	2656086	2656086	0
			.debug_loc	`2797535`	`2797535`	0
			.debug_ranges	281768	281768	0
			.debug_str	`3023369`	`3023369`	0
			.ramVecs	780	780	0
			.resetVecs	64	64	0
			.rodata	105769	105769	0
			.shstrtab	232	232	0
			.stab	204	204	0
			.stabstr	441	441	0
			.stack	2048	2048	0
			.strtab	377825	377825	0
			.symtab	256256	256256	0
			.text	534788	534788	0

…ceholder apps (project-chip#25155) * [chip-tool] Add chip-tool adapter for running tests with matter_yamltests * [examples/placeholder] Add placeholder adapter for running tests with matter_yamltests * Add python YAML runner script to runs test over chip-tool and the placeholder apps * Add chip-tool over python runs to the test suite * Use a scoped lock to make Tsan happy * [chip-tool] Update DiscoverCommissionablesCommand to avoid a data race when cleaning up parameters * [chip-tool] Do not call ErrorStr after RunCommand in interactive mode as ErrorStr use a single static buffer to compute the error string and in interactive mode the stack continue to do some work and it may race

vivien-apple self-assigned this Feb 17, 2023

vivien-apple force-pushed the YamlRunner_With_ChipTool_And_Placeholder_Adapters branch from 8a96856 to 684c45c Compare February 17, 2023 14:54

vivien-apple force-pushed the YamlRunner_With_ChipTool_And_Placeholder_Adapters branch from 684c45c to b9d44ff Compare February 17, 2023 14:57

bzbarsky-apple reviewed Feb 23, 2023

View reviewed changes

vivien-apple force-pushed the YamlRunner_With_ChipTool_And_Placeholder_Adapters branch from fa0e7fe to 3b1db94 Compare February 23, 2023 19:00

vivien-apple force-pushed the YamlRunner_With_ChipTool_And_Placeholder_Adapters branch from 3b1db94 to 8e91231 Compare February 24, 2023 11:33

vivien-apple mentioned this pull request Feb 24, 2023

Should ErrorStr be calling assertChipStackLockedByCurrentThread? #25306

Open

vivien-apple force-pushed the YamlRunner_With_ChipTool_And_Placeholder_Adapters branch from 8e91231 to 8a00d47 Compare February 24, 2023 12:26

vivien-apple requested a review from bzbarsky-apple February 24, 2023 12:27

bzbarsky-apple approved these changes Feb 24, 2023

View reviewed changes

pullapprove bot added review - approved and removed review - pending labels Feb 24, 2023

vivien-apple added 7 commits February 24, 2023 17:31

[chip-tool] Add chip-tool adapter for running tests with matter_yamlt…

24a268a

…ests

[examples/placeholder] Add placeholder adapter for running tests with…

73018b2

… matter_yamltests

Add python YAML runner script to runs test over chip-tool and the pla…

f24c588

…ceholder apps

Add chip-tool over python runs to the test suite

6d8c145

Use a scoped lock to make Tsan happy

37b52ec

[chip-tool] Update DiscoverCommissionablesCommand to avoid a data rac…

98ff3a5

…e when cleaning up parameters

[chip-tool] Do not call ErrorStr after RunCommand in interactive mode…

a4a0933

… as ErrorStr use a single static buffer to compute the error string and in interactive mode the stack continue to do some work and it may race

vivien-apple force-pushed the YamlRunner_With_ChipTool_And_Placeholder_Adapters branch from 8a00d47 to a4a0933 Compare February 24, 2023 16:31

vivien-apple enabled auto-merge (squash) February 24, 2023 16:47

vivien-apple merged commit 31da720 into project-chip:master Feb 24, 2023

tehampson mentioned this pull request Feb 27, 2023

[Feature] Python yamltest parsing that runs using chip-repl #24600

Open

21 tasks

tehampson mentioned this pull request Mar 30, 2023

Adding chip-repl into runner script #25900

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add python YAML runner script to runs test over chip-tool and the placeholder apps #25155

Add python YAML runner script to runs test over chip-tool and the placeholder apps #25155

vivien-apple commented Feb 17, 2023 •

edited

Loading

vivien-apple commented Feb 17, 2023

github-actions bot commented Feb 22, 2023

github-actions bot commented Feb 23, 2023

vivien-apple commented Feb 24, 2023

vivien-apple commented Feb 24, 2023

github-actions bot commented Feb 24, 2023

github-actions bot commented Feb 24, 2023

Add python YAML runner script to runs test over chip-tool and the placeholder apps #25155

Add python YAML runner script to runs test over chip-tool and the placeholder apps #25155

Conversation

vivien-apple commented Feb 17, 2023 • edited Loading

Problem

vivien-apple commented Feb 17, 2023

github-actions bot commented Feb 22, 2023

github-actions bot commented Feb 23, 2023

vivien-apple commented Feb 24, 2023

vivien-apple commented Feb 24, 2023

github-actions bot commented Feb 24, 2023

github-actions bot commented Feb 24, 2023

vivien-apple commented Feb 17, 2023 •

edited

Loading