@@ -7,29 +7,36 @@ the tests are referred to as "tasks" in the API, but since we primarily use it
77for testing, this document refers to them as "tests".
88
99Current statuses for the devicelab are available at
10- https://flutter-dashboard.appspot.com/#/build . See [ dashboard user guide] ( https://github.com/flutter/cocoon/blob/master/app_flutter/USER_GUIDE.md )
10+ < https://flutter-dashboard.appspot.com/#/build > . See [ dashboard user
11+ guide] ( https://github.com/flutter/cocoon/blob/master/app_flutter/USER_GUIDE.md )
1112for information on using the dashboards.
1213
1314## Table of Contents
15+
1416* [ How the DeviceLab runs tests] ( #how-the-devicelab-runs-tests )
1517* [ Running tests locally] ( #running-tests-locally )
1618* [ Writing tests] ( #writing-tests )
17- * [ Adding tests to continuous integration] ( #adding-tests-to-continuous-integration )
19+ * [ Adding tests to continuous
20+ integration] ( #adding-tests-to-continuous-integration )
1821* [ Adding tests to presubmit] ( #adding-tests-to-presubmit )
1922
20-
2123## How the DeviceLab runs tests
2224
23- DeviceLab tests are run against physical devices in Flutter's lab (the "DeviceLab").
25+ DeviceLab tests are run against physical devices in Flutter's lab (the
26+ "DeviceLab").
2427
25- Tasks specify the type of device they are to run on (` linux_android ` , ` mac_ios ` , ` mac_android ` , ` windows_android ` , etc).
26- When a device in the lab is free, it will pickup tasks that need to be completed.
28+ Tasks specify the type of device they are to run on (` linux_android ` , ` mac_ios ` ,
29+ ` mac_android ` , ` windows_android ` , etc). When a device in the lab is free, it
30+ will pickup tasks that need to be completed.
2731
28- 1 . If the task succeeds, the test runner reports the success and uploads its performance metrics to Flutter's infrastructure. Not
29- all tasks record performance metrics.
30- 2 . If task fails, an auto rerun happens. Whenever the last run succeeds, the task will be reported as a success. For this case,
31- a flake will be flagged and populated to the test result.
32- 3 . If the task fails in all reruns, the test runner reports the failure to Flutter's infrastructure and no performance metrics are collected
32+ 1 . If the task succeeds, the test runner reports the success and uploads its
33+ performance metrics to Flutter's infrastructure. Not all tasks record
34+ performance metrics.
35+ 2 . If task fails, an auto rerun happens. Whenever the last run succeeds, the
36+ task will be reported as a success. For this case, a flake will be flagged and
37+ populated to the test result.
38+ 3 . If the task fails in all reruns, the test runner reports the failure to
39+ Flutter's infrastructure and no performance metrics are collected
3340
3441## Running tests locally
3542
@@ -63,10 +70,11 @@ To run a test, use option `-t` (`--task`):
6370
6471Where ` NAME_OR_PATH_OF_TEST ` can be either of:
6572
66- - the _ name_ of a task, which is a file's basename in ` bin/tasks ` . Example: ` complex_layout__start_up ` .
67- - the path to a Dart _ file_ corresponding to a task, which resides in ` bin/tasks ` .
68- Tip: most shells support path auto-completion using the Tab key. Example:
69- ` bin/tasks/complex_layout__start_up.dart ` .
73+ * the _ name_ of a task, which is a file's basename in ` bin/tasks ` . Example:
74+ ` complex_layout__start_up ` .
75+ * the path to a Dart _ file_ corresponding to a task, which resides in
76+ ` bin/tasks ` . Tip: most shells support path auto-completion using the Tab key.
77+ Example: ` bin/tasks/complex_layout__start_up.dart ` .
7078
7179To run multiple tests, repeat option ` -t ` (` --task ` ) multiple times:
7280
@@ -107,19 +115,19 @@ Example:
107115
108116The ` --ab=10 ` tells the runner to run an A/B test 10 times.
109117
110- ` --local-engine=host_debug_unopt ` tells the A/B test to use the ` host_debug_unopt `
111- engine build. ` --local-engine ` is required for A/B test.
118+ ` --local-engine=host_debug_unopt ` tells the A/B test to use the
119+ ` host_debug_unopt ` engine build. ` --local-engine ` is required for A/B test.
112120
113- ` --ab-result-file=filename ` can be used to provide an alternate location to output
114- the JSON results file (defaults to ` ABresults#.json ` ). A single ` # ` character can be
115- used to indicate where to insert a serial number if a file with that name already
116- exists, otherwise, the file will be overwritten.
121+ ` --ab-result-file=filename ` can be used to provide an alternate location to
122+ output the JSON results file (defaults to ` ABresults#.json ` ). A single ` # `
123+ character can be used to indicate where to insert a serial number if a file with
124+ that name already exists, otherwise, the file will be overwritten.
117125
118126A/B can run exactly one task. Multiple tasks are not supported.
119127
120128Example output:
121129
122- ```
130+ ``` text
123131Score Average A (noise) Average B (noise) Speed-up
124132bench_card_infinite_scroll.canvaskit.drawFrameDuration.average 2900.20 (8.44%) 2426.70 (8.94%) 1.20x
125133bench_card_infinite_scroll.canvaskit.totalUiFrame.average 4964.00 (6.29%) 4098.00 (8.03%) 1.21x
@@ -142,13 +150,14 @@ Summarize tool example:
142150 ABresults.json ABresults1.json ABresults2.json ...
143151```
144152
145- ` --[no-]tsv-table ` tells the tool to print the summary in a table with tabs for easy spreadsheet
146- entry. (defaults to on)
153+ ` --[no-]tsv-table ` tells the tool to print the summary in a table with tabs for
154+ easy spreadsheet entry. (defaults to on)
147155
148- ` --[no-]raw-summary ` tells the tool to print all per-run data collected by the A/B test formatted
149- with tabs for easy spreadsheet entry. (defaults to on)
156+ ` --[no-]raw-summary ` tells the tool to print all per-run data collected by the
157+ A/B test formatted with tabs for easy spreadsheet entry. (defaults to on)
150158
151- Multiple trailing filenames can be specified and each such results file will be processed in turn.
159+ Multiple trailing filenames can be specified and each such results file will be
160+ processed in turn.
152161
153162## Reproducing broken builds locally
154163
@@ -208,7 +217,7 @@ _TASK_- the name of your test that also matches the name of the
208217
2092181 . Add target to
210219 [ .ci.yaml] ( https://github.com/flutter/flutter/blob/master/.ci.yaml )
211- - Mirror an existing one that has the recipe ` devicelab_drone `
220+ * Mirror an existing one that has the recipe ` devicelab_drone `
212221
213222If your test needs to run on multiple operating systems, create a separate
214223target for each operating system.
0 commit comments