Skip to content

add --exit flag to dev/devicelab/bin/test_runner.dart #134165

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Sep 7, 2023
Merged

add --exit flag to dev/devicelab/bin/test_runner.dart #134165

merged 2 commits into from
Sep 7, 2023

Conversation

@andrewkolos
Copy link
Contributor

@andrewkolos andrewkolos commented Sep 6, 2023 •
edited
Loading

Resolves #134070

Adds a flag to the test_runner.dart test script that will cause the test runner to exit upon first failure (or, said another way, exit without retrying).

This is in parity with the --exit flag of dev/devicelab/bin/run.dart.

Pre-launch Checklist

  • I read the Contributor Guide and followed the process outlined there for submitting PRs.
  • I read the Tree Hygiene wiki page, which explains my responsibilities.
  • I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
  • I signed the CLA.
  • I listed at least one issue that this PR fixes in the description above.
  • I updated/added relevant documentation (doc comments with ///).
  • I added new tests to check the change I am making, or this PR is test-exempt.
  • All existing and new tests are passing.

@flutter-dashboard
Copy link

flutter-dashboard bot commented Sep 6, 2023

It looks like this pull request may not have tests. Please make sure to add tests before merging. If you need an exemption to this rule, contact Hixie or stuartmorgan on the #hackers channel in Chat (don't just cc them here, they won't see it! Use Discord!).

If you are not sure if you need tests, consider this rule of thumb: the purpose of a test is to make sure someone doesn't accidentally revert the fix. Ask yourself, is there anything in your PR that you feel it is important we not accidentally revert back to how it was before your fix?

Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing.

christopherfujino
christopherfujino reviewed Sep 7, 2023
View reviewed changes
dev/devicelab/lib/command/test.dart
argParser.addFlag(
'exit',
help: 'Exit on the first test failure. Currently flakes are intentionally (though '
'incorrectly) not considered to be failures.',
Copy link
Contributor

@christopherfujino christopherfujino Sep 7, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why incorrectly?

Copy link
Contributor Author

@andrewkolos andrewkolos Sep 7, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is copy-pasted from the --exit flag in run.dart. This verbiage was introduced in a commit from @Hixie. I interpret this as an opinion that tasks should be considered failing if they fail on their first run, regardless of whether or not they pass after retrying.

Copy link
Contributor

@Hixie Hixie Sep 7, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

flakes are by definition failing tests (that sometimes don't report their failure).

Copy link
Contributor

@christopherfujino christopherfujino Sep 7, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ahh, i see

christopherfujino
christopherfujino reviewed Sep 7, 2023
View reviewed changes
@Hixie
Copy link
Contributor

Hixie commented Sep 7, 2023

test-exempt: test tooling

christopherfujino
christopherfujino approved these changes Sep 7, 2023
View reviewed changes
Copy link
Contributor

@christopherfujino christopherfujino left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. BTW, about the print, what I do is I add an invalid todo comment (like print('debug'); // TODO), which I know Linux analyze will fail on (it enforces the // TODO(ldap) format), so I don't accidentally merge it. The only downside to this approach is it's annoying in editor if your editor highlights lint errors.

@andrewkolos andrewkolos added the autosubmit Merge PR when tree becomes green via auto submit App label Sep 7, 2023
@auto-submit auto-submit bot merged commit aea4552 into flutter:master Sep 7, 2023
@engine-flutter-autoroll engine-flutter-autoroll mentioned this pull request Sep 7, 2023
Merged
engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Sep 7, 2023
@engine-flutter-autoroll
aea4552 add --exit flag to dev/devicelab/bin/test_runner.dart (flutte…
d06a650 
…r/flutter#134165)
auto-submit bot pushed a commit to flutter/packages that referenced this pull request Sep 7, 2023
@engine-flutter-autoroll
Manual roll Flutter from 685ce14 to aea4552 (64 revisions) (#4870)
913c946 
Manual roll Flutter from 685ce14 to aea4552 (64 revisions)

Manual roll requested by dit@google.com

flutter/flutter@685ce14...aea4552

2023-09-07 andrewrkolos@gmail.com add --exit flag to dev/devicelab/bin/test_runner.dart (flutter/flutter#134165)
2023-09-07 andrewrkolos@gmail.com fix `--exit` flag in dev/devicelab/bin/run.dart (flutter/flutter#134162)
2023-09-07 engine-flutter-autoroll@skia.org Roll Packages from e7d812c to 22d4754 (9 revisions) (flutter/flutter#134232)
2023-09-07 engine-flutter-autoroll@skia.org Roll Flutter Engine from 71bea01d3abe to f0b718e28779 (2 revisions) (flutter/flutter#134231)
2023-09-07 polinach@google.com DropdownRoutePage should dispose the created ScrollController. (flutter/flutter#133941)
2023-09-07 sokolovskyi.konstantin@gmail.com Cover some test/widgets tests with leak tracking (flutter/flutter#133803)
2023-09-07 polinach@google.com SearchDelegate should dispose resources. (flutter/flutter#133948)
2023-09-07 matheus@btor.com.br Fixed [NavigationRailDestination]'s label opacity while disabled not being coherent with the icon (flutter/flutter#132345)
2023-09-07 engine-flutter-autoroll@skia.org Roll Flutter Engine from 558136a1ccbf to 71bea01d3abe (2 revisions) (flutter/flutter#134216)
2023-09-07 engine-flutter-autoroll@skia.org Roll Flutter Engine from 5a45ecd24aa3 to 558136a1ccbf (1 revision) (flutter/flutter#134206)
2023-09-07 engine-flutter-autoroll@skia.org Roll Flutter Engine from d864ae68db3c to 5a45ecd24aa3 (1 revision) (flutter/flutter#134201)
2023-09-07 tessertaha@gmail.com Fix `TabBar` doesn't use `labelStyle` & `unselectedLabelStyle` color (flutter/flutter#133989)
2023-09-07 tessertaha@gmail.com Fix `DataTable`'s `headingTextStyle` & `dataTextStyle` are not merged with default text style (flutter/flutter#134138)
2023-09-07 engine-flutter-autoroll@skia.org Roll Flutter Engine from 187c5b3c5f71 to d864ae68db3c (2 revisions) (flutter/flutter#134199)
2023-09-07 tessertaha@gmail.com Reland "Fix `Chip.shape`'s side is not used when provided in Material 3" (flutter/flutter#133856)
2023-09-07 engine-flutter-autoroll@skia.org Roll Flutter Engine from 75437a3bd002 to 187c5b3c5f71 (1 revision) (flutter/flutter#134193)
2023-09-07 engine-flutter-autoroll@skia.org Manual roll Flutter Engine from 2c69d05dfafb to 75437a3bd002 (15 revisions) (flutter/flutter#134188)
2023-09-07 zanderso@users.noreply.github.com Revert "Roll Flutter Engine from 2c69d05dfafb to fa14d337449b (6 revisions)" (flutter/flutter#134183)
2023-09-06 49699333+dependabot[bot]@users.noreply.github.com Bump actions/upload-artifact from 3.1.2 to 3.1.3 (flutter/flutter#134173)
2023-09-06 engine-flutter-autoroll@skia.org Roll Flutter Engine from 2c69d05dfafb to fa14d337449b (6 revisions) (flutter/flutter#134169)
2023-09-06 polinach@google.com DraggableScrollableActuator should dispose notifier. (flutter/flutter#133917)
2023-09-06 engine-flutter-autoroll@skia.org Roll Flutter Engine from b04c2a378302 to 2c69d05dfafb (3 revisions) (flutter/flutter#134164)
2023-09-06 polinach@google.com Clean the fixed TODOs. (flutter/flutter#133859)
2023-09-06 engine-flutter-autoroll@skia.org Roll Flutter Engine from 839051596b1d to b04c2a378302 (7 revisions) (flutter/flutter#134158)
2023-09-06 737941+loic-sharma@users.noreply.github.com [Windows Arm64] Also use Windows 11 for Devicelab tests (flutter/flutter#134082)
2023-09-06 70351342+burakJs@users.noreply.github.com Fix `subtitleTextStyle.color` isn't applied to the `ListTile.subtitle` in Material 2 (flutter/flutter#133422)
2023-09-06 pateltirth454@gmail.com Add `CheckedPopupMenuItem.onTap` callback (flutter/flutter#134000)
2023-09-06 polinach@google.com MinimumTextContrastGuideline should dispose image. (flutter/flutter#133861)
2023-09-06 christopherfujino@gmail.com [flutter_tools] Fix "FormatException: Invalid date format" during version freshness check (flutter/flutter#134088)
2023-09-06 polinach@google.com Fix not disposed items in Cupertino app and route. (flutter/flutter#134085)
2023-09-06 engine-flutter-autoroll@skia.org Roll Flutter Engine from a5e7fa6bf81a to 839051596b1d (2 revisions) (flutter/flutter#134140)
2023-09-06 polinach@google.com _DropdownMenuState should dispose TextEditingController. (flutter/flutter#133914)
2023-09-06 engine-flutter-autoroll@skia.org Roll Flutter Engine from 5253a33096d1 to a5e7fa6bf81a (1 revision) (flutter/flutter#134137)
2023-09-06 engine-flutter-autoroll@skia.org Roll Flutter Engine from c7fd088291e2 to 5253a33096d1 (1 revision) (flutter/flutter#134135)
2023-09-06 engine-flutter-autoroll@skia.org Roll Flutter Engine from 3d9989f1e155 to c7fd088291e2 (1 revision) (flutter/flutter#134132)
2023-09-06 engine-flutter-autoroll@skia.org Roll Flutter Engine from bace539bb654 to 3d9989f1e155 (3 revisions) (flutter/flutter#134128)
2023-09-06 engine-flutter-autoroll@skia.org Roll Flutter Engine from 9344685efbc3 to bace539bb654 (1 revision) (flutter/flutter#134104)
2023-09-06 engine-flutter-autoroll@skia.org Roll Flutter Engine from 0c8c1647dcd0 to 9344685efbc3 (1 revision) (flutter/flutter#134103)
2023-09-06 engine-flutter-autoroll@skia.org Roll Flutter Engine from 0c663258fd09 to 0c8c1647dcd0 (1 revision) (flutter/flutter#134100)
2023-09-06 engine-flutter-autoroll@skia.org Roll Flutter Engine from 8bacc3b38707 to 0c663258fd09 (3 revisions) (flutter/flutter#134096)
2023-09-06 engine-flutter-autoroll@skia.org Roll Flutter Engine from 590349006d23 to 8bacc3b38707 (5 revisions) (flutter/flutter#134089)
2023-09-06 engine-flutter-autoroll@skia.org Roll Flutter Engine from 5b2cc9d9b8fe to 590349006d23 (2 revisions) (flutter/flutter#134081)
2023-09-05 engine-flutter-autoroll@skia.org Roll Flutter Engine from 98b036ae708e to 5b2cc9d9b8fe (2 revisions) (flutter/flutter#134080)
2023-09-05 engine-flutter-autoroll@skia.org Roll Flutter Engine from f4975e04f35e to 98b036ae708e (3 revisions) (flutter/flutter#134077)
...
engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Nov 15, 2023
@engine-flutter-autoroll
aea4552 add --exit flag to dev/devicelab/bin/test_runner.dart (flutte…
549b631 
…r/flutter#134165)
@andrewkolos andrewkolos deleted the add-exit-flag-to-test_runner branch April 29, 2024 18:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Hixie Hixie Hixie left review comments

+1 more reviewer

@christopherfujino christopherfujino christopherfujino approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

autosubmit Merge PR when tree becomes green via auto submit App

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Consider adding a --no-retry flag to the devicelab test runner command

3 participants

@andrewkolos @Hixie @christopherfujino