Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

skip flaky tests in official builds #8497

Merged
merged 2 commits into from
Feb 7, 2020
Merged

Conversation

brettfo
Copy link
Member

@brettfo brettfo commented Feb 6, 2020

Some MailBoxProcessor tests are flaky so we're skipping them on official builds, but keeping them alive on PRs. This way we'll definitely know when they break, but the inherent flakiness won't prevent official builds.

@brettfo brettfo requested a review from KevinRansom February 6, 2020 21:03
@brettfo
Copy link
Member Author

brettfo commented Feb 6, 2020

Currently marked [WIP] until I can confirm that the tests run in this PR.

@baronfel
Copy link
Member

baronfel commented Feb 6, 2020

dotnet test (and nunit) support filtering test groups by category. Would it make sense to use that pre-existing mechanism here to exclude specific categories of flaky tests?

@brettfo
Copy link
Member Author

brettfo commented Feb 6, 2020

@baronfel Good call! Since I'm trying to selectively disable a small set of tests I've had to invert the example in the link, but let's hope that the != operator is supported for TestCategory.

@brettfo
Copy link
Member Author

brettfo commented Feb 7, 2020

I manually ran this so that it wasn't considered a PR build and verified that the expected tests were skipped, but everything else was run. Ready to merge once green.

@brettfo brettfo changed the title [WIP] skip flaky tests in official builds skip flaky tests in official builds Feb 7, 2020
@KevinRansom KevinRansom merged commit c648e2a into dotnet:master Feb 7, 2020
@brettfo brettfo deleted the flaky-tests branch February 7, 2020 18:43
nosami pushed a commit to xamarin/visualfsharp that referenced this pull request Feb 23, 2021
* skip flaky tests in official builds

* also filter tests on linux
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants