Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

be more loosy-goosy with the test thresholds for less flaky tests #188

Merged
merged 1 commit into from
Nov 20, 2020

Conversation

amueller
Copy link
Contributor

cc @bpkroth @byte-sculptor current tests rely too much on randomness, we can
a) change the thresholds (this PR)
b) run the optimizer for more iterations (how many? takes longer)
c) fix random seeds

I think this is the easiest fix to keep it somewhat meaningful and fast and not have master failing a lot.

@bpkroth
Copy link
Contributor

bpkroth commented Nov 19, 2020

cc @bpkroth @byte-sculptor current tests rely too much on randomness, we can
a) change the thresholds (this PR)
b) run the optimizer for more iterations (how many? takes longer)
c) fix random seeds

I think this is the easiest fix to keep it somewhat meaningful and fast and not have master failing a lot.

Seems fine for now. I think this is not the only place for this unfortunately.

One other thing we discussed in chat was to keep the randomness to get better coverage, but to print the seed used out prior to execution so that we could at least hack together a reproduction locally for debugging if need be.

@bpkroth bpkroth self-requested a review November 19, 2020 23:44
@amueller amueller merged commit 2353fcf into microsoft:main Nov 20, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants