-
Notifications
You must be signed in to change notification settings - Fork 219
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Tweak sample counts and other details in tests #2433
Open
mhauru
wants to merge
20
commits into
master
Choose a base branch
from
mhauru/reduce-test-iterations
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+288
−309
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #2433 +/- ##
==========================================
- Coverage 44.72% 44.46% -0.26%
==========================================
Files 22 22
Lines 1554 1554
==========================================
- Hits 695 691 -4
- Misses 859 863 +4 ☔ View full report in Codecov by Sentry. |
Pull Request Test Coverage Report for Build 12261740370Details
💛 - Coveralls |
mhauru
changed the title
Reduce iterations in tests
Tweak sample counts and other details in tests
Dec 13, 2024
I'm hoping CI will now pass, except for Mooncake stack overflows. See updated OP for what this PR does. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR makes a number of small changes to the tests
Random.seed!
. Every@testset
resets the seed anyway.verbose = true
to many of the top-level@testset
s, to get some timing numbers even when tests pass.This started with me thinking that since we do a lot of samples in many tests, and our test suite is quite slow, clearly we should reduce the sample counts if we can. What I learned in the process is that most of the tests that do sampling are negligble compared to the tests that use particle methods. Anything involving PG/CSMC tends to take 10-100 times longer than comparable tests with other samplers.
In the end I've tried to adjust iteration counts only when either the count truly doesn't matter (we are not checking stats of the chain), or when doing so actually has a noticable, though not necessarily huge, impact on run time. I may have slipped a few times, and reduced something that didn' really make a difference. Also, when reducing iteration counts I tried to check that they pass if I pick a few different seeds, to make sure I'm not making them very brittle to future changes.