Shorten the time it takes to process bucket pool #7356
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Create test buckets in the background instead of before the tests. This shortens time in the case of no running tests, all tests are skipped, or a test only requires a single bucket. - Do not wait for buckets to be torn down for test exit, since they will be deleted in the next test run. - Only create 3 collections when we need to, otherwise create two. - Put bucket into readyBucketPool directly instead of addToBucketReadierQueue after creation. Dropping the immediate flush saves about 6 secs per bucket for the buckets to become ready. - Running the bucket creation in parallel is faster when Flush buckets are used, but slower when GSI is used, so running parallel init is off by default.
bbrks
approved these changes
Feb 7, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I would say my favorite features are creating test buckets asynchronously and not waiting for bucket teardown in test exit. This speeds up running tests interactively a good deal, especially when running small numbers of tests. One thing that it protects against is using
-run TestFoowhereTestFooaccidentally doesn't match anything, but the bucket pool has to start up for you to know that. Additionally, this means not having to resetSG_TEST_BUCKET_POOL_SIZEwhen running tests locally, since you only have to wait for as many buckets as you need for the test to run.I can take or leave some of the other changes. If there's consensus about any of the changes, I'll dump them back into a jira ticket.
Integration Tests
GSI=true,xattrs=truehttps://jenkins.sgwdev.com/job/SyncGateway-Integration/2894/GSI=true,xattrs=truehttps://jenkins.sgwdev.com/job/SyncGateway-Integration/2947/