BucketListDB in-memory Buckets #4630
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was referenced Jan 29, 2025
Closed
dmkozh
reviewed
Jan 30, 2025
dmkozh
approved these changes
Jan 31, 2025
dmkozh
approved these changes
Jan 31, 2025
github-merge-queue bot
pushed a commit
that referenced
this pull request
Feb 13, 2025
# Description Resolves #4633 This PR indexes bucket files during `VerifyBucketsWork`. Previously we would download Buckets, iterate all the buckets to check their hash, then iterate through all the buckets again to index them. Since startup is primarily disk bound, iterating through the entire BucketList twice is expensive. This change does the hash verification and indexing step in the same pass so we only have to read the BucketList once. This introduces no new DOS vectors. Indexing unverified buckets could lead to an OOM based DOS attack, where a malicious History Archive provider hosts malicious buckets that are very large. However, such OOM attacks are already possible via a zip bomb, and History Archive providers are fairly trusted, so this is not a significant concern. To mitigate this I've added an INFO level log message saying what history archive a given file is being downloaded from. In the event of a DOS attack, these logs would give us enough info to quickly assign blame to the attacker and remove them from quorum sets. On my laptop, this decreases startup time from `new-db` by about 16%. Rebased on top of #4630. # Checklist - [x] Reviewed the [contributing](https://github.com/stellar/stellar-core/blob/master/CONTRIBUTING.md#submitting-changes) document - [x] Rebased on top of master (no merge commits) - [x] Ran `clang-format` v8.0.0 (via `make format` or the Visual Studio extension) - [x] Compiles - [x] Ran all tests - [ ] If change impacts performance, include supporting evidence per the [performance document](https://github.com/stellar/stellar-core/blob/master/performance-eval/performance-eval.md)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Partially resolves #3696
This change refactors the
BucketIndexparts of BucketListDB to be more friendly towards the Hot Archive BucketList. This includes cleanups to metrics, which previously only tracked metrics for the main thread access of the BucketList. Now, both BucketList types and background threads record metrics properly.Additionally, this change removes the
IndividualIndexand instead caches small Buckets entirely in-memory so we never read from disk.RangeIndexis largely unchanged but has been renamed toDiskIndex.A follow up PR will add a random eviction cache to the
DiskIndex. I tried to break this up as much as possible, but it was easiest to do the refactor + in-memory buckets at the same time so I didn't have to refactorIndividualIndex.The
BUCKETLIST_DB_INDEX_CUTOFFconfig setting determines the maximum size at which we keep bucket in-memory. I've set this to 250 MB, which is approximately the first 4-5 levels of the BucketList. This increases total memory consumption of stellar-core from 2.2 GB to 3 GB. This seems reasonable, and we could probably go even higher, but I'm holding off for now as the random eviction cache will further increase memory requirements.Checklist
clang-formatv8.0.0 (viamake formator the Visual Studio extension)