-
Notifications
You must be signed in to change notification settings - Fork 27
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix(indexes): efficient rocksdb mempool-tips initialization #938
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jansegre
force-pushed
the
fix/rocksdb-mempool-tips-index
branch
from
February 5, 2024 17:35
3f52b0a
to
02fa789
Compare
jansegre
force-pushed
the
fix/rocksdb-mempool-tips-index
branch
from
February 22, 2024 16:34
02fa789
to
bbcb9bf
Compare
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #938 +/- ##
==========================================
+ Coverage 84.91% 85.04% +0.12%
==========================================
Files 314 314
Lines 23901 23919 +18
Branches 3609 3614 +5
==========================================
+ Hits 20296 20342 +46
+ Misses 2899 2870 -29
- Partials 706 707 +1 ☔ View full report in Codecov by Sentry. |
msbrogli
requested changes
Feb 23, 2024
jansegre
force-pushed
the
fix/rocksdb-mempool-tips-index
branch
from
March 20, 2024 22:55
bbcb9bf
to
053da25
Compare
1 task
jansegre
force-pushed
the
fix/rocksdb-mempool-tips-index
branch
from
March 21, 2024 21:15
053da25
to
224f7f7
Compare
glevco
approved these changes
Mar 21, 2024
jansegre
force-pushed
the
fix/rocksdb-mempool-tips-index
branch
from
April 19, 2024 16:29
224f7f7
to
d263ea3
Compare
jansegre
force-pushed
the
fix/rocksdb-mempool-tips-index
branch
from
May 3, 2024 14:30
d263ea3
to
4705d5c
Compare
jansegre
force-pushed
the
fix/rocksdb-mempool-tips-index
branch
from
June 10, 2024 13:01
4705d5c
to
b1c4bec
Compare
jansegre
force-pushed
the
fix/rocksdb-mempool-tips-index
branch
from
July 24, 2024 14:36
b1c4bec
to
12304da
Compare
msbrogli
approved these changes
Jul 24, 2024
2 tasks
jansegre
force-pushed
the
fix/rocksdb-mempool-tips-index
branch
from
July 26, 2024 14:20
12304da
to
bd8c2aa
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
When we made sync-v2 always available, even when not enabled, it implied having to initialize the mempool-tips index, but its RocksDB implementation had been disabled for having a very slow initialization but using a memory index has the downside of always initializing the index on every start.
The mempool is defined as the set of transactions that don't have a
first_block
and are not voided, the mempool-tips index only stores only the mempool transactions that are "tips" (in the sense that all the other transactions can be reached by traversing to the left), the practical definition is the set of all transactions that don't have afirst_block
, don't have a non-voided child and don't have a non-voided spent_output (that is, they aren't literal "tips" because they can have children or spent outputs, as long as those are voided).Before this PR, the initialization was implemented as simply processing all transactions in topological order the same way they would be if a node received them from the network, with the disadvantage that the it would almost always cache miss due to how the update has to load transactions "to the right" of the iteration loop, making it very slow.
This PR simplifies the loading by looking at the
first_block
,children
,spent_outputs
andvoided_by
metadata fields to determine if a transaction is in the mempool and only adding it to the index if it is, instead of adding and removing transactions.Acceptance Criteria
MemoryTipsIndex.init_loop_step
by restricting the scope and making it only add transactions without a first_block, children or spent outputs.Checklist
master
, confirm this code is production-ready and can be included in future releases as soon as it gets merged