Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(indexes): efficient rocksdb mempool-tips initialization #938

Merged
merged 1 commit into from
Jul 26, 2024

Conversation

jansegre
Copy link
Member

@jansegre jansegre commented Feb 5, 2024

Motivation

When we made sync-v2 always available, even when not enabled, it implied having to initialize the mempool-tips index, but its RocksDB implementation had been disabled for having a very slow initialization but using a memory index has the downside of always initializing the index on every start.

The mempool is defined as the set of transactions that don't have a first_block and are not voided, the mempool-tips index only stores only the mempool transactions that are "tips" (in the sense that all the other transactions can be reached by traversing to the left), the practical definition is the set of all transactions that don't have a first_block, don't have a non-voided child and don't have a non-voided spent_output (that is, they aren't literal "tips" because they can have children or spent outputs, as long as those are voided).

Before this PR, the initialization was implemented as simply processing all transactions in topological order the same way they would be if a node received them from the network, with the disadvantage that the it would almost always cache miss due to how the update has to load transactions "to the right" of the iteration loop, making it very slow.

This PR simplifies the loading by looking at the first_block, children, spent_outputs and voided_by metadata fields to determine if a transaction is in the mempool and only adding it to the index if it is, instead of adding and removing transactions.

Acceptance Criteria

  • Optimize MemoryTipsIndex.init_loop_step by restricting the scope and making it only add transactions without a first_block, children or spent outputs.
  • Use the RocksDB implementation when RocksDB indexes are being used.

Checklist

  • If you are requesting a merge into master, confirm this code is production-ready and can be included in future releases as soon as it gets merged

@jansegre jansegre requested a review from msbrogli as a code owner February 5, 2024 17:35
@jansegre jansegre self-assigned this Feb 5, 2024
@jansegre jansegre requested a review from glevco February 5, 2024 17:35
@jansegre jansegre force-pushed the fix/rocksdb-mempool-tips-index branch from 3f52b0a to 02fa789 Compare February 5, 2024 17:35
@jansegre jansegre force-pushed the fix/rocksdb-mempool-tips-index branch from 02fa789 to bbcb9bf Compare February 22, 2024 16:34
Copy link

codecov bot commented Feb 22, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 85.04%. Comparing base (3ffc83d) to head (bd8c2aa).

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #938      +/-   ##
==========================================
+ Coverage   84.91%   85.04%   +0.12%     
==========================================
  Files         314      314              
  Lines       23901    23919      +18     
  Branches     3609     3614       +5     
==========================================
+ Hits        20296    20342      +46     
+ Misses       2899     2870      -29     
- Partials      706      707       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@jansegre jansegre force-pushed the fix/rocksdb-mempool-tips-index branch from bbcb9bf to 053da25 Compare March 20, 2024 22:55
@jansegre jansegre force-pushed the fix/rocksdb-mempool-tips-index branch from 053da25 to 224f7f7 Compare March 21, 2024 21:15
@jansegre jansegre requested a review from msbrogli April 19, 2024 16:28
@jansegre jansegre force-pushed the fix/rocksdb-mempool-tips-index branch from 224f7f7 to d263ea3 Compare April 19, 2024 16:29
@jansegre jansegre force-pushed the fix/rocksdb-mempool-tips-index branch from d263ea3 to 4705d5c Compare May 3, 2024 14:30
@jansegre jansegre force-pushed the fix/rocksdb-mempool-tips-index branch from 4705d5c to b1c4bec Compare June 10, 2024 13:01
@jansegre jansegre force-pushed the fix/rocksdb-mempool-tips-index branch from b1c4bec to 12304da Compare July 24, 2024 14:36
@jansegre jansegre mentioned this pull request Jul 25, 2024
2 tasks
@jansegre jansegre force-pushed the fix/rocksdb-mempool-tips-index branch from 12304da to bd8c2aa Compare July 26, 2024 14:20
@jansegre jansegre merged commit e61463e into master Jul 26, 2024
12 checks passed
@jansegre jansegre deleted the fix/rocksdb-mempool-tips-index branch July 26, 2024 16:35
@jansegre jansegre mentioned this pull request Aug 1, 2024
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

3 participants