feat(indexes): make sync-v2 indexes only load when sync-v2 is enabled

[jansegre]

This was originally included in #480 but was removed for that to be implemented and reviewed quicker.

The main advantage of this PR is a significant initialization speed up and slight memory usage reduction when not using sync-v2 (the default).

Comparison

Before:

2022-10-18 17:28:40 [info     ] [hathor.cli.run_node] hathor-core v0.51.0            genesis=3fdff62 hathor=0.51.0 my_peer_id=c68251cd14fb3d02eb743f39c5663f291f17d40f466fc0207eae28a82cd4e0ba pid=96302 platform=macOS-12.5.1-arm64-arm-64bit python=3.10.4-CPython settings=/Users/jan/Projects/hathor-nodes/data-2/conf.py
2022-10-18 17:28:40 [info     ] [hathor.cli.run_node] with storage                   path=/Users/jan/Projects/hathor-nodes/data-2 storage_class=TransactionRocksDBStorage
2022-10-18 17:28:40 [info     ] [hathor.cli.run_node] with cache                     capacity=100000 interval=5
2022-10-18 17:28:40 [info     ] [hathor.cli.run_node] with indexes                   indexes_class=RocksDBIndexesManager
2022-10-18 17:28:40 [info     ] [hathor.manager] start manager                  network=mainnet
2022-10-18 17:28:40 [info     ] [hathor.p2p.manager] update whitelist
2022-10-18 17:28:40 [info     ] [hathor.manager] initialize
2022-10-18 17:28:40 [info     ] [hathor.indexes.rocksdb_info_index] loaded info-index              block_count=2927624 first_timestamp=1578075305 latest_timestamp=1666114015 tx_count=962584
2022-10-18 17:29:10 [info     ] [hathor.indexes.memory_tips_index] loading... 33%                 dt=~30s index=tips-all iv_new=954813 progress=0.33058644710965845 rate=31827.051182835276 total=954813
2022-10-18 17:29:40 [info     ] [hathor.indexes.memory_tips_index] loading... 61%                 dt=~30s index=tips-all iv_new=818491 progress=0.613973907985486 rate=27283.02704537881 total=1773304
2022-10-18 17:30:10 [info     ] [hathor.indexes.memory_tips_index] loading... 87%                 dt=~30s index=tips-all iv_new=766721 progress=0.8794369581475223 rate=25430.293131467963 total=2540025
2022-10-18 17:30:40 [info     ] [hathor.indexes.memory_tips_index] loading... 117%                dt=~30s index=tips-all iv_new=843401 progress=1.1714490485555218 rate=28113.361081043666 total=3383426
2022-10-18 17:30:56 [info     ] [hathor.indexes.memory_tips_index] loaded                         count=3890208 index=tips-all rate=28743.2777728335 total_dt=~135s
2022-10-18 17:31:26 [info     ] [hathor.indexes.memory_tips_index] loading... 30%                 dt=~30s index=tips-blocks iv_new=901384 progress=0.3078988349576247 rate=30046.110171185443 total=901384
2022-10-18 17:31:56 [info     ] [hathor.indexes.memory_tips_index] loading... 57%                 dt=~30s index=tips-blocks iv_new=792255 progress=0.5785208911393996 rate=26408.498320992894 total=1693639
2022-10-18 17:32:26 [info     ] [hathor.indexes.memory_tips_index] loading... 82%                 dt=~30s index=tips-blocks iv_new=714725 progress=0.822659898282957 rate=23824.163447930187 total=2408364
2022-10-18 17:32:41 [info     ] [hathor.indexes.memory_tips_index] loaded                         count=2927624 index=tips-blocks rate=27714.078073914006 total_dt=~105s
2022-10-18 17:33:17 [info     ] [hathor.indexes.memory_tips_index] loading... 96%                 dt=~35s index=tips-txs iv_new=927241 progress=0.9645628295643327 rate=25975.451713487688 total=927241
2022-10-18 17:33:18 [info     ] [hathor.indexes.memory_tips_index] loaded                         count=961307 index=tips-txs rate=26133.621484008563 total_dt=~36s
2022-10-18 17:33:48 [info     ] [hathor.indexes.manager] loading...  7%                 dt=~30s height=292180 latest_ts=2020-04-14 16:58:50 progress=0.07406241087406286 total=294221 tx_new=294221 tx_rate=9807.356768049323
2022-10-18 17:34:18 [info     ] [hathor.indexes.manager] loading... 12%                 dt=~30s height=476551 latest_ts=2020-06-18 03:15:12 progress=0.1208883129449689 total=480242 tx_new=186021 tx_rate=6200.581191243711
2022-10-18 17:34:48 [info     ] [hathor.indexes.manager] loading... 17%                 dt=~30s height=680211 latest_ts=2020-08-28 06:48:58 progress=0.17372009175833816 total=690122 tx_new=209880 tx_rate=6995.99438448403
2022-10-18 17:35:18 [info     ] [hathor.indexes.manager] loading... 22%                 dt=~30s height=874264 latest_ts=2020-11-04 10:02:58 progress=0.22463197359720022 total=892375 tx_new=202253 tx_rate=6741.744431559729
2022-10-18 17:35:48 [info     ] [hathor.indexes.manager] loading... 27%                 dt=~30s height=1066305 latest_ts=2021-01-10 15:53:48 progress=0.2763284783375359 total=1097745 tx_new=205370 tx_rate=6845.6593764753625
2022-10-18 17:36:18 [info     ] [hathor.indexes.manager] loading... 32%                 dt=~30s height=1229578 latest_ts=2021-03-08 18:01:14 progress=0.32123095930155726 total=1276125 tx_new=178380 tx_rate=5945.998251581706
2022-10-18 17:36:48 [info     ] [hathor.indexes.manager] loading... 36%                 dt=~30s height=1382144 latest_ts=2021-05-01 03:33:52 progress=0.365080983303416 total=1450324 tx_new=174199 tx_rate=5806.629595430714
2022-10-18 17:37:18 [info     ] [hathor.indexes.manager] loading... 40%                 dt=~30s height=1539514 latest_ts=2021-06-25 05:01:39 progress=0.409429420312948 total=1626503 tx_new=176179 tx_rate=5872.61555154701
2022-10-18 17:37:48 [info     ] [hathor.indexes.manager] loading... 45%                 dt=~30s height=1696475 latest_ts=2021-08-19 11:26:47 progress=0.45457154227863855 total=1805835 tx_new=179332 tx_rate=5977.7191763534975
2022-10-18 17:38:18 [info     ] [hathor.indexes.manager] loading... 50%                 dt=~30s height=1813839 latest_ts=2021-09-29 12:07:40 progress=0.501787867872222 total=1993407 tx_new=187572 tx_rate=6252.38683228087
2022-10-18 17:38:48 [info     ] [hathor.indexes.manager] loading... 54%                 dt=~30s height=1925286 latest_ts=2021-11-07 08:45:40 progress=0.5486988525676703 total=2179766 tx_new=186359 tx_rate=6211.942871249009
2022-10-18 17:39:18 [info     ] [hathor.indexes.manager] loading... 59%                 dt=~30s height=2015664 latest_ts=2021-12-08 18:27:28 progress=0.5954570409521803 total=2365518 tx_new=185752 tx_rate=6191.724229963759
2022-10-18 17:39:48 [info     ] [hathor.indexes.manager] loading... 64%                 dt=~30s height=2107458 latest_ts=2022-01-09 16:26:07 progress=0.6427806008595359 total=2553516 tx_new=187998 tx_rate=6266.585208703621
2022-10-18 17:40:18 [info     ] [hathor.indexes.manager] loading... 69%                 dt=~30s height=2198160 latest_ts=2022-02-10 05:47:42 progress=0.692242553948803 total=2750009 tx_new=196493 tx_rate=6549.765573556606
2022-10-18 17:40:48 [info     ] [hathor.indexes.manager] loading... 74%                 dt=~30s height=2283794 latest_ts=2022-03-12 01:55:08 progress=0.7419169618756842 total=2947346 tx_new=197337 tx_rate=6577.875900643745
2022-10-18 17:41:18 [info     ] [hathor.indexes.manager] loading... 78%                 dt=~30s height=2389108 latest_ts=2022-04-17 20:23:51 progress=0.7899700171851798 total=3138242 tx_new=190896 tx_rate=6363.177496353897
2022-10-18 17:41:48 [info     ] [hathor.indexes.manager] loading... 83%                 dt=~30s height=2503203 latest_ts=2022-05-27 16:27:41 progress=0.8359604481588799 total=3320944 tx_new=182702 tx_rate=6090.061584723263
2022-10-18 17:42:18 [info     ] [hathor.indexes.manager] loading... 88%                 dt=~30s height=2631295 latest_ts=2022-07-11 09:49:34 progress=0.881244542314635 total=3500840 tx_new=179896 tx_rate=5996.530330996427
2022-10-18 17:42:48 [info     ] [hathor.indexes.manager] loading... 92%                 dt=~30s height=2761696 latest_ts=2022-08-25 22:47:31 progress=0.9265500329883963 total=3680821 tx_new=179981 tx_rate=5999.35069435054
2022-10-18 17:43:18 [info     ] [hathor.indexes.manager] loading... 97%                 dt=~30s height=2889756 latest_ts=2022-10-09 17:02:10 progress=0.9711489351204712 total=3857995 tx_new=177174 tx_rate=5905.779113962605
2022-10-18 17:43:42 [info     ] [hathor.indexes.manager] loaded                         blocks=2927624 height=2915791 total_dt=~624s tx_count=3890208 tx_rate=6229.770581432996 txs=962584
2022-10-18 17:43:44 [info     ] [hathor.manager] ready                          hathor_core_args=Namespace(hostname=None, auto_hostname=False, unsafe_mode=None, testnet=False, test_mode_tx_weight=False, dns=None, peer='/Users/jan/Projects/hathor-nodes/data-2/peer.json', listen=['tcp:9101'], bootstrap=None, status=9180, stratum=9181, data='/Users/jan/Projects/hathor-nodes/data-2', rocksdb_storage=False, memory_storage=False, json_storage=False, memory_indexes=False, rocksdb_cache=None, wallet=None, wallet_enable_api=False, words=None, passphrase=False, unlock_wallet=False, wallet_index=True, utxo_index=True, prometheus=False, prometheus_prefix='', cache=True, cache_size=100000, cache_interval=None, recursion_limit=10000, allow_mining_without_peers=False, x_full_verification=False, x_fast_init_beta=False, procname_prefix='', allow_non_standard_script=False, max_output_script_size=None, sentry_dsn=None, enable_debug_api=True, enable_crash_api=False, x_sync_bridge=False, x_sync_v2_only=False, x_localhost_only=False, x_rocksdb_indexes=False, x_enable_event_queue=False, x_retain_events=False, peer_id_blacklist=[]) hathor_core_version=0.51.0 network=mainnet network_full=mainnet peer_id=c68251cd14fb3d02eb743f39c5663f291f17d40f466fc0207eae28a82cd4e0ba python_implementation=namespace(name='cpython', cache_tag='cpython-310', version=sys.version_info(major=3, minor=10, micro=4, releaselevel='final', serial=0), hexversion=50988272, _multiarch='darwin') total_load_time=~903s vertex_count=3890208
2022-10-18 17:43:44 [info     ] [hathor.metrics] Transaction cache hits during initialization hits=44
2022-10-18 17:43:44 [info     ] [hathor.metrics] Transaction cache misses during initialization misses=3901971

After:

2022-10-18 17:45:08 [info     ] [hathor.cli.run_node] hathor-core v0.51.0            genesis=3fdff62 hathor=0.51.0 my_peer_id=c68251cd14fb3d02eb743f39c5663f291f17d40f466fc0207eae28a82cd4e0ba pid=96570 platform=macOS-12.5.1-arm64-arm-64bit python=3.10.4-CPython settings=/Users/jan/Projects/hathor-nodes/data-2/conf.py
2022-10-18 17:45:08 [info     ] [hathor.cli.run_node] with storage                   path=/Users/jan/Projects/hathor-nodes/data-2 storage_class=TransactionRocksDBStorage
2022-10-18 17:45:08 [info     ] [hathor.cli.run_node] with cache                     capacity=100000 interval=5
2022-10-18 17:45:08 [info     ] [hathor.cli.run_node] with indexes                   indexes_class=RocksDBIndexesManager
2022-10-18 17:45:08 [info     ] [hathor.manager] start manager                  network=mainnet
2022-10-18 17:45:08 [info     ] [hathor.p2p.manager] update whitelist
2022-10-18 17:45:08 [info     ] [hathor.manager] initialize
2022-10-18 17:45:08 [info     ] [hathor.indexes.manager] there are no indexes that need initialization
2022-10-18 17:45:08 [info     ] [hathor.indexes.rocksdb_info_index] loaded info-index              block_count=2927624 first_timestamp=1578075305 latest_timestamp=1666114015 tx_count=962584
2022-10-18 17:45:08 [info     ] [hathor.indexes.partial_rocksdb_tips_index] loading... 0%                  index=tips-all progress=0
2022-10-18 17:45:38 [info     ] [hathor.indexes.partial_rocksdb_tips_index] loading... 23%                 dt=~30s index=tips-all iv_new=921919 progress=0.2369845005716918 rate=30730.615016459415 total=921919
2022-10-18 17:46:08 [info     ] [hathor.indexes.partial_rocksdb_tips_index] loading... 45%                 dt=~30s index=tips-all iv_new=850039 progress=0.45549184002500637 rate=28334.62477636543 total=1771958
2022-10-18 17:46:38 [info     ] [hathor.indexes.partial_rocksdb_tips_index] loading... 66%                 dt=~30s index=tips-all iv_new=795604 progress=0.6600063544160106 rate=26520.113943143824 total=2567562
2022-10-18 17:47:08 [info     ] [hathor.indexes.partial_rocksdb_tips_index] loading... 87%                 dt=~30s index=tips-all iv_new=817803 progress=0.8702272474890803 rate=27260.09371733095 total=3385365
2022-10-18 17:47:24 [info     ] [hathor.indexes.partial_rocksdb_tips_index] loaded...  100%                count=3890208 index=tips-all progress=1.0 rate=28648.266035089626 total_dt=~135s
2022-10-18 17:47:24 [info     ] [hathor.indexes.partial_rocksdb_tips_index] loading... 0%                  index=tips-blocks progress=0
2022-10-18 17:47:54 [info     ] [hathor.indexes.partial_rocksdb_tips_index] loading... 29%                 dt=~30s index=tips-blocks iv_new=872592 progress=0.298054668222422 rate=29086.392371790025 total=872592
2022-10-18 17:48:24 [info     ] [hathor.indexes.partial_rocksdb_tips_index] loading... 56%                 dt=~30s index=tips-blocks iv_new=794634 progress=0.5694809169483513 rate=26487.784212050312 total=1667226
2022-10-18 17:48:54 [info     ] [hathor.indexes.partial_rocksdb_tips_index] loading... 80%                 dt=~30s index=tips-blocks iv_new=699744 progress=0.8084952165988528 rate=23324.778867969584 total=2366970
2022-10-18 17:49:11 [info     ] [hathor.indexes.partial_rocksdb_tips_index] loaded...  100%                count=2927624 index=tips-blocks progress=1.0 rate=27295.5313089163 total_dt=~107s
2022-10-18 17:49:11 [info     ] [hathor.indexes.partial_rocksdb_tips_index] loading... 0%                  index=tips-txs progress=0
2022-10-18 17:49:47 [info     ] [hathor.indexes.partial_rocksdb_tips_index] loading... 96%                 dt=~35s index=tips-txs iv_new=927280 progress=0.9633237203194734 rate=26044.72400741315 total=927280
2022-10-18 17:49:48 [info     ] [hathor.indexes.partial_rocksdb_tips_index] loaded...  99%                 count=961307 index=tips-txs progress=0.9986733625325166 rate=26301.90473690139 total_dt=~36s
2022-10-18 17:49:49 [info     ] [hathor.manager] ready                          hathor_core_args=Namespace(hostname=None, auto_hostname=False, unsafe_mode=None, testnet=False, test_mode_tx_weight=False, dns=None, peer='/Users/jan/Projects/hathor-nodes/data-2/peer.json', listen=['tcp:9101'], bootstrap=None, status=9180, stratum=9181, data='/Users/jan/Projects/hathor-nodes/data-2', rocksdb_storage=False, memory_storage=False, json_storage=False, memory_indexes=False, rocksdb_cache=None, wallet=None, wallet_enable_api=False, words=None, passphrase=False, unlock_wallet=False, wallet_index=True, utxo_index=True, prometheus=False, prometheus_prefix='', cache=True, cache_size=100000, cache_interval=None, recursion_limit=10000, allow_mining_without_peers=False, x_full_verification=False, x_fast_init_beta=False, procname_prefix='', allow_non_standard_script=False, max_output_script_size=None, sentry_dsn=None, enable_debug_api=True, enable_crash_api=False, x_sync_bridge=False, x_sync_v2_only=False, x_localhost_only=False, x_rocksdb_indexes=False, x_enable_event_queue=False, x_retain_events=False, peer_id_blacklist=[]) hathor_core_version=0.51.0 network=mainnet network_full=mainnet peer_id=c68251cd14fb3d02eb743f39c5663f291f17d40f466fc0207eae28a82cd4e0ba python_implementation=namespace(name='cpython', cache_tag='cpython-310', version=sys.version_info(major=3, minor=10, micro=4, releaselevel='final', serial=0), hexversion=50988272, _multiarch='darwin') total_load_time=~280s vertex_count=3890208
2022-10-18 17:49:49 [info     ] [hathor.metrics] Transaction cache hits during initialization hits=4
2022-10-18 17:49:49 [info     ] [hathor.metrics] Transaction cache misses during initialization misses=11803

Total load time went from ~908s to ~280s which is at least a 3x speed up. The next step in speeding up initialization will most likely be when rolling out sync-v2 by making sync-v1 indexes optional, then no loading will be needed and it will likely be in the order of <1min.

Acceptance Criteria

• when starting the node with sync-v2 disabled (the default), do not load/initialize/instantiate the following indexes: deps-index, mempool-index;
• enabling sync-v2 should correctly enable those indexes;
• both sync-v1 and sync-v2 can be enabled which would end up with the same indexes that were enabled before this change;
• mempool resource should still work, but use tx-tips index when mempool-tips is not available;

[codecov]

Codecov Report

Merging #486 (88eb0e9) into dev (60b0f33) will decrease coverage by 0.03%.
The diff coverage is 83.58%.

❗ Current head 88eb0e9 differs from pull request most recent head 7f34cce. Consider uploading reports for the commit 7f34cce to get more accurate results

@@            Coverage Diff             @@
##              dev     #486      +/-   ##
==========================================
- Coverage   83.56%   83.53%   -0.04%     
==========================================
  Files         187      187              
  Lines       16829    16918      +89     
  Branches     2590     2616      +26     
==========================================
+ Hits        14063    14132      +69     
- Misses       2301     2314      +13     
- Partials      465      472       +7     

Impacted Files	Coverage Δ
hathor/indexes/partial_rocksdb_tips_index.py	67.28% <61.11%> (-2.93%)	⬇️
hathor/transaction/resources/mempool.py	81.25% <70.00%> (-18.75%)	⬇️
hathor/indexes/manager.py	92.63% <89.13%> (-0.96%)	⬇️
hathor/transaction/storage/transaction_storage.py	90.14% <96.96%> (+0.34%)	⬆️
hathor/consensus.py	95.96% <100.00%> (ø)
hathor/indexes/base_index.py	100.00% <100.00%> (ø)
hathor/indexes/memory_info_index.py	91.66% <100.00%> (ø)
hathor/indexes/mempool_tips_index.py	92.07% <100.00%> (-1.50%)	⬇️
hathor/indexes/rocksdb_info_index.py	100.00% <100.00%> (ø)
... and 1 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.