feat(init): speed up initialization

[jansegre]

Using the latest snapshot (data-20220303175324.tar.gz) as a base, initialization is about 60% faster when using all existing in-disk indexes (some indexes are still in-memory only), and about 120% faster than using all in-disk indexes but rebuilding them on every initialization (the current behavior before this PR):

• cold run (all indexes rebuilt, in-disk will be reused later):

  2022-05-11 20:55:29 [info     ] [hathor.indexes.manager] loaded                         blocks=2267901 height=2259960 total_dt=~3509s tx_count=2895834 tx_rate=825.1686592546121 txs=627933
  

• hot run (only in-memory indexes are rebuilt):

  2022-05-11 21:29:53 [info     ] [hathor.indexes.manager] loaded                         blocks=2267901 height=2259960 total_dt=~1596s tx_count=2895834 tx_rate=1814.152552360921 txs=627933
  

• hot run (all in-memory indexes are rebuilt, no in-disk index is used):

  2022-05-11 22:19:42 [info     ] [hathor.indexes.manager] loaded                         blocks=2267901 height=2259960 total_dt=~2615s tx_count=2895834 tx_rate=1107.2753905973273 txs=627933
  

But still in-disk indexes are not enabled by default, they require cli argument --x-rocksdb-indexes.

Last bits:

• hathor/indexes/address_index.py:36 tx_meta.voided_by should not be skipped, add a test for that
• hathor/indexes/deps_index.py:110 check whether the it really makes sense to filter-out voided txs
• hathor/indexes/tips_index.py:63 check whether looking at validation.is_final makes sense for all indexes
• hathor/indexes/manager.py:44 check if soft-voided txs are handled correctly
• hathor/indexes/manager.py:147 move iterator selector to the storage and only consider DFS sort when a flag is used

[codecov]

Codecov Report

Merging #392 (038731a) into dev (52ec99c) will decrease coverage by 0.01%.
The diff coverage is 85.67%.

❗ Current head 038731a differs from pull request most recent head 718bc00. Consider uploading reports for the commit 718bc00 to get more accurate results

@@            Coverage Diff             @@
##              dev     #392      +/-   ##
==========================================
- Coverage   82.65%   82.63%   -0.02%     
==========================================
  Files         174      175       +1     
  Lines       15939    16228     +289     
  Branches     2503     2550      +47     
==========================================
+ Hits        13174    13410     +236     
- Misses       2309     2342      +33     
- Partials      456      476      +20     

Impacted Files	Coverage Δ
hathor/manager.py	67.89% <44.92%> (-5.02%)	⬇️
hathor/indexes/rocksdb_mempool_tips_index.py	64.00% <60.00%> (-31.00%)	⬇️
hathor/indexes/rocksdb_deps_index.py	61.18% <66.66%> (-0.36%)	⬇️
hathor/indexes/tips_index.py	97.56% <84.61%> (-2.44%)	⬇️
hathor/indexes/manager.py	94.27% <91.95%> (+0.10%)	⬆️
hathor/transaction/storage/transaction_storage.py	94.51% <95.16%> (-0.22%)	⬇️
hathor/indexes/rocksdb_utils.py	71.62% <95.65%> (-22.83%)	⬇️
hathor/exception.py	100.00% <100.00%> (ø)
hathor/indexes/address_index.py	93.10% <100.00%> (+0.79%)	⬆️
hathor/indexes/base_index.py	100.00% <100.00%> (ø)
... and 28 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 52ec99c...718bc00. Read the comment docs.