You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Checkpoint file is around 160GB and growing. Also, state extraction during mainnet spork should be faster. Lastly, we currently evict entire 160GB checkpoint file from Linux page cache when the file is closed.
Determine if we should split up the checkpoint file and process portions in parallel to:
Speedup checkpointing.
Speedup state extraction during mainnet spork.
Eliminate the need to evict segments of 160GB checkpoint file from Linux page cache during use.
PR #3050 (merged on Aug 23 and not deployed to August Spork) set the foundation for parallel processing of subtries by splitting up the mtrie into 16 subtries. It will reduce peak RAM use (max RSS) by dozens of GB once deployed. Parallel processing in future PR will use up of some or all of that memory reduction.
Caveats
Considerations include third parties who currently read and parse the checkpoint file with their own software.
The text was updated successfully, but these errors were encountered:
Problem
Checkpoint file is around 160GB and growing. Also, state extraction during mainnet spork should be faster. Lastly, we currently evict entire 160GB checkpoint file from Linux page cache when the file is closed.
Updates #1744
The Proposed Solution
Determine if we should split up the checkpoint file and process portions in parallel to:
PR #3050 (merged on Aug 23 and not deployed to August Spork) set the foundation for parallel processing of subtries by splitting up the mtrie into 16 subtries. It will reduce peak RAM use (max RSS) by dozens of GB once deployed. Parallel processing in future PR will use up of some or all of that memory reduction.
Caveats
Considerations include third parties who currently read and parse the checkpoint file with their own software.
The text was updated successfully, but these errors were encountered: