Stream reader for chain state #1519
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…el-core into feature/stream-genesis-state
…el-core into feature/stream-genesis-state
MujkicA
commented
Feb 8, 2024
xgreenx
reviewed
Feb 14, 2024
xgreenx
added a commit
that referenced
this pull request
Feb 17, 2024
) Almost closes #1583 The change moves the `OwnedCoins` and `OwnedMessageIds` tables and their implementation to the GraphQL module. Now, the off-chain worker is responsible for tracking owners of coins and messages. For that purpose, `Executor` generates execution events to track which messages/coins are consumed and created during execution. The worker uses these events and updates corresponding tables. It is not required to produce these events from the executor's perspective since anyone can calculate them based on the imported block and events from the relayer(knowing the execution rules of the state transition). However, I decided to embed that logic into the executor to simplify support for it. In the future, if we decide to change how coins and messages are created/spent, we don't need to update a bunch of places. Events from the `ExecutionResult` are insufficient to complete the change since we also need to support coins and messages from the genesis block. I added a new function into the `genesis` module to perform an update of the `OwnedCoins` and `OwnedMessageIds` tables from the `StateConfig`(in the future, it will be done by an off-chain regenesis process). The initial idea was to emit events, but after reviewing the #1519 I realized that we can have so many events that it will be hard to fit them into the memory. So, I decided to implement a separate function that later can work with batches and be parallelizable. --------- Co-authored-by: Mitchell Turner <james.mitchell.turner@gmail.com> Co-authored-by: Brandon Kite <brandonkite92@gmail.com>
xgreenx
reviewed
Feb 18, 2024
xgreenx
approved these changes
Feb 21, 2024
crypto523
pushed a commit
to crypto523/fuel-core
that referenced
this pull request
Oct 7, 2024
Part of the FuelLabs/fuel-core#1583. The change moves the genesis block execution and commitment from the `FuelService::new` to the `FuelService::Task::into_task`. It allows us to notify other services about the genesis block because all services are already subscribed to the block importer(it is what we need for FuelLabs/fuel-core#1583 to process new messages inside the off-chain worker). Plus, it adds support for the `async` syntax(it will be used by the parallel regenesis process from FuelLabs/fuel-core#1519). Moving genesis block initialization from the constructor to the starting level breaks p2p because `P2PService` requires knowing the `Genesis` type to create `FuelP2PService`(It is used to filter connections with peers). Because of that, I moved the creation of the `FuelP2PService` to `UninitializedTask::into_task` where the genesis block is already available.
crypto523
pushed a commit
to crypto523/fuel-core
that referenced
this pull request
Oct 7, 2024
…635) Almost closes FuelLabs/fuel-core#1583 The change moves the `OwnedCoins` and `OwnedMessageIds` tables and their implementation to the GraphQL module. Now, the off-chain worker is responsible for tracking owners of coins and messages. For that purpose, `Executor` generates execution events to track which messages/coins are consumed and created during execution. The worker uses these events and updates corresponding tables. It is not required to produce these events from the executor's perspective since anyone can calculate them based on the imported block and events from the relayer(knowing the execution rules of the state transition). However, I decided to embed that logic into the executor to simplify support for it. In the future, if we decide to change how coins and messages are created/spent, we don't need to update a bunch of places. Events from the `ExecutionResult` are insufficient to complete the change since we also need to support coins and messages from the genesis block. I added a new function into the `genesis` module to perform an update of the `OwnedCoins` and `OwnedMessageIds` tables from the `StateConfig`(in the future, it will be done by an off-chain regenesis process). The initial idea was to emit events, but after reviewing the FuelLabs/fuel-core#1519 I realized that we can have so many events that it will be hard to fit them into the memory. So, I decided to implement a separate function that later can work with batches and be parallelizable. --------- Co-authored-by: Mitchell Turner <james.mitchell.turner@gmail.com> Co-authored-by: Brandon Kite <brandonkite92@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Relevant issue: #1209
Relevant previous PRs: #1474
This PR builds upon the previous work by introducing a resumable, parallelized (re)genesis process.
Starting with the run command, users provide a snapshot metadata file containing paths to the chain config file and files containing chain state items (such as coins, messages, contracts, contract states, and balances), which are loaded via streaming.
Each item group in the genesis process is handled by a separate worker, allowing for parallel loading. Workers stream file contents in batches, with the batch size adaptable for future performance optimization.
A database transaction is committed every time an item group is succesfully loaded. Resumability is achieved by recording the last loaded group index within the same db tx. If loading is aborted, the remaining workers are shutdown. Upon restart, workers resume from the last processed group.
Database tables used for the genesis process are cleared once it's finalized.
Contract States and Balances
The use of uniform-sized batches may result in batches containing items from multiple contracts. We expect that best performance is achieved by selecting a batch size that typically encompasses an entire contract's state or balance, allowing for immediate initialization of relevant Merkle trees through
nodes_from_set.Note:
genesis.rs, so it might be a good entry point for the review. Analysing how workers process the batches will guide you through the majority of the changes. Coin and message configs are simply loaded into the db, contract configs are loaded but the contracts tree root cannot be calculated until the contract states and balances workers are finished.When inserting contract states and balances, we also have to init/update the corresponding trees.