Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added boilerplate for Bonsai storage migration #70

Merged
merged 33 commits into from
Apr 24, 2024

Conversation

Trantorian1
Copy link
Collaborator

@Trantorian1 Trantorian1 commented Apr 22, 2024

Pull Request type

  • Feature

What is the current behavior?

Currently, Deoxys relies heavily on the Substrate runtime storage. This poses issues as it was designed for sparse usage, and is not pertinent for the storage of data which is not consensus-critical. Furthermore, use of the Substrate runtime storage and the associated StorageOverrides built on top has lead to a convoluted use of storage which is hard to reason about.

Another issue worth noting is that we currently use rocksdb / substrate for sequential insertions: this can be hevilly parallelized.

What is the new behavior?

  • Added new storage primitives in crates/client/src/storage_handler for accessing storage statically and outside of the Substrate runtime.
  • Started migrating Substrate runtime storage to the Bonsai lib BonsaiStorage and RevertibleStorage via storage_handler. This inculdes the following mappings:
    • block_number -> block_hash
    • block_hash -> block_number
    • contract_address -> class_hash & nonce
    • class_hash -> contract_abi & contract_class
    • class_hash -> compiled_class_hash
  • Insertion and retrieval from any of these new storage primitives can now be done in parallel lock-free.
  • All storage apart from new blocks have been moved out of the runtime: as a result on_initialized is no longer used for storing data. Any related logic, such as serializing data to communicate it to the runtime for storage, have been removed. Storage updates are now handled as part of l2.rs.
  • Storage insertions have been heavily parallelized where possible.
  • Removed the use of StorageOverride from the code base.

Does this introduce a breaking change?

Yes. Any node already running will have to be re-run from the block 0 to update the new databases.

Other information

Note that this is PR mostly introduces the boilerplate necessary for a full migration to Bonsai lib storage. Further improvements remain to be made:

  • currently, there remains an issue where the wrong state root is being calculated. This is probably due to me inserting default Nonces in the db in some special cases.
  • significant improvements in storage access using transactional state are possible after pr #28 on the bonsai-lib

Please keep these points in mind as you review this PR.

Note that multiple changes to the bonsai-lib are also essential to this pr. These are present on my fork of the repo and are yet to be merged.

@Trantorian1 Trantorian1 added the enhancement New feature or request label Apr 22, 2024
@Trantorian1 Trantorian1 self-assigned this Apr 22, 2024
crates/client/db/src/storage_handler/contract_data.rs Outdated Show resolved Hide resolved
Ok(Some(block_number)) => block_number,
_ => get_highest_block_hash_and_number().1,
},
BlockId::Tag(_) => get_highest_block_hash_and_number().1,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tag pending not supported

@jbcaron jbcaron merged commit a7168f7 into madara-alliance:main Apr 24, 2024
6 checks passed
@Trantorian1 Trantorian1 deleted the feat/storage branch April 24, 2024 15:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants