chainHead: Add support for storage closest merkle descendant #14818 #1153

lexnv · 2023-08-25T11:04:47Z

This PR adds support for fetching the closest merkle value of some key.

Builds on top of

trie-db: Fetch the closest merkle value trie#199

Migrates paritytech/substrate#14818 to the monorepo.
Closes: paritytech/substrate#14550
Closes: #1506

// @paritytech/subxt-team

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

and TrieBackend Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

…_db_merkle Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

…_db_merkle

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

This PR updates: - trie-db from 0.27.1 to 0.28.0 - trie-bench from 0.37.0 to 0.38.0 (deb-dependency) While at it, also adapts the recorder to take into account the newly added `TrieAccess::InlineValue`. Needed by: - #1153 @paritytech/subxt-team --------- Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> Co-authored-by: Bastian Köcher <git@kchr.de>

…_db_merkle Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

This PR updates: - trie-db from 0.27.1 to 0.28.0 - trie-bench from 0.37.0 to 0.38.0 (deb-dependency) While at it, also adapts the recorder to take into account the newly added `TrieAccess::InlineValue`. Needed by: - paritytech/polkadot-sdk#1153 @paritytech/subxt-team --------- Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> Co-authored-by: Bastian Köcher <git@kchr.de>

skunert

Looks nice!

substrate/client/rpc-spec-v2/src/chain_head/tests.rs

Co-authored-by: Sebastian Kunert <skunert49@gmail.com>

tadeohepperle

I don't know too much about how substrate handles storage, but since the changes got merged into sp-trie and this PR mostly forwards to functions from sp-trie I think this should be all good. Good work!

tadeohepperle · 2023-09-15T16:04:48Z

substrate/client/db/src/lib.rs

+		key: &[u8],
+	) -> Result<Option<MerkleValue<B::Hash>>, Self::Error> {
+		self.state.closest_merkle_value(key)
+	}


Question: What does it mean here, if an Error is returned, vs. an Option is returned? Under what circumstances can errors happen?

In this case, the Option tells us if there is descendant merkle value below the provided key, since the key may not be in the database and may not have any descendants.

On the other hand, the error is a way to propagate that something went wrong during this operation. Mainly on the trie-db side:

unable to fetch bytes from the database (invalid state root / incomplete data)

corrupted data (failing to decode the node)

Here would be the main place from which those errors originate

This PR updates: - trie-db from 0.27.1 to 0.28.0 - trie-bench from 0.37.0 to 0.38.0 (deb-dependency) While at it, also adapts the recorder to take into account the newly added `TrieAccess::InlineValue`. Needed by: - paritytech#1153 @paritytech/subxt-team --------- Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> Co-authored-by: Bastian Köcher <git@kchr.de>

…aritytech#1153) This PR adds support for fetching the closest merkle value of some key. Builds on top of - paritytech/trie#199 Migrates paritytech/substrate#14818 to the monorepo. Closes: paritytech/substrate#14550 Closes: paritytech#1506 // @paritytech/subxt-team --------- Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io> Co-authored-by: Sebastian Kunert <skunert49@gmail.com>

…nt` (paritytech#1153) * Refactor logic * Thanks svyatonik help, it compile * Fix failed unit test * Remove compile warning * Rename * Return result in pay_relayers_rewards * Fix runtime compile issue * Use MessageNonce * Fix review issue * Missing u64 replacement * Revert return type changes * Fix merge issue * Remove useless clone

…nt` (#1153) * Refactor logic * Thanks svyatonik help, it compile * Fix failed unit test * Remove compile warning * Rename * Return result in pay_relayers_rewards * Fix runtime compile issue * Use MessageNonce * Fix review issue * Missing u64 replacement * Revert return type changes * Fix merge issue * Remove useless clone

lexnv added 9 commits August 25, 2023 13:30

git: Ignore target folder for main repo

d9728fb

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

Cargo: Update trie to github lexnv/expose_merkle_value

289732a

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

client: Expose merkle value on storageProvider trait

e51afac

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

primitives/backend: Expose merkle value on Backend trait

bc39b11

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

primitives/trie: Low level functions for reading the merkle value

d23ad73

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

primitives/trie: Implement closest merkle value on TrieBackendEssence..

fe12896

and TrieBackend Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

client: Implement closest merkle value up until StorageProvider

cb7fd4d

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

chainHead/storage: Query the storage for closest merkle value

766de69

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

chainHead/tests: Check merkle values propagate to the chainHead_storage

d23e1c1

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

lexnv added A0-needs_burnin Pull request needs to be tested on a live validator node before merge. DevOps is notified via matrix R0-silent Changes should not be mentioned in any release notes labels Aug 25, 2023

lexnv self-assigned this Aug 25, 2023

Merge branch 'master' into lexnv/chainhed_trie_db_merkle

4f215a9

paritytech-ci requested review from a team August 28, 2023 09:45

lexnv added 9 commits September 11, 2023 14:17

Merge remote-tracking branch 'origin/master' into lexnv/chainhed_trie…

6493a10

…_db_merkle Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

primitives: Adjust merkle value api

3607785

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

Update interface with merkle value changes

62f677a

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

chainHead/storage: Adjust the returned values

b6dd92e

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

Merge remote-tracking branch 'origin/master' into lexnv/chainhed_trie…

445f225

…_db_merkle

bench: Use trie github link

1e52670

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

chainHead/tests: Adjust testing

520c651

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

chainHead: Support merkle value types

ba184a0

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

chainHead: Fix build

3294a1d

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

This was referenced Sep 12, 2023

chore: Release trie-db 0.28.0 paritytech/trie#200

Merged

Update trie-db version to 0.28.0 #1522

Merged

Merge remote-tracking branch 'origin/master' into lexnv/chainhed_trie…

3e7a60d

…_db_merkle Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

lexnv requested review from jsdw and skunert September 13, 2023 16:11

skunert approved these changes Sep 15, 2023

View reviewed changes

substrate/client/rpc-spec-v2/src/chain_head/tests.rs Outdated Show resolved Hide resolved

This comment was marked as duplicate.

Sign in to view

Update substrate/client/rpc-spec-v2/src/chain_head/tests.rs

cd0c0c1

Co-authored-by: Sebastian Kunert <skunert49@gmail.com>

tadeohepperle approved these changes Sep 15, 2023

View reviewed changes

lexnv merged commit 5d34664 into master Sep 18, 2023
9 checks passed

lexnv deleted the lexnv/chainhed_trie_db_merkle branch September 18, 2023 10:54

kiltbot mentioned this pull request Oct 19, 2023

[AUTOMATIC] Update Polkadot dependencies from 1.1.0 to 1.2.0 KILTprotocol/kilt-node#571

Closed

ukint-vs mentioned this pull request Apr 17, 2024

feat: Update substrate to v1.2.0 gear-tech/gear#3898

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chainHead: Add support for storage closest merkle descendant #14818 #1153

chainHead: Add support for storage closest merkle descendant #14818 #1153

lexnv commented Aug 25, 2023 •

edited

Loading

skunert left a comment

This comment was marked as duplicate.

tadeohepperle left a comment

tadeohepperle Sep 15, 2023

lexnv Sep 18, 2023

chainHead: Add support for storage closest merkle descendant #14818 #1153

chainHead: Add support for storage closest merkle descendant #14818 #1153

Conversation

lexnv commented Aug 25, 2023 • edited Loading

skunert left a comment

Choose a reason for hiding this comment

This comment was marked as duplicate.

tadeohepperle left a comment

Choose a reason for hiding this comment

tadeohepperle Sep 15, 2023

Choose a reason for hiding this comment

lexnv Sep 18, 2023

Choose a reason for hiding this comment

lexnv commented Aug 25, 2023 •

edited

Loading