Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PoV Reclaim (Clawback) Node Side #1462

Merged
merged 80 commits into from
Nov 30, 2023
Merged
Show file tree
Hide file tree
Changes from 50 commits
Commits
Show all changes
80 commits
Select commit Hold shift + click to select a range
01b858a
Transfer substrate clawback to monorepo
skunert Aug 25, 2023
9c21d3c
Clawback cumulus monorepo transfer
skunert Aug 25, 2023
f71f137
debugging: Implement SignedExtension
skunert Aug 29, 2023
a30abde
Recorder Experiment
skunert Aug 29, 2023
0370360
Merge branch 'master' into skunert/clawback
skunert Aug 31, 2023
84eb1e7
Make recorder impl compile
skunert Sep 4, 2023
649ec11
Imports
skunert Sep 5, 2023
b17e91c
Simplify
skunert Sep 5, 2023
da5fe57
Merge branch 'master' into clawbac-simplified
skunert Sep 5, 2023
dfefa16
Simplify
skunert Sep 5, 2023
177d2dd
Clean up and implement externalities
skunert Sep 5, 2023
aeee135
Rename clawback -> reclaim
skunert Sep 5, 2023
d70f3c3
Merge branch 'master' into clawbac-simplified
skunert Sep 5, 2023
42688a3
Cleanup
skunert Sep 5, 2023
d9ee3e3
Remove debug logs
skunert Sep 5, 2023
678bb97
Implement `trie_nodes_recorded_for_key`
skunert Sep 6, 2023
08fab5e
Improve validate_block bench
skunert Sep 6, 2023
9ebc861
Move client parameter to config
skunert Sep 7, 2023
a303061
Merge branch 'master' into clawbac-simplified
skunert Sep 7, 2023
99ccf64
Remove unwanted changes
skunert Sep 7, 2023
783e0f3
Add some tests for recorder
skunert Sep 7, 2023
8726f78
Add cache to test
skunert Sep 8, 2023
f44111f
Improve tests and rename Recorder
skunert Sep 8, 2023
d57e689
Remove last debugging relicts
skunert Sep 8, 2023
5b317fd
Fix name in validate_block implementation
skunert Sep 8, 2023
2793696
Reset trie-cache between runs
skunert Sep 8, 2023
652d1d8
Improve benchmarks
skunert Sep 8, 2023
00c734a
".git/.scripts/commands/fmt/fmt.sh"
Sep 8, 2023
961f750
CI fixes
skunert Sep 8, 2023
15f8c5d
Reintroduce extension
skunert Sep 11, 2023
0a30b70
Use recorder in validate_block
skunert Sep 12, 2023
a1d7db5
Move recorder into static mut
skunert Sep 12, 2023
4abdc43
Use `Execution_Extensions` to register the extensions
skunert Sep 12, 2023
61193c9
Merge branch 'master' into pov-reclaim
skunert Sep 12, 2023
77fbed6
Unchange `proof_size`
skunert Sep 12, 2023
7057770
Zepter & CI
skunert Sep 12, 2023
9e195ce
CI
skunert Sep 12, 2023
bd2219f
Add some test for the host function
skunert Sep 14, 2023
fb09e49
Merge branch 'master' into pov-reclaim
skunert Sep 18, 2023
b51bf3a
Adjust recorder to match new trie-db version
skunert Sep 18, 2023
480d990
Apply suggestions from code review
skunert Sep 19, 2023
bb5ecb1
Review comments
skunert Sep 19, 2023
f2a5720
Merge branch 'master' into pov-reclaim
skunert Sep 19, 2023
4e3cd31
Review comments
skunert Sep 19, 2023
91ffa72
Remove size estimation and make proof optional
skunert Sep 19, 2023
a7fe651
Move extension to sp-trie crate
skunert Sep 19, 2023
a367f06
Improve docs & format features
skunert Sep 19, 2023
3701a7a
Sorry clippy
skunert Sep 19, 2023
795e68e
Remove reference to private struct.
skunert Sep 20, 2023
fe46117
Merge branch 'master' into pov-reclaim
skunert Sep 26, 2023
57976f6
Apply suggestions from code review
skunert Sep 29, 2023
7bda9cc
Use environmental crate instead of `static mut`
skunert Sep 29, 2023
fa22eb4
Review comments
skunert Sep 29, 2023
0de221a
Merge branch 'master' into pov-reclaim
skunert Sep 29, 2023
44a76cd
Merge branch 'master' into pov-reclaim
skunert Oct 16, 2023
7ef514e
".git/.scripts/commands/fmt/fmt.sh"
Oct 17, 2023
bb0489e
Move to substrate
skunert Oct 27, 2023
205c34d
u32->u64
skunert Oct 27, 2023
2e0c9e0
Merge branch 'master' into pov-reclaim
skunert Oct 27, 2023
ae410c7
fmt
skunert Oct 27, 2023
14be797
Adjust naming to differentiate from trait
skunert Oct 30, 2023
4d6f547
Review comments
skunert Oct 30, 2023
89e59dd
Merge branch 'master' into pov-reclaim
skunert Oct 30, 2023
cdb04ce
Move back to cumulus
skunert Oct 31, 2023
416a507
Instantiate extension only in block builder and import
skunert Oct 31, 2023
8014f90
Fix compilation & clippy
skunert Nov 1, 2023
c69cf59
".git/.scripts/commands/fmt/fmt.sh"
Nov 1, 2023
21b388a
Remove sp-trie from rpc-spec-v2
skunert Nov 1, 2023
c4b0e05
Remove usages of host function
skunert Nov 2, 2023
0df1c6b
Merge branch 'master' into pov-reclaim
skunert Nov 2, 2023
8739e61
Update lib.rs
skunert Nov 3, 2023
a7ea2a5
Fix map_or
skunert Nov 6, 2023
c33a27c
Merge branch 'master' into pov-reclaim
skunert Nov 6, 2023
37ebd47
Merge branch 'master' into pov-reclaim
skunert Nov 9, 2023
be98bf2
Merge branch 'master' into pov-reclaim
skunert Nov 10, 2023
115db20
Use u64::MAX as proof recording disabled signal
skunert Nov 10, 2023
f0a8699
Merge branch 'master' into pov-reclaim
skunert Nov 27, 2023
ea917d8
Merge branch 'master' into pov-reclaim
skunert Nov 30, 2023
1035709
Add crate description
skunert Nov 30, 2023
711e1f1
Remove some unwanted proof recordings
skunert Nov 30, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 4 additions & 0 deletions cumulus/pallets/parachain-system/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -35,11 +35,14 @@ xcm = { package = "staging-xcm", path = "../../../polkadot/xcm", default-feature
cumulus-pallet-parachain-system-proc-macro = { path = "proc-macro", default-features = false }
cumulus-primitives-core = { path = "../../primitives/core", default-features = false }
cumulus-primitives-parachain-inherent = { path = "../../primitives/parachain-inherent", default-features = false }
cumulus-primitives-pov-reclaim = { path = "../../primitives/pov-reclaim", default-features = false }

[dev-dependencies]
assert_matches = "1.5"
hex-literal = "0.4.1"
lazy_static = "1.4"
trie-standardmap = "0.16.0"
rand = "0.8.5"

# Substrate
sc-client-api = { path = "../../../substrate/client/api" }
Expand All @@ -58,6 +61,7 @@ std = [
"cumulus-pallet-parachain-system-proc-macro/std",
"cumulus-primitives-core/std",
"cumulus-primitives-parachain-inherent/std",
"cumulus-primitives-pov-reclaim/std",
"environmental/std",
"frame-support/std",
"frame-system/std",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@

//! The actual implementation of the validate block functionality.

use super::{trie_cache, MemoryOptimizedValidationParams};
use super::{trie_cache, trie_recorder, MemoryOptimizedValidationParams};
use cumulus_primitives_core::{
relay_chain::Hash as RHash, ParachainBlockData, PersistedValidationData,
};
Expand All @@ -33,13 +33,15 @@ use sp_core::storage::{ChildInfo, StateVersion};
use sp_externalities::{set_and_run_with_externalities, Externalities};
use sp_io::KillStorageResult;
use sp_runtime::traits::{Block as BlockT, Extrinsic, HashingFor, Header as HeaderT};
use sp_std::prelude::*;
use sp_trie::MemoryDB;
use sp_std::{prelude::*, sync::Arc};
use sp_trie::{MemoryDB, ProofSizeProvider, TrieRecorderProvider};
use trie_recorder::SizeOnlyRecorderProvider;

type TrieBackend<B> = sp_state_machine::TrieBackend<
MemoryDB<HashingFor<B>>,
HashingFor<B>,
trie_cache::CacheProvider<HashingFor<B>>,
SizeOnlyRecorderProvider<HashingFor<B>>,
>;

type Ext<'a, B> = sp_state_machine::Ext<'a, HashingFor<B>, TrieBackend<B>>;
Expand All @@ -48,6 +50,10 @@ fn with_externalities<F: FnOnce(&mut dyn Externalities) -> R, R>(f: F) -> R {
sp_externalities::with_externalities(f).expect("Environmental externalities not set.")
}

/// Recorder instance to be used during this validate_block call.
/// Static mut is fine here because we are single-threaded in WASM.
static mut RECORDER: Option<Box<dyn ProofSizeProvider + Send + Sync>> = None;
skunert marked this conversation as resolved.
Show resolved Hide resolved

/// Validate the given parachain block.
///
/// This function is doing roughly the following:
Expand Down Expand Up @@ -120,6 +126,7 @@ where

sp_std::mem::drop(storage_proof);

let recorder = SizeOnlyRecorderProvider::new();
let cache_provider = trie_cache::CacheProvider::new();
// We use the storage root of the `parent_head` to ensure that it is the correct root.
// This is already being done above while creating the in-memory db, but let's be paranoid!!
Expand All @@ -128,8 +135,11 @@ where
*parent_header.state_root(),
cache_provider,
)
.with_recorder(recorder.clone())
.build();

set_recorder(recorder);

let _guard = (
// Replace storage calls with our own implementations
sp_io::storage::host_read.replace_implementation(host_storage_read),
Expand Down Expand Up @@ -167,6 +177,8 @@ where
.replace_implementation(host_default_child_storage_next_key),
sp_io::offchain_index::host_set.replace_implementation(host_offchain_index_set),
sp_io::offchain_index::host_clear.replace_implementation(host_offchain_index_clear),
cumulus_primitives_pov_reclaim::pov_reclaim_host_functions::host_current_storage_proof_size
.replace_implementation(host_current_storage_proof_size),
);

run_with_externalities::<B, _, _>(&backend, || {
Expand Down Expand Up @@ -305,6 +317,14 @@ fn host_storage_clear(key: &[u8]) {
with_externalities(|ext| ext.place_storage(key.to_vec(), None))
}

fn host_current_storage_proof_size() -> u32 {
get_recorder_ref()
.map(|r| r.estimate_encoded_size())
.unwrap_or_default()
skunert marked this conversation as resolved.
Show resolved Hide resolved
.try_into()
.unwrap_or_default()
skunert marked this conversation as resolved.
Show resolved Hide resolved
}

fn host_storage_root(version: StateVersion) -> Vec<u8> {
with_externalities(|ext| ext.storage_root(version))
}
Expand Down Expand Up @@ -406,3 +426,14 @@ fn host_default_child_storage_next_key(storage_key: &[u8], key: &[u8]) -> Option
fn host_offchain_index_set(_key: &[u8], _value: &[u8]) {}

fn host_offchain_index_clear(_key: &[u8]) {}

fn set_recorder<H: sp_core::Hasher + 'static>(recorder: SizeOnlyRecorderProvider<H>) {
// This is safe here, there is strictly sequential access.
unsafe {
RECORDER = Some(Box::new(recorder));
}
}

fn get_recorder_ref() -> Option<&'static Box<dyn ProofSizeProvider + Send + Sync>> {
unsafe { RECORDER.as_ref() }
}
4 changes: 4 additions & 0 deletions cumulus/pallets/parachain-system/src/validate_block/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,10 @@ mod tests;
#[doc(hidden)]
mod trie_cache;

#[cfg(any(test, not(feature = "std")))]
#[doc(hidden)]
mod trie_recorder;

#[cfg(not(feature = "std"))]
#[doc(hidden)]
pub use bytes;
Expand Down
Loading
Loading