Add Proof Size to Weight Output (#11637)

* initial impl * add template test * linear fit proof size * always record proof when tracking storage * calculate worst case pov * remove duplicate worst case * cargo run --quiet --profile=production --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark pallet --chain=dev --steps=50 --repeat=20 --pallet=pallet_assets --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/assets/src/weights.rs --template=./.maintain/frame-weight-template.hbs * more comment output * add cli for worst case map size * update name * clap does not support underscores * rename * expose worst case map values * improve some comments * cargo run --quiet --profile=production --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark pallet --chain=dev --steps=50 --repeat=20 --pallet=pallet_assets --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/assets/src/weights.rs --template=./.maintain/frame-weight-template.hbs * update template * cargo run --quiet --profile=production --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark pallet --chain=dev --steps=50 --repeat=20 --pallet=pallet_assets --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/assets/src/weights.rs --template=./.maintain/frame-weight-template.hbs * fix fmt * more fmt * more fmt * Dont panic when there is no proof Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Fix test features Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Whitelist :extrinsic_index Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Use whitelist when recording proof Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Add logs Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Add PoV testing pallet Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Deploy PoV testing pallet Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Storage benches reside in the PoV pallet Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Linear regress PoV per component Splits the PoV calculation into "measured" and "estimated". The measured part is reported by the Proof recorder and linear regressed over all components at once. The estimated part is calculated as worst-case by using the max PoV size per storage access and calculating one linear regress per component. This gives each component a (possibly) independent PoV. For now the measured size will always be lower than the PoV on Polkadot since it is measured on an empty snapshot. The measured part is therefor only used as diagnostic for debugging. Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Put PoV into the weight templates Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * fmt Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Extra alanysis choise for PoV Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Add+Fix tests Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Make benches faster Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Cleanup Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Use same template comments Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * ".git/.scripts/bench-bot.sh" pallet dev pallet_balances * ".git/.scripts/bench-bot.sh" pallet dev pallet_democracy * Update referenda mock BlockWeights Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Take measured value size into account Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * clippy Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * ".git/.scripts/bench-bot.sh" pallet dev pallet_scheduler * WIP Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * proof_size: None Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * WIP Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * WIP Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * WIP Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * ugly, but works Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * WIP Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * WIP Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * WIP Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * wup Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * WIP Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * WIP Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * WIP Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * WIP Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Add pov_mode attribute to the benchmarks! macro Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Use pov_mode attribute in PoV benchmarking Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Update tests Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Scheduler, Whitelist: Add pov_mode attr Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Update PoV weights * Add CLI arg: default-pov-mode Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Fix tests Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * fmt Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * fix Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Revert "Update PoV weights" This reverts commit 2f3ac23. * Revert "WIP" This reverts commit c34b538. * Revert first approach This reverts commit range 8ddaa2f^..4c84f87 Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Clippy Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Add extra benchmarks Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * ".git/.scripts/commands/bench/bench.sh" pallet dev pallet_alliance * ".git/.scripts/commands/bench/bench.sh" pallet dev pallet_whitelist * ".git/.scripts/commands/bench/bench.sh" pallet dev pallet_scheduler * fmt Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Clippy Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Clippy 🤦 Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Add reference benchmarks Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Fix doc comments Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Undo logging Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Add 'Ignored' pov_mode Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Allow multiple attributes per benchmark Turns out that the current benchmarking syntax does not support multiple attributes per bench 🤦. Changing it to support that since otherwise the `pov_mode` would conflict with the others. Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Validate pov_mode syntax Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Ignore PoV for all contract benchmarks Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Test Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * test Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Bump macro recursion limit Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * fmt Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Update contract weights They dont have a PoV component anymore. Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * fix test ffs Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * pov_mode is unsupported in V2 syntax Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Fix pallet ui tests Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * update pallet ui Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Fix pallet ui tests Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Update weights Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Co-authored-by: Parity Bot <admin@parity.io> Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Co-authored-by: command-bot <> Co-authored-by: Your Name <you@example.com>
paritytech · Feb 3, 2023 · ab1e032 · ab1e032
1 parent 5913d39
commit ab1e032
Show file tree

Hide file tree

Showing 74 changed files with 19,169 additions and 9,619 deletions.
diff --git a/.maintain/frame-weight-template.hbs b/.maintain/frame-weight-template.hbs
@@ -2,7 +2,8 @@
 //! Autogenerated weights for {{pallet}}
 //!
 //! THIS FILE WAS AUTO-GENERATED USING THE SUBSTRATE BENCHMARK CLI VERSION {{version}}
-//! DATE: {{date}}, STEPS: `{{cmd.steps}}`, REPEAT: {{cmd.repeat}}, LOW RANGE: `{{cmd.lowest_range_values}}`, HIGH RANGE: `{{cmd.highest_range_values}}`
+//! DATE: {{date}}, STEPS: `{{cmd.steps}}`, REPEAT: `{{cmd.repeat}}`, LOW RANGE: `{{cmd.lowest_range_values}}`, HIGH RANGE: `{{cmd.highest_range_values}}`
+//! WORST CASE MAP SIZE: `{{cmd.worst_case_map_values}}`
 //! HOSTNAME: `{{hostname}}`, CPU: `{{cpuname}}`
 //! EXECUTION: {{cmd.execution}}, WASM-EXECUTION: {{cmd.wasm_execution}}, CHAIN: {{cmd.chain}}, DB CACHE: {{cmd.db_cache}}
 
@@ -38,7 +39,7 @@ impl<T: frame_system::Config> WeightInfo for SubstrateWeight<T> {
 {{/if}}
 	{{#each benchmarks as |benchmark|}}
 	{{#each benchmark.comments as |comment|}}
-	// {{comment}}
+	/// {{comment}}
 	{{/each}}
 	{{#each benchmark.component_ranges as |range|}}
 	/// The range of component `{{range.name}}` is `[{{range.min}}, {{range.max}}]`.
@@ -48,24 +49,34 @@ impl<T: frame_system::Config> WeightInfo for SubstrateWeight<T> {
 		{{~#each benchmark.components as |c| ~}}
 		{{~#if (not c.is_used)}}_{{/if}}{{c.name}}: u32, {{/each~}}
 	) -> Weight {
+		// Proof Size summary in bytes:
+		//  Measured:  `{{benchmark.base_recorded_proof_size}}{{#each benchmark.component_recorded_proof_size as |cp|}} + {{cp.name}} * ({{cp.slope}} ±{{underscore cp.error}}){{/each}}`
+		//  Estimated: `{{benchmark.base_calculated_proof_size}}{{#each benchmark.component_calculated_proof_size as |cp|}} + {{cp.name}} * ({{cp.slope}} ±{{underscore cp.error}}){{/each}}`
 		// Minimum execution time: {{underscore benchmark.min_execution_time}} nanoseconds.
+		{{#if (ne benchmark.base_calculated_proof_size "0")}}
+		Weight::from_parts({{underscore benchmark.base_weight}}, {{benchmark.base_calculated_proof_size}})
+		{{else}}
 		Weight::from_ref_time({{underscore benchmark.base_weight}})
+		{{/if}}
 			{{#each benchmark.component_weight as |cw|}}
 			// Standard Error: {{underscore cw.error}}
 			.saturating_add(Weight::from_ref_time({{underscore cw.slope}}).saturating_mul({{cw.name}}.into()))
 			{{/each}}
 			{{#if (ne benchmark.base_reads "0")}}
-			.saturating_add(T::DbWeight::get().reads({{benchmark.base_reads}}))
+			.saturating_add(T::DbWeight::get().reads({{benchmark.base_reads}}_u64))
 			{{/if}}
 			{{#each benchmark.component_reads as |cr|}}
 			.saturating_add(T::DbWeight::get().reads(({{cr.slope}}_u64).saturating_mul({{cr.name}}.into())))
 			{{/each}}
 			{{#if (ne benchmark.base_writes "0")}}
-			.saturating_add(T::DbWeight::get().writes({{benchmark.base_writes}}))
+			.saturating_add(T::DbWeight::get().writes({{benchmark.base_writes}}_u64))
 			{{/if}}
 			{{#each benchmark.component_writes as |cw|}}
 			.saturating_add(T::DbWeight::get().writes(({{cw.slope}}_u64).saturating_mul({{cw.name}}.into())))
 			{{/each}}
+			{{#each benchmark.component_calculated_proof_size as |cp|}}
+			.saturating_add(Weight::from_proof_size({{cp.slope}}).saturating_mul({{cp.name}}.into()))
+			{{/each}}
 	}
 	{{/each}}
 }
@@ -74,7 +85,7 @@ impl<T: frame_system::Config> WeightInfo for SubstrateWeight<T> {
 impl WeightInfo for () {
 	{{#each benchmarks as |benchmark|}}
 	{{#each benchmark.comments as |comment|}}
-	// {{comment}}
+	/// {{comment}}
 	{{/each}}
 	{{#each benchmark.component_ranges as |range|}}
 	/// The range of component `{{range.name}}` is `[{{range.min}}, {{range.max}}]`.
@@ -84,24 +95,34 @@ impl WeightInfo for () {
 		{{~#each benchmark.components as |c| ~}}
 		{{~#if (not c.is_used)}}_{{/if}}{{c.name}}: u32, {{/each~}}
 	) -> Weight {
+		// Proof Size summary in bytes:
+		//  Measured:  `{{benchmark.base_recorded_proof_size}}{{#each benchmark.component_recorded_proof_size as |cp|}} + {{cp.name}} * ({{cp.slope}} ±{{underscore cp.error}}){{/each}}`
+		//  Estimated: `{{benchmark.base_calculated_proof_size}}{{#each benchmark.component_calculated_proof_size as |cp|}} + {{cp.name}} * ({{cp.slope}} ±{{underscore cp.error}}){{/each}}`
 		// Minimum execution time: {{underscore benchmark.min_execution_time}} nanoseconds.
+		{{#if (ne benchmark.base_calculated_proof_size "0")}}
+		Weight::from_parts({{underscore benchmark.base_weight}}, {{benchmark.base_calculated_proof_size}})
+		{{else}}
 		Weight::from_ref_time({{underscore benchmark.base_weight}})
+		{{/if}}
 			{{#each benchmark.component_weight as |cw|}}
 			// Standard Error: {{underscore cw.error}}
 			.saturating_add(Weight::from_ref_time({{underscore cw.slope}}).saturating_mul({{cw.name}}.into()))
 			{{/each}}
 			{{#if (ne benchmark.base_reads "0")}}
-			.saturating_add(RocksDbWeight::get().reads({{benchmark.base_reads}}))
+			.saturating_add(RocksDbWeight::get().reads({{benchmark.base_reads}}_u64))
 			{{/if}}
 			{{#each benchmark.component_reads as |cr|}}
 			.saturating_add(RocksDbWeight::get().reads(({{cr.slope}}_u64).saturating_mul({{cr.name}}.into())))
 			{{/each}}
 			{{#if (ne benchmark.base_writes "0")}}
-			.saturating_add(RocksDbWeight::get().writes({{benchmark.base_writes}}))
+			.saturating_add(RocksDbWeight::get().writes({{benchmark.base_writes}}_u64))
 			{{/if}}
 			{{#each benchmark.component_writes as |cw|}}
 			.saturating_add(RocksDbWeight::get().writes(({{cw.slope}}_u64).saturating_mul({{cw.name}}.into())))
 			{{/each}}
+			{{#each benchmark.component_calculated_proof_size as |cp|}}
+			.saturating_add(Weight::from_proof_size({{cp.slope}}).saturating_mul({{cp.name}}.into()))
+			{{/each}}
 	}
 	{{/each}}
 }
diff --git a/Cargo.lock b/Cargo.lock
diff --git a/Cargo.toml b/Cargo.toml
@@ -88,6 +88,7 @@ members = [
 	"frame/beefy-mmr",
 	"frame/beefy-mmr/primitives",
 	"frame/benchmarking",
+	"frame/benchmarking/pov",
 	"frame/bounties",
 	"frame/child-bounties",
 	"frame/collective",

diff --git a/bin/node/runtime/Cargo.toml b/bin/node/runtime/Cargo.toml
@@ -43,7 +43,8 @@ sp-io = { version = "7.0.0", default-features = false, path = "../../../primitiv
 
 # frame dependencies
 frame-executive = { version = "4.0.0-dev", default-features = false, path = "../../../frame/executive" }
-frame-benchmarking = { version = "4.0.0-dev", default-features = false, path = "../../../frame/benchmarking", optional = true }
+frame-benchmarking = { version = "4.0.0-dev", default-features = false, path = "../../../frame/benchmarking" }
+frame-benchmarking-pallet-pov = { version = "4.0.0-dev", default-features = false, path = "../../../frame/benchmarking/pov" }
 frame-support = { version = "4.0.0-dev", default-features = false, path = "../../../frame/support" }
 frame-system = { version = "4.0.0-dev", default-features = false, path = "../../../frame/system" }
 frame-system-benchmarking = { version = "4.0.0-dev", default-features = false, path = "../../../frame/system/benchmarking", optional = true }
@@ -178,7 +179,8 @@ std = [
 	"sp-session/std",
 	"pallet-sudo/std",
 	"frame-support/std",
-	"frame-benchmarking?/std",
+	"frame-benchmarking/std",
+	"frame-benchmarking-pallet-pov/std",
 	"frame-system-rpc-runtime-api/std",
 	"frame-system/std",
 	"pallet-election-provider-multi-phase/std",
@@ -209,6 +211,7 @@ std = [
 ]
 runtime-benchmarks = [
 	"frame-benchmarking/runtime-benchmarks",
+	"frame-benchmarking-pallet-pov/runtime-benchmarks",
 	"frame-support/runtime-benchmarks",
 	"frame-system/runtime-benchmarks",
 	"sp-runtime/runtime-benchmarks",
@@ -263,6 +266,7 @@ runtime-benchmarks = [
 ]
 try-runtime = [
 	"frame-try-runtime/try-runtime",
+	"frame-benchmarking-pallet-pov/try-runtime",
 	"frame-executive/try-runtime",
 	"frame-system/try-runtime",
 	"frame-support/try-runtime",

diff --git a/bin/node/runtime/src/lib.rs b/bin/node/runtime/src/lib.rs
@@ -1696,6 +1696,10 @@ impl pallet_alliance::Config for Runtime {
 	type RetirementPeriod = RetirementPeriod;
 }
 
+impl frame_benchmarking_pallet_pov::Config for Runtime {
+	type RuntimeEvent = RuntimeEvent;
+}
+
 construct_runtime!(
 	pub enum Runtime where
 		Block = Block,
@@ -1762,6 +1766,7 @@ construct_runtime!(
 		RankedCollective: pallet_ranked_collective,
 		FastUnstake: pallet_fast_unstake,
 		MessageQueue: pallet_message_queue,
+		Pov: frame_benchmarking_pallet_pov,
 	}
 );
 
@@ -1830,6 +1835,7 @@ mod mmr {
 mod benches {
 	frame_benchmarking::define_benchmarks!(
 		[frame_benchmarking, BaselineBench::<Runtime>]
+		[frame_benchmarking_pallet_pov, Pov]
 		[pallet_alliance, Alliance]
 		[pallet_assets, Assets]
 		[pallet_babe, Babe]

diff --git a/client/db/src/bench.rs b/client/db/src/bench.rs
@@ -606,10 +606,14 @@ impl<B: BlockT> StateBackend<HashFor<B>> for BenchmarkingState<B> {
 			let proof_recorder_root = self.proof_recorder_root.get();
 			if proof_recorder_root == Default::default() || proof_size == 1 {
 				// empty trie
+				log::debug!(target: "benchmark", "Some proof size: {}", &proof_size);
 				proof_size
 			} else {
 				if let Some(size) = proof.encoded_compact_size::<HashFor<B>>(proof_recorder_root) {
 					size as u32
+				} else if proof_recorder_root == self.root.get() {
+					log::debug!(target: "benchmark", "No changes - no proof");
+					0
 				} else {
 					panic!(
 						"proof rec root {:?}, root {:?}, genesis {:?}, rec_len {:?}",

diff --git a/client/db/src/upgrade.rs b/client/db/src/upgrade.rs
@@ -190,7 +190,7 @@ fn version_file_path(path: &Path) -> PathBuf {
 	file_path
 }
 
-#[cfg(test)]
+#[cfg(all(test, feature = "rocksdb"))]
 mod tests {
 	use super::*;
 	use crate::{tests::Block, DatabaseSource};

diff --git a/client/db/src/utils.rs b/client/db/src/utils.rs
@@ -583,12 +583,12 @@ mod tests {
 	use super::*;
 	use codec::Input;
 	use sp_runtime::testing::{Block as RawBlock, ExtrinsicWrapper};
-	use std::path::PathBuf;
 	type Block = RawBlock<ExtrinsicWrapper<u32>>;
 
-	#[cfg(any(feature = "rocksdb", test))]
+	#[cfg(feature = "rocksdb")]
 	#[test]
 	fn database_type_subdir_migration() {
+		use std::path::PathBuf;
 		type Block = RawBlock<ExtrinsicWrapper<u64>>;
 
 		fn check_dir_for_db_type(
@@ -685,6 +685,7 @@ mod tests {
 		assert_eq!(joined.remaining_len().unwrap(), Some(0));
 	}
 
+	#[cfg(feature = "rocksdb")]
 	#[test]
 	fn test_open_database_auto_new() {
 		let db_dir = tempfile::TempDir::new().unwrap();
@@ -730,6 +731,7 @@ mod tests {
 		}
 	}
 
+	#[cfg(feature = "rocksdb")]
 	#[test]
 	fn test_open_database_rocksdb_new() {
 		let db_dir = tempfile::TempDir::new().unwrap();
@@ -780,6 +782,7 @@ mod tests {
 		}
 	}
 
+	#[cfg(feature = "rocksdb")]
 	#[test]
 	fn test_open_database_paritydb_new() {
 		let db_dir = tempfile::TempDir::new().unwrap();