-
Notifications
You must be signed in to change notification settings - Fork 683
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[FRAME] Runtime Omni Bencher #3512
Merged
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
ggwpez
changed the title
[FRAME] Free Standing pallet benchmark runner
[FRAME] Free Standing Pallet Benchmark Runner
Feb 28, 2024
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
ggwpez
changed the title
[FRAME] Free Standing Pallet Benchmark Runner
[FRAME] Runtime Omni Bencher
Mar 21, 2024
ggwpez
added
the
T12-benchmarks
This PR/Issue is related to benchmarking and weights.
label
Mar 21, 2024
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
This reverts commit 95b0e36.
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Ank4n
pushed a commit
that referenced
this pull request
Apr 9, 2024
This MR contains two major changes and some maintenance cleanup. ## 1. Free Standing Pallet Benchmark Runner Closes #3045, depends on your runtime exposing the `GenesisBuilderApi` (like #1492). Introduces a new binary crate: `frame-omni-bencher`. It allows to directly benchmark a WASM blob - without needing a node or chain spec. This makes it much easier to generate pallet weights and should allow us to remove bloaty code from the node. It should work for all FRAME runtimes that dont use 3rd party host calls or non `BlakeTwo256` block hashing (basically all polkadot parachains should work). It is 100% backwards compatible with the old CLI args, when the `v1` compatibility command is used. This is done to allow for forwards compatible addition of new commands. ### Example (full example in the Rust docs) Installing the CLI: ```sh cargo install --locked --path substrate/utils/frame/omni-bencher frame-omni-bencher --help ``` Building the Westend runtime: ```sh cargo build -p westend-runtime --release --features runtime-benchmarks ``` Benchmarking the runtime: ```sh frame-omni-bencher v1 benchmark pallet --runtime target/release/wbuild/westend-runtime/westend_runtime.compact.compressed.wasm --all ``` ## 2. Building the Benchmark Genesis State in the Runtime Closes #2664 This adds `--runtime` and `--genesis-builder=none|runtime|spec` arguments to the `benchmark pallet` command to make it possible to generate the genesis storage by the runtime. This can be used with both the node and the freestanding benchmark runners. It utilizes the new `GenesisBuilder` RA and depends on having #3412 deployed. ## 3. Simpler args for `PalletCmd::run` You can do three things here to integrate the changes into your node: - nothing: old code keeps working as before but emits a deprecated warning - delete: remove the pallet benchmarking code from your node and use the omni-bencher instead - patch: apply the patch below and keep using as currently. This emits a deprecated warning at runtime, since it uses the old way to generate a genesis state, but is the smallest change. ```patch runner.sync_run(|config| cmd - .run::<HashingFor<Block>, ReclaimHostFunctions>(config) + .run_with_spec::<HashingFor<Block>, ReclaimHostFunctions>(Some(config.chain_spec)) ) ``` ## 4. Maintenance Change - `pallet-nis` get a `BenchmarkSetup` config item to prepare its counterparty asset. - Add percent progress print when running benchmarks. - Dont immediately exit on benchmark error but try to run as many as possible and print errors last. --------- Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Co-authored-by: Liam Aharon <liam.aharon@hotmail.com>
dharjeezy
pushed a commit
to dharjeezy/polkadot-sdk
that referenced
this pull request
Apr 9, 2024
This MR contains two major changes and some maintenance cleanup. ## 1. Free Standing Pallet Benchmark Runner Closes paritytech#3045, depends on your runtime exposing the `GenesisBuilderApi` (like paritytech#1492). Introduces a new binary crate: `frame-omni-bencher`. It allows to directly benchmark a WASM blob - without needing a node or chain spec. This makes it much easier to generate pallet weights and should allow us to remove bloaty code from the node. It should work for all FRAME runtimes that dont use 3rd party host calls or non `BlakeTwo256` block hashing (basically all polkadot parachains should work). It is 100% backwards compatible with the old CLI args, when the `v1` compatibility command is used. This is done to allow for forwards compatible addition of new commands. ### Example (full example in the Rust docs) Installing the CLI: ```sh cargo install --locked --path substrate/utils/frame/omni-bencher frame-omni-bencher --help ``` Building the Westend runtime: ```sh cargo build -p westend-runtime --release --features runtime-benchmarks ``` Benchmarking the runtime: ```sh frame-omni-bencher v1 benchmark pallet --runtime target/release/wbuild/westend-runtime/westend_runtime.compact.compressed.wasm --all ``` ## 2. Building the Benchmark Genesis State in the Runtime Closes paritytech#2664 This adds `--runtime` and `--genesis-builder=none|runtime|spec` arguments to the `benchmark pallet` command to make it possible to generate the genesis storage by the runtime. This can be used with both the node and the freestanding benchmark runners. It utilizes the new `GenesisBuilder` RA and depends on having paritytech#3412 deployed. ## 3. Simpler args for `PalletCmd::run` You can do three things here to integrate the changes into your node: - nothing: old code keeps working as before but emits a deprecated warning - delete: remove the pallet benchmarking code from your node and use the omni-bencher instead - patch: apply the patch below and keep using as currently. This emits a deprecated warning at runtime, since it uses the old way to generate a genesis state, but is the smallest change. ```patch runner.sync_run(|config| cmd - .run::<HashingFor<Block>, ReclaimHostFunctions>(config) + .run_with_spec::<HashingFor<Block>, ReclaimHostFunctions>(Some(config.chain_spec)) ) ``` ## 4. Maintenance Change - `pallet-nis` get a `BenchmarkSetup` config item to prepare its counterparty asset. - Add percent progress print when running benchmarks. - Dont immediately exit on benchmark error but try to run as many as possible and print errors last. --------- Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Co-authored-by: Liam Aharon <liam.aharon@hotmail.com>
This was referenced Jun 5, 2024
4 tasks
2 tasks
enddynayn
added a commit
to frequency-chain/frequency
that referenced
this pull request
Aug 12, 2024
- Upgrade Polkadot-sdk 1.10.0 to 1.11.0 - Update weights to reflect the new version. Notable Changes: - [Apply patch `run_with_spec`](paritytech/polkadot-sdk#3512) - [Genesis Presets](paritytech/polkadot-sdk#2714) - [Integrate LiteP2P](paritytech/polkadot-sdk#2944) - [Template Simplification](https://github.com/paritytech/polkadot-sdk/pull/3801/files) For more details, please refer to: [Release Notes](https://github.com/paritytech/polkadot-sdk/releases/tag/polkadot-v1.11.0) issue-1957
enddynayn
added a commit
to frequency-chain/frequency
that referenced
this pull request
Aug 13, 2024
- Upgrade Polkadot-sdk 1.10.0 to 1.11.0 - Update weights to reflect the new version. Notable Changes: - [Apply patch `run_with_spec`](paritytech/polkadot-sdk#3512) - [Genesis Presets](paritytech/polkadot-sdk#2714) - [Integrate LiteP2P](paritytech/polkadot-sdk#2944) - [Template Simplification](https://github.com/paritytech/polkadot-sdk/pull/3801/files) For more details, please refer to: [Release Notes](https://github.com/paritytech/polkadot-sdk/releases/tag/polkadot-v1.11.0) issue-1957
enddynayn
added a commit
to frequency-chain/frequency
that referenced
this pull request
Aug 13, 2024
- Upgrade Polkadot-sdk 1.10.0 to 1.11.0 - Update weights to reflect the new version. Notable Changes: - [Apply patch `run_with_spec`](paritytech/polkadot-sdk#3512) - [Genesis Presets](paritytech/polkadot-sdk#2714) - [Integrate LiteP2P](paritytech/polkadot-sdk#2944) - [Template Simplification](https://github.com/paritytech/polkadot-sdk/pull/3801/files) For more details, please refer to: [Release Notes](https://github.com/paritytech/polkadot-sdk/releases/tag/polkadot-v1.11.0) issue-1957
enddynayn
added a commit
to frequency-chain/frequency
that referenced
this pull request
Aug 13, 2024
- Upgrade Polkadot-sdk 1.10.0 to 1.11.0 - Update weights to reflect the new version. Notable Changes: - [Apply patch `run_with_spec`](paritytech/polkadot-sdk#3512) - [Genesis Presets](paritytech/polkadot-sdk#2714) - [Integrate LiteP2P](paritytech/polkadot-sdk#2944) - [Template Simplification](https://github.com/paritytech/polkadot-sdk/pull/3801/files) For more details, please refer to: [Release Notes](https://github.com/paritytech/polkadot-sdk/releases/tag/polkadot-v1.11.0) issue-1957
enddynayn
added a commit
to frequency-chain/frequency
that referenced
this pull request
Aug 13, 2024
- Upgrade Polkadot-sdk 1.10.0 to 1.11.0 - Update weights to reflect the new version. Notable Changes: - [Apply patch `run_with_spec`](paritytech/polkadot-sdk#3512) - [Genesis Presets](paritytech/polkadot-sdk#2714) - [Integrate LiteP2P](paritytech/polkadot-sdk#2944) - [Template Simplification](https://github.com/paritytech/polkadot-sdk/pull/3801/files) - [Remove try-runtime-cli](https://github.com/paritytech/polkadot-sdk/pull/4017/files#diff-3a3aa5e088741f8ab1ca38fbe314c7a150d18b7466426f4ad2569113017e8439) For more details, please refer to: [Release Notes](https://github.com/paritytech/polkadot-sdk/releases/tag/polkadot-v1.11.0) #1957
aramikm
pushed a commit
to frequency-chain/frequency
that referenced
this pull request
Aug 15, 2024
- Upgrade Polkadot-sdk 1.10.0 to 1.11.0 - Update weights to reflect the new version. Notable Changes: - [Apply patch `run_with_spec`](paritytech/polkadot-sdk#3512) - [Genesis Presets](paritytech/polkadot-sdk#2714) - [Integrate LiteP2P](paritytech/polkadot-sdk#2944) - [Template Simplification](https://github.com/paritytech/polkadot-sdk/pull/3801/files) - [Remove try-runtime-cli](https://github.com/paritytech/polkadot-sdk/pull/4017/files#diff-3a3aa5e088741f8ab1ca38fbe314c7a150d18b7466426f4ad2569113017e8439) For more details, please refer to: [Release Notes](https://github.com/paritytech/polkadot-sdk/releases/tag/polkadot-v1.11.0)
rustadot
pushed a commit
to rustadot/recurrency
that referenced
this pull request
Sep 5, 2024
- Upgrade Polkadot-sdk 1.10.0 to 1.11.0 - Update weights to reflect the new version. Notable Changes: - [Apply patch `run_with_spec`](paritytech/polkadot-sdk#3512) - [Genesis Presets](paritytech/polkadot-sdk#2714) - [Integrate LiteP2P](paritytech/polkadot-sdk#2944) - [Template Simplification](https://github.com/paritytech/polkadot-sdk/pull/3801/files) - [Remove try-runtime-cli](https://github.com/paritytech/polkadot-sdk/pull/4017/files#diff-3a3aa5e088741f8ab1ca38fbe314c7a150d18b7466426f4ad2569113017e8439) For more details, please refer to: [Release Notes](https://github.com/paritytech/polkadot-sdk/releases/tag/polkadot-v1.11.0) #1957
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This MR contains two major changes and some maintenance cleanup.
1. Free Standing Pallet Benchmark Runner
Closes #3045, depends on your runtime exposing the
GenesisBuilderApi
(like #1492).Introduces a new binary crate:
frame-omni-bencher
.It allows to directly benchmark a WASM blob - without needing a node or chain spec.
This makes it much easier to generate pallet weights and should allow us to remove bloaty code from the node.
It should work for all FRAME runtimes that dont use 3rd party host calls or non
BlakeTwo256
block hashing (basically all polkadot parachains should work).It is 100% backwards compatible with the old CLI args, when the
v1
compatibility command is used. This is done to allow for forwards compatible addition of new commands.Example (full example in the Rust docs)
Installing the CLI:
Building the Westend runtime:
Benchmarking the runtime:
2. Building the Benchmark Genesis State in the Runtime
Closes #2664
This adds
--runtime
and--genesis-builder=none|runtime|spec
arguments to thebenchmark pallet
command to make it possible to generate the genesis storage by the runtime. This can be used with both the node and the freestanding benchmark runners. It utilizes the newGenesisBuilder
RA and depends on having #3412 deployed.3. Simpler args for
PalletCmd::run
You can do three things here to integrate the changes into your node:
4. Maintenance Change
pallet-nis
get aBenchmarkSetup
config item to prepare its counterparty asset.