Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FRAME] Runtime Omni Bencher #3512

Merged
merged 51 commits into from
Apr 8, 2024
Merged

[FRAME] Runtime Omni Bencher #3512

merged 51 commits into from
Apr 8, 2024

Conversation

ggwpez
Copy link
Member

@ggwpez ggwpez commented Feb 28, 2024

This MR contains two major changes and some maintenance cleanup.

1. Free Standing Pallet Benchmark Runner

Closes #3045, depends on your runtime exposing the GenesisBuilderApi (like #1492).

Introduces a new binary crate: frame-omni-bencher.
It allows to directly benchmark a WASM blob - without needing a node or chain spec.

This makes it much easier to generate pallet weights and should allow us to remove bloaty code from the node.
It should work for all FRAME runtimes that dont use 3rd party host calls or non BlakeTwo256 block hashing (basically all polkadot parachains should work).

It is 100% backwards compatible with the old CLI args, when the v1 compatibility command is used. This is done to allow for forwards compatible addition of new commands.

Example (full example in the Rust docs)

Installing the CLI:

cargo install --locked --path substrate/utils/frame/omni-bencher
frame-omni-bencher --help

Building the Westend runtime:

cargo build -p westend-runtime --release --features runtime-benchmarks

Benchmarking the runtime:

frame-omni-bencher v1 benchmark pallet --runtime target/release/wbuild/westend-runtime/westend_runtime.compact.compressed.wasm --all

2. Building the Benchmark Genesis State in the Runtime

Closes #2664

This adds --runtime and --genesis-builder=none|runtime|spec arguments to the benchmark pallet command to make it possible to generate the genesis storage by the runtime. This can be used with both the node and the freestanding benchmark runners. It utilizes the new GenesisBuilder RA and depends on having #3412 deployed.

3. Simpler args for PalletCmd::run

You can do three things here to integrate the changes into your node:

  • nothing: old code keeps working as before but emits a deprecated warning
  • delete: remove the pallet benchmarking code from your node and use the omni-bencher instead
  • patch: apply the patch below and keep using as currently. This emits a deprecated warning at runtime, since it uses the old way to generate a genesis state, but is the smallest change.
runner.sync_run(|config| cmd
-    .run::<HashingFor<Block>, ReclaimHostFunctions>(config)
+    .run_with_spec::<HashingFor<Block>, ReclaimHostFunctions>(Some(config.chain_spec))
)

4. Maintenance Change

  • pallet-nis get a BenchmarkSetup config item to prepare its counterparty asset.
  • Add percent progress print when running benchmarks.
  • Dont immediately exit on benchmark error but try to run as many as possible and print errors last.

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
@ggwpez ggwpez changed the title [FRAME] Free Standing pallet benchmark runner [FRAME] Free Standing Pallet Benchmark Runner Feb 28, 2024
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
@ggwpez ggwpez changed the title [FRAME] Free Standing Pallet Benchmark Runner [FRAME] Runtime Omni Bencher Mar 21, 2024
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
@ggwpez ggwpez added the T12-benchmarks This PR/Issue is related to benchmarking and weights. label Mar 21, 2024
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
@ggwpez ggwpez marked this pull request as ready for review March 21, 2024 21:32
@ggwpez ggwpez requested a review from athei as a code owner March 21, 2024 21:32
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
@ggwpez ggwpez added this pull request to the merge queue Apr 8, 2024
Merged via the queue into master with commit 9543d31 Apr 8, 2024
131 of 135 checks passed
@ggwpez ggwpez deleted the oty-benchmark-from-wasm branch April 8, 2024 17:01
Ank4n pushed a commit that referenced this pull request Apr 9, 2024
This MR contains two major changes and some maintenance cleanup.  

## 1. Free Standing Pallet Benchmark Runner

Closes #3045, depends
on your runtime exposing the `GenesisBuilderApi` (like
#1492).

Introduces a new binary crate: `frame-omni-bencher`.  
It allows to directly benchmark a WASM blob - without needing a node or
chain spec.

This makes it much easier to generate pallet weights and should allow us
to remove bloaty code from the node.
It should work for all FRAME runtimes that dont use 3rd party host calls
or non `BlakeTwo256` block hashing (basically all polkadot parachains
should work).

It is 100% backwards compatible with the old CLI args, when the `v1`
compatibility command is used. This is done to allow for forwards
compatible addition of new commands.

### Example (full example in the Rust docs)

Installing the CLI:
```sh
cargo install --locked --path substrate/utils/frame/omni-bencher
frame-omni-bencher --help
```

Building the Westend runtime:
```sh
cargo build -p westend-runtime --release --features runtime-benchmarks
```

Benchmarking the runtime:
```sh
frame-omni-bencher v1 benchmark pallet --runtime target/release/wbuild/westend-runtime/westend_runtime.compact.compressed.wasm --all
```

## 2. Building the Benchmark Genesis State in the Runtime

Closes #2664

This adds `--runtime` and `--genesis-builder=none|runtime|spec`
arguments to the `benchmark pallet` command to make it possible to
generate the genesis storage by the runtime. This can be used with both
the node and the freestanding benchmark runners. It utilizes the new
`GenesisBuilder` RA and depends on having
#3412 deployed.

## 3. Simpler args for `PalletCmd::run`

You can do three things here to integrate the changes into your node:
- nothing: old code keeps working as before but emits a deprecated
warning
- delete: remove the pallet benchmarking code from your node and use the
omni-bencher instead
- patch: apply the patch below and keep using as currently. This emits a
deprecated warning at runtime, since it uses the old way to generate a
genesis state, but is the smallest change.

```patch
runner.sync_run(|config| cmd
-    .run::<HashingFor<Block>, ReclaimHostFunctions>(config)
+    .run_with_spec::<HashingFor<Block>, ReclaimHostFunctions>(Some(config.chain_spec))
)
```

## 4. Maintenance Change
- `pallet-nis` get a `BenchmarkSetup` config item to prepare its
counterparty asset.
- Add percent progress print when running benchmarks.
- Dont immediately exit on benchmark error but try to run as many as
possible and print errors last.

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Liam Aharon <liam.aharon@hotmail.com>
dharjeezy pushed a commit to dharjeezy/polkadot-sdk that referenced this pull request Apr 9, 2024
This MR contains two major changes and some maintenance cleanup.  

## 1. Free Standing Pallet Benchmark Runner

Closes paritytech#3045, depends
on your runtime exposing the `GenesisBuilderApi` (like
paritytech#1492).

Introduces a new binary crate: `frame-omni-bencher`.  
It allows to directly benchmark a WASM blob - without needing a node or
chain spec.

This makes it much easier to generate pallet weights and should allow us
to remove bloaty code from the node.
It should work for all FRAME runtimes that dont use 3rd party host calls
or non `BlakeTwo256` block hashing (basically all polkadot parachains
should work).

It is 100% backwards compatible with the old CLI args, when the `v1`
compatibility command is used. This is done to allow for forwards
compatible addition of new commands.

### Example (full example in the Rust docs)

Installing the CLI:
```sh
cargo install --locked --path substrate/utils/frame/omni-bencher
frame-omni-bencher --help
```

Building the Westend runtime:
```sh
cargo build -p westend-runtime --release --features runtime-benchmarks
```

Benchmarking the runtime:
```sh
frame-omni-bencher v1 benchmark pallet --runtime target/release/wbuild/westend-runtime/westend_runtime.compact.compressed.wasm --all
```

## 2. Building the Benchmark Genesis State in the Runtime

Closes paritytech#2664

This adds `--runtime` and `--genesis-builder=none|runtime|spec`
arguments to the `benchmark pallet` command to make it possible to
generate the genesis storage by the runtime. This can be used with both
the node and the freestanding benchmark runners. It utilizes the new
`GenesisBuilder` RA and depends on having
paritytech#3412 deployed.

## 3. Simpler args for `PalletCmd::run`

You can do three things here to integrate the changes into your node:
- nothing: old code keeps working as before but emits a deprecated
warning
- delete: remove the pallet benchmarking code from your node and use the
omni-bencher instead
- patch: apply the patch below and keep using as currently. This emits a
deprecated warning at runtime, since it uses the old way to generate a
genesis state, but is the smallest change.

```patch
runner.sync_run(|config| cmd
-    .run::<HashingFor<Block>, ReclaimHostFunctions>(config)
+    .run_with_spec::<HashingFor<Block>, ReclaimHostFunctions>(Some(config.chain_spec))
)
```

## 4. Maintenance Change
- `pallet-nis` get a `BenchmarkSetup` config item to prepare its
counterparty asset.
- Add percent progress print when running benchmarks.
- Dont immediately exit on benchmark error but try to run as many as
possible and print errors last.

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Liam Aharon <liam.aharon@hotmail.com>
@girazoki girazoki mentioned this pull request Jun 7, 2024
4 tasks
enddynayn added a commit to frequency-chain/frequency that referenced this pull request Aug 12, 2024
- Upgrade Polkadot-sdk 1.10.0 to 1.11.0
- Update weights to reflect the new version.

Notable Changes:
- [Apply patch `run_with_spec`](paritytech/polkadot-sdk#3512)
- [Genesis Presets](paritytech/polkadot-sdk#2714)
- [Integrate LiteP2P](paritytech/polkadot-sdk#2944)
- [Template Simplification](https://github.com/paritytech/polkadot-sdk/pull/3801/files)

For more details, please refer to:

[Release Notes](https://github.com/paritytech/polkadot-sdk/releases/tag/polkadot-v1.11.0)

issue-1957
enddynayn added a commit to frequency-chain/frequency that referenced this pull request Aug 13, 2024
- Upgrade Polkadot-sdk 1.10.0 to 1.11.0
- Update weights to reflect the new version.

Notable Changes:
- [Apply patch `run_with_spec`](paritytech/polkadot-sdk#3512)
- [Genesis Presets](paritytech/polkadot-sdk#2714)
- [Integrate LiteP2P](paritytech/polkadot-sdk#2944)
- [Template Simplification](https://github.com/paritytech/polkadot-sdk/pull/3801/files)

For more details, please refer to:

[Release Notes](https://github.com/paritytech/polkadot-sdk/releases/tag/polkadot-v1.11.0)

issue-1957
enddynayn added a commit to frequency-chain/frequency that referenced this pull request Aug 13, 2024
- Upgrade Polkadot-sdk 1.10.0 to 1.11.0
- Update weights to reflect the new version.

Notable Changes:
- [Apply patch `run_with_spec`](paritytech/polkadot-sdk#3512)
- [Genesis Presets](paritytech/polkadot-sdk#2714)
- [Integrate LiteP2P](paritytech/polkadot-sdk#2944)
- [Template Simplification](https://github.com/paritytech/polkadot-sdk/pull/3801/files)

For more details, please refer to:

[Release Notes](https://github.com/paritytech/polkadot-sdk/releases/tag/polkadot-v1.11.0)

issue-1957
enddynayn added a commit to frequency-chain/frequency that referenced this pull request Aug 13, 2024
- Upgrade Polkadot-sdk 1.10.0 to 1.11.0
- Update weights to reflect the new version.

Notable Changes:
- [Apply patch `run_with_spec`](paritytech/polkadot-sdk#3512)
- [Genesis Presets](paritytech/polkadot-sdk#2714)
- [Integrate LiteP2P](paritytech/polkadot-sdk#2944)
- [Template Simplification](https://github.com/paritytech/polkadot-sdk/pull/3801/files)

For more details, please refer to:

[Release Notes](https://github.com/paritytech/polkadot-sdk/releases/tag/polkadot-v1.11.0)

issue-1957
enddynayn added a commit to frequency-chain/frequency that referenced this pull request Aug 13, 2024
- Upgrade Polkadot-sdk 1.10.0 to 1.11.0
- Update weights to reflect the new version.

Notable Changes:
- [Apply patch
`run_with_spec`](paritytech/polkadot-sdk#3512)
- [Genesis
Presets](paritytech/polkadot-sdk#2714)
- [Integrate
LiteP2P](paritytech/polkadot-sdk#2944)
- [Template
Simplification](https://github.com/paritytech/polkadot-sdk/pull/3801/files)
- [Remove
try-runtime-cli](https://github.com/paritytech/polkadot-sdk/pull/4017/files#diff-3a3aa5e088741f8ab1ca38fbe314c7a150d18b7466426f4ad2569113017e8439)

For more details, please refer to:

[Release
Notes](https://github.com/paritytech/polkadot-sdk/releases/tag/polkadot-v1.11.0)

#1957
aramikm pushed a commit to frequency-chain/frequency that referenced this pull request Aug 15, 2024
- Upgrade Polkadot-sdk 1.10.0 to 1.11.0
- Update weights to reflect the new version.

Notable Changes:
- [Apply patch
`run_with_spec`](paritytech/polkadot-sdk#3512)
- [Genesis
Presets](paritytech/polkadot-sdk#2714)
- [Integrate
LiteP2P](paritytech/polkadot-sdk#2944)
- [Template
Simplification](https://github.com/paritytech/polkadot-sdk/pull/3801/files)
- [Remove
try-runtime-cli](https://github.com/paritytech/polkadot-sdk/pull/4017/files#diff-3a3aa5e088741f8ab1ca38fbe314c7a150d18b7466426f4ad2569113017e8439)

For more details, please refer to:

[Release
Notes](https://github.com/paritytech/polkadot-sdk/releases/tag/polkadot-v1.11.0)
rustadot pushed a commit to rustadot/recurrency that referenced this pull request Sep 5, 2024
- Upgrade Polkadot-sdk 1.10.0 to 1.11.0
- Update weights to reflect the new version.

Notable Changes:
- [Apply patch
`run_with_spec`](paritytech/polkadot-sdk#3512)
- [Genesis
Presets](paritytech/polkadot-sdk#2714)
- [Integrate
LiteP2P](paritytech/polkadot-sdk#2944)
- [Template
Simplification](https://github.com/paritytech/polkadot-sdk/pull/3801/files)
- [Remove
try-runtime-cli](https://github.com/paritytech/polkadot-sdk/pull/4017/files#diff-3a3aa5e088741f8ab1ca38fbe314c7a150d18b7466426f4ad2569113017e8439)

For more details, please refer to:

[Release
Notes](https://github.com/paritytech/polkadot-sdk/releases/tag/polkadot-v1.11.0)

#1957
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
T12-benchmarks This PR/Issue is related to benchmarking and weights.
Projects
None yet
7 participants