Skip to content
This repository has been archived by the owner on Nov 15, 2023. It is now read-only.

Benchmarking also benchmark for decoding the call #9343

Merged
10 commits merged into from
Jul 22, 2021
Merged

Conversation

gui1117
Copy link
Contributor

@gui1117 gui1117 commented Jul 13, 2021

in the benchmarking macro, when user use the syntax: bench_name { $code }: $dispatch_name($origin, $args) verify { $code }, the benchmark only dispatch the call. This PR changes it, instead the benchmark decode the call and dispatch it.

Breaking change

Some benchmark code won't compile complaining about partially moved variables. It should be generally easily fixable using clone.

@github-actions github-actions bot added the A3-in_progress Pull request is in progress. No review needed at this stage. label Jul 13, 2021
@kianenigma
Copy link
Contributor

Sensible to have it IMO 👍🏻

@shawntabrizi
Copy link
Member

I agree, if we are finding this is more and more relevant, we should definitely include it.

How can we add a test so that we know this is working as expected?

@emostov
Copy link
Contributor

emostov commented Jul 13, 2021

I agree, if we are finding this is more and more relevant, we should definitely include it.

Was there any specific rational for not including it before?

From my limited understanding it makes sense to always do this iff when processing the extrinsics in a block we always have to decode each one.

@gui1117 gui1117 marked this pull request as ready for review July 14, 2021 07:43
@gui1117 gui1117 added A0-please_review Pull request needs code review. B3-apinoteworthy C1-low PR touches the given topic and has a low impact on builders. D5-nicetohaveaudit ⚠️ PR contains trivial changes to logic that should be properly reviewed. and removed A3-in_progress Pull request is in progress. No review needed at this stage. labels Jul 14, 2021
@gui1117
Copy link
Contributor Author

gui1117 commented Jul 14, 2021

/benchmark runtime pallet pallet_lottery

@parity-benchapp
Copy link

parity-benchapp bot commented Jul 14, 2021

Benchmark Runtime Pallet for branch "gui-benchmark-decoding" with command cargo run --quiet --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_lottery --extrinsic="*" --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/lottery/src/weights.rs --template=./.maintain/frame-weight-template.hbs

Results
error[E0382]: use of partially moved value: `instance`
    --> frame/contracts/src/benchmarking/mod.rs:280:1
     |
280  | / benchmarks! {
281  | |     where_clause { where
282  | |         T::AccountId: UncheckedFrom<T::Hash>,
283  | |         T::AccountId: AsRef<[u8]>,
...    |
730  | |     }: call(origin, instance.addr, 0u32.into(), Weight::max_value(), vec![])
     | |                     ------------- value partially moved here
...    |
733  | |             assert_eq!(T::Currency::total_balance(&instance.account_id), 0u32.into());
     | |                                                    -------- use occurs due to use in closure
...    |
2532 | |     }: {}
2533 | | }
     | |_^ value used here after partial move
     |
     = note: partial move occurs because `instance.addr` has type `<<T as frame_system::Config>::Lookup as sp_runtime::traits::StaticLookup>::Source`, which does not implement the `Copy` trait
     = note: this error originates in a macro (in Nightly builds, run with -Z macro-backtrace for more info)

error: aborting due to previous error

For more information about this error, try `rustc --explain E0382`.
error: could not compile `pallet-contracts`

To learn more, run the command again with --verbose.
error: build failed

@gui1117 gui1117 requested a review from athei as a code owner July 14, 2021 07:51
@gui1117
Copy link
Contributor Author

gui1117 commented Jul 14, 2021

/benchmark runtime pallet pallet_lottery

@parity-benchapp
Copy link

parity-benchapp bot commented Jul 14, 2021

Benchmark Runtime Pallet for branch "gui-benchmark-decoding" with command cargo run --quiet --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_lottery --extrinsic="*" --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/lottery/src/weights.rs --template=./.maintain/frame-weight-template.hbs

Results
Pallet: "pallet_lottery", Extrinsic: "buy_ticket", Lowest values: [], Highest values: [], Steps: [50], Repeat: 20
Median Slopes Analysis
========
-- Extrinsic Time --

Model:
Time ~=    74.85
              µs

Reads = 6
Writes = 4
Min Squares Analysis
========
-- Extrinsic Time --

Model:
Time ~=    74.85
              µs

Reads = 6
Writes = 4
Pallet: "pallet_lottery", Extrinsic: "set_calls", Lowest values: [], Highest values: [], Steps: [50], Repeat: 20
Median Slopes Analysis
========
-- Extrinsic Time --

Model:
Time ~=    15.68
    + n    0.264
              µs

Reads = 0 + (0 * n)
Writes = 1 + (0 * n)
Min Squares Analysis
========
-- Extrinsic Time --

Data points distribution:
    n   mean µs  sigma µs       %
    0     14.97     0.059    0.3%
    1     15.95     0.069    0.4%
    2      16.3     0.068    0.4%
    3     16.53     0.072    0.4%
    4     16.84     0.026    0.1%
    5     17.09     0.151    0.8%
    6     17.29     0.067    0.3%
    7     17.54      0.08    0.4%
    8     17.78     0.079    0.4%
    9     18.02     0.086    0.4%
   10     18.12     0.078    0.4%

Quality and confidence:
param     error
n         0.007

Model:
Time ~=    15.54
    + n    0.281
              µs

Reads = 0 + (0 * n)
Writes = 1 + (0 * n)
Pallet: "pallet_lottery", Extrinsic: "start_lottery", Lowest values: [], Highest values: [], Steps: [50], Repeat: 20
Median Slopes Analysis
========
-- Extrinsic Time --

Model:
Time ~=     58.9
              µs

Reads = 3
Writes = 3
Min Squares Analysis
========
-- Extrinsic Time --

Model:
Time ~=     58.9
              µs

Reads = 3
Writes = 3
Pallet: "pallet_lottery", Extrinsic: "stop_repeat", Lowest values: [], Highest values: [], Steps: [50], Repeat: 20
Median Slopes Analysis
========
-- Extrinsic Time --

Model:
Time ~=    7.714
              µs

Reads = 1
Writes = 1
Min Squares Analysis
========
-- Extrinsic Time --

Model:
Time ~=    7.714
              µs

Reads = 1
Writes = 1
Pallet: "pallet_lottery", Extrinsic: "on_initialize_end", Lowest values: [], Highest values: [], Steps: [50], Repeat: 20
Median Slopes Analysis
========
-- Extrinsic Time --

Model:
Time ~=    117.4
              µs

Reads = 6
Writes = 4
Min Squares Analysis
========
-- Extrinsic Time --

Model:
Time ~=    117.4
              µs

Reads = 6
Writes = 4
Pallet: "pallet_lottery", Extrinsic: "on_initialize_repeat", Lowest values: [], Highest values: [], Steps: [50], Repeat: 20
Median Slopes Analysis
========
-- Extrinsic Time --

Model:
Time ~=      123
              µs

Reads = 7
Writes = 5
Min Squares Analysis
========
-- Extrinsic Time --

Model:
Time ~=      123
              µs

Reads = 7
Writes = 5

…path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_lottery --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/lottery/src/weights.rs --template=./.maintain/frame-weight-template.hbs
@gui1117
Copy link
Contributor Author

gui1117 commented Jul 14, 2021

/benchmark runtime pallet pallet_utility

@parity-benchapp
Copy link

parity-benchapp bot commented Jul 14, 2021

Benchmark Runtime Pallet for branch "gui-benchmark-decoding" with command cargo run --quiet --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_utility --extrinsic="*" --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/utility/src/weights.rs --template=./.maintain/frame-weight-template.hbs

Results
Pallet: "pallet_utility", Extrinsic: "batch", Lowest values: [], Highest values: [], Steps: [50], Repeat: 20
Median Slopes Analysis
========
-- Extrinsic Time --

Model:
Time ~=    18.13
    + c    1.082
              µs

Reads = 0 + (0 * c)
Writes = 0 + (0 * c)
Min Squares Analysis
========
-- Extrinsic Time --

Data points distribution:
    c   mean µs  sigma µs       %
    0     13.95     0.103    0.7%
   20     38.18     0.073    0.1%
   40     61.12     0.122    0.1%
   60      79.7     0.082    0.1%
   80     107.4     0.118    0.1%
  100     125.7     0.061    0.0%
  120     144.5     0.082    0.0%
  140     180.8     0.137    0.0%
  160     199.1       0.1    0.0%
  180     217.6     0.128    0.0%
  200     235.9     0.082    0.0%
  220     254.5      0.07    0.0%
  240       273     0.101    0.0%
  260     326.6     0.201    0.0%
  280     344.9     0.081    0.0%
  300     363.5     0.185    0.0%
  320     363.8     0.216    0.0%
  340     382.5      0.16    0.0%
  360     400.7     0.185    0.0%
  380     419.3     0.147    0.0%
  400       490     0.201    0.0%
  420     473.2     0.296    0.0%
  440     491.8     0.231    0.0%
  460       510     0.225    0.0%
  480     528.8     0.437    0.0%
  500     549.4     4.822    0.8%
  520     582.7      0.22    0.0%
  540     600.7     0.227    0.0%
  560     619.5     0.249    0.0%
  580     637.6      0.47    0.0%
  600       658     5.849    0.8%
  620     694.1      0.23    0.0%
  640       712     0.379    0.0%
  660     730.8     0.332    0.0%
  680       749     0.212    0.0%
  700     770.7     8.672    1.1%
  720     803.7     0.238    0.0%
  740     822.2     0.725    0.0%
  760     842.5     4.635    0.5%
  780     862.3     8.511    0.9%
  800     876.7     0.858    0.0%
  820     912.2     0.897    0.0%
  840     930.5     1.239    0.1%
  860     954.9     10.99    1.1%
  880     967.5      1.16    0.1%
  900     986.4     0.411    0.0%
  920      1028     10.92    1.0%
  940      1041     4.681    0.4%
  960      1058     0.364    0.0%
  980      1076     0.889    0.0%
 1000      1100     10.12    0.9%

Quality and confidence:
param     error
c         0.001

Model:
Time ~=    20.77
    + c     1.08
              µs

Reads = 0 + (0 * c)
Writes = 0 + (0 * c)
Pallet: "pallet_utility", Extrinsic: "as_derivative", Lowest values: [], Highest values: [], Steps: [50], Repeat: 20
Median Slopes Analysis
========
-- Extrinsic Time --

Model:
Time ~=    3.994
              µs

Reads = 0
Writes = 0
Min Squares Analysis
========
-- Extrinsic Time --

Model:
Time ~=    3.994
              µs

Reads = 0
Writes = 0
Pallet: "pallet_utility", Extrinsic: "batch_all", Lowest values: [], Highest values: [], Steps: [50], Repeat: 20
Median Slopes Analysis
========
-- Extrinsic Time --

Model:
Time ~=    19.16
    + c    1.508
              µs

Reads = 0 + (0 * c)
Writes = 0 + (0 * c)
Min Squares Analysis
========
-- Extrinsic Time --

Data points distribution:
    c   mean µs  sigma µs       %
    0      14.9     0.121    0.8%
   20     47.64     0.082    0.1%
   40      79.3     0.099    0.1%
   60     106.2     0.115    0.1%
   80     142.3     0.066    0.0%
  100     169.3     0.067    0.0%
  120     196.6     0.102    0.0%
  140     241.4     0.093    0.0%
  160     268.2      0.16    0.0%
  180     295.1     0.211    0.0%
  200     321.9     0.129    0.0%
  220       349     0.158    0.0%
  240     375.9     0.293    0.0%
  260       438     0.156    0.0%
  280     464.9     0.175    0.0%
  300     491.6     0.153    0.0%
  320       501      0.22    0.0%
  340       529     3.714    0.7%
  360     554.6     0.292    0.0%
  380     581.8     0.834    0.1%
  400       664     8.652    1.3%
  420     652.6      0.21    0.0%
  440     681.3     0.882    0.1%
  460     708.2     0.605    0.0%
  480     735.4     0.372    0.0%
  500     772.1     11.78    1.5%
  520     805.8     1.286    0.1%
  540     833.2     0.856    0.1%
  560     858.9     1.214    0.1%
  580     894.5     12.56    1.4%
  600     914.1     0.856    0.0%
  620     958.7     1.008    0.1%
  640     985.1     0.906    0.0%
  660      1021     12.78    1.2%
  680      1038      1.14    0.1%
  700      1067      5.12    0.4%
  720      1122     13.91    1.2%
  740      1139     8.026    0.7%
  760      1162     1.042    0.0%
  780      1190      0.95    0.0%
  800      1220     7.576    0.6%
  820      1269     10.84    0.8%
  840      1291      7.77    0.6%
  860      1315     1.298    0.0%
  880      1342     0.669    0.0%
  900      1373     9.434    0.6%
  920      1412     1.016    0.0%
  940      1441     4.249    0.2%
  960      1474     11.18    0.7%
  980      1494     1.468    0.0%
 1000      1527     9.688    0.6%

Quality and confidence:
param     error
c         0.001

Model:
Time ~=    22.18
    + c    1.506
              µs

Reads = 0 + (0 * c)
Writes = 0 + (0 * c)

…path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_utility --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/utility/src/weights.rs --template=./.maintain/frame-weight-template.hbs
Copy link
Contributor

@emostov emostov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My macro literacy is not great, but from what I understand this makes sense.

Should we re-benchmark everything (including in polkadot) so when we make logic changes in the future they are based off a baseline that used the same benchmarking formula? Otherwise I would think all future benchmarks would be a bit high.

@parity-benchapp
Copy link

parity-benchapp bot commented Jul 20, 2021

Benchmark Runtime Pallet for branch "gui-benchmark-decoding" with command cargo run --quiet --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_mmr --extrinsic="*" --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/mmr/src/weights.rs --template=./.maintain/frame-weight-template.hbs

Results
error[E0433]: failed to resolve: use of undeclared crate or module `frame_system`
  --> frame/system/src/weights.rs:58:9
   |
58 | impl<T: frame_system::Config> WeightInfo for SubstrateWeight<T> {
   |         ^^^^^^^^^^^^ use of undeclared crate or module `frame_system`

error[E0433]: failed to resolve: use of undeclared crate or module `frame_system`
  --> frame/system/src/weights.rs:58:9
   |
58 | impl<T: frame_system::Config> WeightInfo for SubstrateWeight<T> {
   |         ^^^^^^^^^^^^ use of undeclared crate or module `frame_system`

error: aborting due to previous error

error: aborting due to previous error

For more information about this error, try `rustc --explain E0433`.
For more information about this error, try `rustc --explain E0433`.
error: could not compile `frame-system`

To learn more, run the command again with --verbose.
error: build failed

@gui1117
Copy link
Contributor Author

gui1117 commented Jul 20, 2021

/benchmark runtime pallet pallet_lottery

@parity-benchapp
Copy link

parity-benchapp bot commented Jul 20, 2021

Benchmark Runtime Pallet for branch "gui-benchmark-decoding" with command cargo run --quiet --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_lottery --extrinsic="*" --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/lottery/src/weights.rs --template=./.maintain/frame-weight-template.hbs

Results
error[E0433]: failed to resolve: use of undeclared crate or module `frame_system`
  --> frame/system/src/weights.rs:58:9
   |
58 | impl<T: frame_system::Config> WeightInfo for SubstrateWeight<T> {
   |         ^^^^^^^^^^^^ use of undeclared crate or module `frame_system`

error[E0433]: failed to resolve: use of undeclared crate or module `frame_system`
  --> frame/system/src/weights.rs:58:9
   |
58 | impl<T: frame_system::Config> WeightInfo for SubstrateWeight<T> {
   |         ^^^^^^^^^^^^ use of undeclared crate or module `frame_system`

error: aborting due to previous error

For more information about this error, try `rustc --explain E0433`.
error: aborting due to previous error

For more information about this error, try `rustc --explain E0433`.
error: could not compile `frame-system`

To learn more, run the command again with --verbose.
error: build failed

@gui1117
Copy link
Contributor Author

gui1117 commented Jul 20, 2021

/benchmark runtime pallet pallet_gilt

@parity-benchapp
Copy link

parity-benchapp bot commented Jul 20, 2021

Benchmark Runtime Pallet for branch "gui-benchmark-decoding" with command cargo run --quiet --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_gilt --extrinsic="*" --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/gilt/src/weights.rs --template=./.maintain/frame-weight-template.hbs

Results
error[E0433]: failed to resolve: use of undeclared crate or module `frame_system`
  --> frame/system/src/weights.rs:58:9
   |
58 | impl<T: frame_system::Config> WeightInfo for SubstrateWeight<T> {
   |         ^^^^^^^^^^^^ use of undeclared crate or module `frame_system`

error[E0433]: failed to resolve: use of undeclared crate or module `frame_system`
  --> frame/system/src/weights.rs:58:9
   |
58 | impl<T: frame_system::Config> WeightInfo for SubstrateWeight<T> {
   |         ^^^^^^^^^^^^ use of undeclared crate or module `frame_system`

error: aborting due to previous error

error: aborting due to previous error

For more information about this error, try `rustc --explain E0433`.
For more information about this error, try `rustc --explain E0433`.
error: could not compile `frame-system`

To learn more, run the command again with --verbose.
error: build failed

@gui1117
Copy link
Contributor Author

gui1117 commented Jul 20, 2021

/benchmark runtime pallet pallet_uniques

@parity-benchapp
Copy link

parity-benchapp bot commented Jul 20, 2021

Benchmark Runtime Pallet for branch "gui-benchmark-decoding" with command cargo run --quiet --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_uniques --extrinsic="*" --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/uniques/src/weights.rs --template=./.maintain/frame-weight-template.hbs

Results
error[E0433]: failed to resolve: use of undeclared crate or module `frame_system`
  --> frame/system/src/weights.rs:58:9
   |
58 | impl<T: frame_system::Config> WeightInfo for SubstrateWeight<T> {
   |         ^^^^^^^^^^^^ use of undeclared crate or module `frame_system`

error[E0433]: failed to resolve: use of undeclared crate or module `frame_system`
  --> frame/system/src/weights.rs:58:9
   |
58 | impl<T: frame_system::Config> WeightInfo for SubstrateWeight<T> {
   |         ^^^^^^^^^^^^ use of undeclared crate or module `frame_system`

error: aborting due to previous error

For more information about this error, try `rustc --explain E0433`.
error: could not compile `frame-system`

To learn more, run the command again with --verbose.
error: aborting due to previous error

For more information about this error, try `rustc --explain E0433`.
error: build failed

@gui1117
Copy link
Contributor Author

gui1117 commented Jul 20, 2021

/benchmark runtime pallet pallet_transaction_storage

@parity-benchapp
Copy link

parity-benchapp bot commented Jul 20, 2021

Benchmark Runtime Pallet for branch "gui-benchmark-decoding" with command cargo run --quiet --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_transaction_storage --extrinsic="*" --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/transaction-storage/src/weights.rs --template=./.maintain/frame-weight-template.hbs

Results
error[E0433]: failed to resolve: use of undeclared crate or module `frame_system`
  --> frame/system/src/weights.rs:58:9
   |
58 | impl<T: frame_system::Config> WeightInfo for SubstrateWeight<T> {
   |         ^^^^^^^^^^^^ use of undeclared crate or module `frame_system`

error[E0433]: failed to resolve: use of undeclared crate or module `frame_system`
  --> frame/system/src/weights.rs:58:9
   |
58 | impl<T: frame_system::Config> WeightInfo for SubstrateWeight<T> {
   |         ^^^^^^^^^^^^ use of undeclared crate or module `frame_system`

error: aborting due to previous error

error: aborting due to previous error

For more information about this error, try `rustc --explain E0433`.
For more information about this error, try `rustc --explain E0433`.
error: could not compile `frame-system`

To learn more, run the command again with --verbose.
error: build failed

…path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=frame_system --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/system/src/weights.rs --template=./.maintain/frame-weight-template.hbs
@shawntabrizi
Copy link
Member

bot merge

@ghost
Copy link

ghost commented Jul 21, 2021

Waiting for commit status.

@ghost
Copy link

ghost commented Jul 21, 2021

Merge failed: "Required status check "continuous-integration/gitlab-check-polkadot-companion-build" is failing."

.saturating_add(T::DbWeight::get().writes((1 as Weight).saturating_mul(p as Weight)))
}
}

// For backwards compatibility and tests
impl WeightInfo for () {
fn remark(_b: u32, ) -> Weight {
(1_038_000 as Weight)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

interesting the weight is now only factor of message length

@gui1117
Copy link
Contributor Author

gui1117 commented Jul 21, 2021

polkadot build will be fixed by paritytech/polkadot#3507

Copy link
Contributor

@kianenigma kianenigma left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@gui1117
Copy link
Contributor Author

gui1117 commented Jul 22, 2021

bot merge

@ghost
Copy link

ghost commented Jul 22, 2021

Trying merge.

@gui1117
Copy link
Contributor Author

gui1117 commented Jul 22, 2021

bot merge

@ghost
Copy link

ghost commented Jul 22, 2021

Trying merge.

This pull request was closed.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
A0-please_review Pull request needs code review. C1-low PR touches the given topic and has a low impact on builders. D5-nicetohaveaudit ⚠️ PR contains trivial changes to logic that should be properly reviewed.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants