contracts: Simplify benchmarks #13595
Conversation
|
bot bench $ pallet dev pallet_contracts |
|
Looks good so far, going to re-run two more times to be sure |
|
bot bench $ pallet dev pallet_contracts |
|
bot clean |
|
@ggwpez Command |
|
@ggwpez Command |
|
Last two changes look clean as well
|
|
Nice. Looks like a clean boy. I think we can go ahead with this PR. |
athei
added
A0-please_review
| @@ -47,10 +46,10 @@ use sp_std::prelude::*; | |||
| use wasm_instrument::parity_wasm::elements::{BlockType, BrTableData, Instruction, ValueType}; | |||
|
|
|||
| /// How many batches we do per API benchmark. | |||
| const API_BENCHMARK_BATCHES: u32 = 20; | |||
| const API_BENCHMARK_BATCHES: u32 = 1600; | |||
There was a problem hiding this comment.
Could mention in the comment why these specific values were picked
|
bot merge |
|
Waiting for commit status. |
| /// This is picked more or less arbitrary. We experimented with different numbers until | ||
| /// the results appeared to be stable. Reducing the number would speed up the benchmarks | ||
| /// but might make the results less precise. | ||
| const API_BENCHMARK_BATCHES: u32 = 1600; |
There was a problem hiding this comment.
Strictly speaking, this is not really batches anymore, is it?
| const API_BENCHMARK_BATCHES: u32 = 1600; | |
| const API_BENCHMARK_RUNS: u32 = 1600; |
* Remove batching * Benchmark in bytes not kilobytes * ".git/.scripts/commands/bench/bench.sh" pallet dev pallet_contracts * ".git/.scripts/commands/bench/bench.sh" pallet dev pallet_contracts * ".git/.scripts/commands/bench/bench.sh" pallet dev pallet_contracts * ".git/.scripts/commands/bench/bench.sh" pallet dev pallet_contracts * Add rationale for picking batch numbers --------- Co-authored-by: command-bot <>
* Remove batching * Benchmark in bytes not kilobytes * ".git/.scripts/commands/bench/bench.sh" pallet dev pallet_contracts * ".git/.scripts/commands/bench/bench.sh" pallet dev pallet_contracts * ".git/.scripts/commands/bench/bench.sh" pallet dev pallet_contracts * ".git/.scripts/commands/bench/bench.sh" pallet dev pallet_contracts * Add rationale for picking batch numbers --------- Co-authored-by: command-bot <>
Inspired by #13581 I reworked the contract benchmarks to be much simpler and hopefully work better with the linear regression algorithm:
rcomponent is now exactly one iteration.*_per_bytebenchmarks (which don't haver) just perform their operation once. I increased the upper limit forrto compensate for the removed batching.I implemented those strategies when writing those benchmarks years ago because back then they seemed to improve the stability of results. I hope they are no longer necessary or may even be harmful.
cc @ggwpez