Benchmarking Roadmap 2022 (Tracking Issue)

[shawntabrizi]

In 2022, we should prioritize the next wave of benchmarking and optimization for Substrate / Polkadot.

Goals

• improve transaction density / throughput in blocks.
• improve developer quality of life for benchmarks.
• improve story specifically in the context of weights for parachains.

Developer Quality of Life

• Migrate existing benchmarks! macro to attribute macros substrate#10848
• Improve list_benchmark, add_benchmark to remove redundancy and manual management (maybe a meta macro for the definition) in Improve usability of add+list_benchmark! substrate#10592
• Add a flag to FRAME storage for "keeping a storage in memory" from block to block, automatically populates whitelist, and remove the need for manual whitelist declaration.
• Better handle pallets with multiple instances and how we run and output their files.

Improve Throughput

• Switch from RocksDB to ParityDB (Q1 2022)
• Reassess previous assumptions (block overhead, transaction overhead, etc...)
• New reference hardware for benchmarking the runtime. Update reference hardware w3f/polkadot-wiki#3564
  • Rerun benchmarks for this hardware.
  • Migration plan for community to switch to this hardware.
• Investigate the use of OS specific optimized binary
• Look at existing performance / benchmarking tools in the RUST ecosystem

Improve Accuracy

• Complete work for PoV benchmarking. PoV Benchmarking Tracking Issue #398
• Track memory read / writes in addition to DB read / writes.
• Dual weight limits for Collators on Parachain Network and Validators on Polkadot Network
  • Remove DB reads / writes for validator blocks (use memory reads / writes instead)
• Dynamic Storage Weights (measuring DB reads / writes impact dynamically based on worst case encoding size)

[vedhavyas]

@shawntabrizi checking if there has been any change to reference hardware used to benchmark the runtime is still the same - w3f/polkadot-wiki#3564

if not, can you point me to the latest reference pls?

[shawntabrizi]

@ggwpez should know

[vedhavyas]

@ggwpez wondering if you can confirm the current hardware reference used for benchmarking runtime

[ggwpez]

It is described in the polkadot-wiki.
For updating the weights we use some Google server with Intel Ice Lake CPU series. The weights come in very consistent so far.