new proc-macro-based benchmarking syntax (paritytech#12924)

* add stub for new benchmark macro

* benchmark syntax

* add #[extrinsic call] separator

* parse #[benchmark] item as a function

* proper emission of error when #[extrinsic_call] annotation is missing

* clean up

* enclosing module via benchmarks! { } working

* use an attribute macro on the module instead of benchmarks! { }

* cargo fmt

* working component implementation

* WIP

* working

* add syntax for Linear<A, B>

* parsing of param ranges (still need to build tuple though)

* params parsing WIP

* clean up (don't need extrinsic call name)

* use proper Result syntax for BenchmarkDef parsing

* proper parsing of Linear<0, 1> style args

* successfully parse and make use of linear component ranges 💥

* rename support variable => home because eventually will be moved

* compile-time check that param range types implement ParamRange

* switch to using balances as example, failing on instance pallet

* successfully set up __origin and __call with balances 💥

* clean up

* use a module

* don't need a variable for transfer

* rename benchmark_transfer -> transfer because no longer conflicts

* clean up

* working with transfer_increasing_users as well 💥

* re-add BareBlock

* add comments for undocumented structs+functions+traits

* refactor in preparation for removing module requirements

* switch to a block instead of a module

* use the outer macro pattern to to enable #[benchmarks] aggregation

* successfully generate SelectedBenchmark 💥

* implement components for SelectedBenchmark

* implement instance for SelectedBenchmark

* properly track #[extra]

* working impl for fn benchmarks()

* run_benchmarks WIP

* finish run_benchmark! impl 💥

* import balances transfer_best_case benchmark

* import transfer_keep_alive balances pallet benchmark

* import set_balance_creating balances pallet benchmark

* import set_balance_killing balances pallet benchmark

* import force_transfer balances pallet benchmark

* add #[extra] annotation and docs to transfer_increasing_users

* import transfer_all balances pallet benchmark

* import force_unreserve balances pallet benchmark

* prepare to implement impl_benchmark_test_suite!

* ensure tests cover #[extra] before and after #[benchmark] tag

* refactor

* clean up

* fix

* move to outer

* switch to benchmarks/instance_benchmarks

* test impl almost done, strange compiler error

* benchmark test suites working 💥

* clean up

* add stub and basic parsing for where_clause

* working except where clause and extrinsic calls containing method chains

* assume option (2) for now wrt paritytech#12924 (comment)

* clean up

* switch to attribute-style

* properly handle where clauses

* fix subtle missing where clause, now just MessageQueue issues

* fix block formatting in message-queue pallet

* switch to block vs non-block parsing of extrinsic call

* working now but some benchmark tests failing

* message-queue tests working (run order issue fixed) 🎉

* add comments and internal docs for fame_support_procedural::benchmark

* fix license years

* docs for lib.rs

* add docs to new support procedural macros

* don't allow #[benchmark] outside of benchmarking module

* add docs

* use benchmark(extra, skip_meta) style args

* update docs accordingly

* appease clippy

* bump ci

* add notes about `extra` and `skip_meta`

* fix doc tests

* re-run CI

* use `ignore` instead of `no_run` on doc examples

* bump CI

* replace some if-lets with if-elses

* more refactoring of if-let statements

* fix remaining if-lets in BenchmarkDef::from()

* fix if-lets in benchmarks()

* fix remaining if-lets, use nested find_map for extrinsic call

* switch to use #[extrinsic_call] or #[block] situationally

* refactor ExtrinsicCallDef => BenchmarkCallDef

* update docs with info about #[block]

* add macro stub for #[extrinsic_call]

* fix docs and add stub for #[block] as well

* remove unused extern crate line

* fix clippy nits

* Use V2 bench syntax in pallet-example-basic

Just testing the dev-ex...

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* carry over comment

* use curly-brace style for impl_benchmark_test_suite!

* remove unneeded parenthesis

* proper handling of _() extrinsic call style

* add docs for _() syntax

* fix crate access

* simplify keyword access

Co-authored-by: Keith Yeung <kungfukeith11@gmail.com>

* simplify module content destructuring

Co-authored-by: Keith Yeung <kungfukeith11@gmail.com>

* fix crate access "frame_benchmarking" => "frame-benchmarking", compiles

* use _() extrinsic call syntax where possible in balances

* simplify attr.path.segments.last()

Co-authored-by: Keith Yeung <kungfukeith11@gmail.com>

* fix compile error being suppressed

* simplify extrinsic call keyword parsing

Co-authored-by: Keith Yeung <kungfukeith11@gmail.com>

* use ? operator instead of return None

Co-authored-by: Keith Yeung <kungfukeith11@gmail.com>

* rename generics => type_use_generics
rename full_generics => type_impl_generics

* simplify extrinsic call extraction with transpose

* bump CI

* nit

* proper handling of too many + too few block/extrinsic call annotations

* change to B >= A

Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* remove unneeded ignore

Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* remove another ignore

Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* add ui tests

* use _() style extrinsic call on accumulate_dummy

Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* add range check to ParamRange

* ui test for bad param ranges

* fix failing example

* add ignore back to other failing example

* tweak expr_call span

Co-authored-by: Keith Yeung <kungfukeith11@gmail.com>

* fix typo

* eliminate a match

Co-authored-by: Keith Yeung <kungfukeith11@gmail.com>

* change pub fn benchmarks to return Result<TokenStream>

* fix origin error span

* more informative error for invalid benchmark parameter name

* fix spans on a few benchmark errors

* remove unneeded clone

* refactor inner loop of benchmark function parsing

* preserve mod attributes

* refactor outer loop of benchmark def parsing code, greatly simplified

* simplify to use a ? operator when parsing benchmark attr path

* fix another ? operator

* further simplify benchmark function attr parsing with more ? ops

* refactor extrinsic call handling to use if let rather than match

* replace is_ok => is_err

Co-authored-by: Keith Yeung <kungfukeith11@gmail.com>

* re-use name during expansion of benchmark def

* remove unneeded clone

* fix span for origin missing error

* fix missing semi

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Keith Yeung <kungfukeith11@gmail.com>
Co-authored-by: parity-processbot <>

frame/balances/src/benchmarking.rs

@@ -1,6 +1,6 @@
 // This file is part of Substrate.
 
-// Copyright (C) 2020-2022 Parity Technologies (UK) Ltd.
+// Copyright (C) 2020-2023 Parity Technologies (UK) Ltd.
 // SPDX-License-Identifier: Apache-2.0
 
 // Licensed under the Apache License, Version 2.0 (the "License");
@@ -20,22 +20,25 @@
 #![cfg(feature = "runtime-benchmarks")]
 
 use super::*;
+use crate::Pallet as Balances;
 
-use frame_benchmarking::{account, benchmarks_instance_pallet, whitelisted_caller};
+use frame_benchmarking::{account, impl_benchmark_test_suite, whitelisted_caller};
+use frame_support::benchmarking::*;
 use frame_system::RawOrigin;
-use sp_runtime::traits::Bounded;
-
-use crate::Pallet as Balances;
 
 const SEED: u32 = 0;
 // existential deposit multiplier
 const ED_MULTIPLIER: u32 = 10;
 
-benchmarks_instance_pallet! {
+#[instance_benchmarks]
+mod benchmarks {
+ use super::*;
+
  // Benchmark `transfer` extrinsic with the worst possible conditions:
  // * Transfer will kill the sender account.
  // * Transfer will create the recipient account.
- transfer {
+ #[benchmark]
+ fn transfer() {
  let existential_deposit = T::ExistentialDeposit::get();
  let caller = whitelisted_caller();
 
@@ -47,84 +50,103 @@ benchmarks_instance_pallet! {
  // and reap this user.
  let recipient: T::AccountId = account("recipient", 0, SEED);
  let recipient_lookup = T::Lookup::unlookup(recipient.clone());
- let transfer_amount = existential_deposit.saturating_mul((ED_MULTIPLIER - 1).into()) + 1u32.into();
- }: transfer(RawOrigin::Signed(caller.clone()), recipient_lookup, transfer_amount)
- verify {
+ let transfer_amount =
+ existential_deposit.saturating_mul((ED_MULTIPLIER - 1).into()) + 1u32.into();
+
+ #[extrinsic_call]
+ _(RawOrigin::Signed(caller.clone()), recipient_lookup, transfer_amount);
+
  assert_eq!(Balances::<T, I>::free_balance(&caller), Zero::zero());
  assert_eq!(Balances::<T, I>::free_balance(&recipient), transfer_amount);
  }
 
  // Benchmark `transfer` with the best possible condition:
  // * Both accounts exist and will continue to exist.
- #[extra]
- transfer_best_case {
+ #[benchmark(extra)]
+ fn transfer_best_case() {
  let caller = whitelisted_caller();
  let recipient: T::AccountId = account("recipient", 0, SEED);
  let recipient_lookup = T::Lookup::unlookup(recipient.clone());
 
- // Give the sender account max funds for transfer (their account will never reasonably be killed).
- let _ = <Balances<T, I> as Currency<_>>::make_free_balance_be(&caller, T::Balance::max_value());
+ // Give the sender account max funds for transfer (their account will never reasonably be
+ // killed).
+ let _ =
+ <Balances<T, I> as Currency<_>>::make_free_balance_be(&caller, T::Balance::max_value());
 
  // Give the recipient account existential deposit (thus their account already exists).
  let existential_deposit = T::ExistentialDeposit::get();
- let _ = <Balances<T, I> as Currency<_>>::make_free_balance_be(&recipient, existential_deposit);
+ let _ =
+ <Balances<T, I> as Currency<_>>::make_free_balance_be(&recipient, existential_deposit);
  let transfer_amount = existential_deposit.saturating_mul(ED_MULTIPLIER.into());
- }: transfer(RawOrigin::Signed(caller.clone()), recipient_lookup, transfer_amount)
- verify {
+
+ #[extrinsic_call]
+ transfer(RawOrigin::Signed(caller.clone()), recipient_lookup, transfer_amount);
+
  assert!(!Balances::<T, I>::free_balance(&caller).is_zero());
  assert!(!Balances::<T, I>::free_balance(&recipient).is_zero());
  }
 
  // Benchmark `transfer_keep_alive` with the worst possible condition:
  // * The recipient account is created.
- transfer_keep_alive {
+ #[benchmark]
+ fn transfer_keep_alive() {
  let caller = whitelisted_caller();
  let recipient: T::AccountId = account("recipient", 0, SEED);
  let recipient_lookup = T::Lookup::unlookup(recipient.clone());
 
  // Give the sender account max funds, thus a transfer will not kill account.
- let _ = <Balances<T, I> as Currency<_>>::make_free_balance_be(&caller, T::Balance::max_value());
+ let _ =
+ <Balances<T, I> as Currency<_>>::make_free_balance_be(&caller, T::Balance::max_value());
  let existential_deposit = T::ExistentialDeposit::get();
  let transfer_amount = existential_deposit.saturating_mul(ED_MULTIPLIER.into());
- }: _(RawOrigin::Signed(caller.clone()), recipient_lookup, transfer_amount)
- verify {
+
+ #[extrinsic_call]
+ _(RawOrigin::Signed(caller.clone()), recipient_lookup, transfer_amount);
+
  assert!(!Balances::<T, I>::free_balance(&caller).is_zero());
  assert_eq!(Balances::<T, I>::free_balance(&recipient), transfer_amount);
  }
 
  // Benchmark `set_balance` coming from ROOT account. This always creates an account.
- set_balance_creating {
+ #[benchmark]
+ fn set_balance_creating() {
  let user: T::AccountId = account("user", 0, SEED);
  let user_lookup = T::Lookup::unlookup(user.clone());
 
  // Give the user some initial balance.
  let existential_deposit = T::ExistentialDeposit::get();
  let balance_amount = existential_deposit.saturating_mul(ED_MULTIPLIER.into());
  let _ = <Balances<T, I> as Currency<_>>::make_free_balance_be(&user, balance_amount);
- }: set_balance(RawOrigin::Root, user_lookup, balance_amount, balance_amount)
- verify {
+
+ #[extrinsic_call]
+ set_balance(RawOrigin::Root, user_lookup, balance_amount, balance_amount);
+
  assert_eq!(Balances::<T, I>::free_balance(&user), balance_amount);
  assert_eq!(Balances::<T, I>::reserved_balance(&user), balance_amount);
  }
 
  // Benchmark `set_balance` coming from ROOT account. This always kills an account.
- set_balance_killing {
+ #[benchmark]
+ fn set_balance_killing() {
  let user: T::AccountId = account("user", 0, SEED);
  let user_lookup = T::Lookup::unlookup(user.clone());
 
  // Give the user some initial balance.
  let existential_deposit = T::ExistentialDeposit::get();
  let balance_amount = existential_deposit.saturating_mul(ED_MULTIPLIER.into());
  let _ = <Balances<T, I> as Currency<_>>::make_free_balance_be(&user, balance_amount);
- }: set_balance(RawOrigin::Root, user_lookup, Zero::zero(), Zero::zero())
- verify {
+
+ #[extrinsic_call]
+ set_balance(RawOrigin::Root, user_lookup, Zero::zero(), Zero::zero());
+
  assert!(Balances::<T, I>::free_balance(&user).is_zero());
  }
 
  // Benchmark `force_transfer` extrinsic with the worst possible conditions:
  // * Transfer will kill the sender account.
  // * Transfer will create the recipient account.
- force_transfer {
+ #[benchmark]
+ fn force_transfer() {
  let existential_deposit = T::ExistentialDeposit::get();
  let source: T::AccountId = account("source", 0, SEED);
  let source_lookup = T::Lookup::unlookup(source.clone());
@@ -133,23 +155,26 @@ benchmarks_instance_pallet! {
  let balance = existential_deposit.saturating_mul(ED_MULTIPLIER.into());
  let _ = <Balances<T, I> as Currency<_>>::make_free_balance_be(&source, balance);
 
- // Transfer `e - 1` existential deposits + 1 unit, which guarantees to create one account, and reap this user.
+ // Transfer `e - 1` existential deposits + 1 unit, which guarantees to create one account,
+ // and reap this user.
  let recipient: T::AccountId = account("recipient", 0, SEED);
  let recipient_lookup = T::Lookup::unlookup(recipient.clone());
- let transfer_amount = existential_deposit.saturating_mul((ED_MULTIPLIER - 1).into()) + 1u32.into();
- }: force_transfer(RawOrigin::Root, source_lookup, recipient_lookup, transfer_amount)
- verify {
+ let transfer_amount =
+ existential_deposit.saturating_mul((ED_MULTIPLIER - 1).into()) + 1u32.into();
+
+ #[extrinsic_call]
+ _(RawOrigin::Root, source_lookup, recipient_lookup, transfer_amount);
+
  assert_eq!(Balances::<T, I>::free_balance(&source), Zero::zero());
  assert_eq!(Balances::<T, I>::free_balance(&recipient), transfer_amount);
  }
 
  // This benchmark performs the same operation as `transfer` in the worst case scenario,
  // but additionally introduces many new users into the storage, increasing the the merkle
  // trie and PoV size.
- #[extra]
- transfer_increasing_users {
+ #[benchmark(extra)]
+ fn transfer_increasing_users(u: Linear<0, 1_000>) {
  // 1_000 is not very much, but this upper bound can be controlled by the CLI.
- let u in 0 .. 1_000;
  let existential_deposit = T::ExistentialDeposit::get();
  let caller = whitelisted_caller();
 
@@ -161,25 +186,29 @@ benchmarks_instance_pallet! {
  // and reap this user.
  let recipient: T::AccountId = account("recipient", 0, SEED);
  let recipient_lookup = T::Lookup::unlookup(recipient.clone());
- let transfer_amount = existential_deposit.saturating_mul((ED_MULTIPLIER - 1).into()) + 1u32.into();
+ let transfer_amount =
+ existential_deposit.saturating_mul((ED_MULTIPLIER - 1).into()) + 1u32.into();
 
  // Create a bunch of users in storage.
- for i in 0 .. u {
+ for i in 0..u {
  // The `account` function uses `blake2_256` to generate unique accounts, so these
  // should be quite random and evenly distributed in the trie.
  let new_user: T::AccountId = account("new_user", i, SEED);
  let _ = <Balances<T, I> as Currency<_>>::make_free_balance_be(&new_user, balance);
  }
- }: transfer(RawOrigin::Signed(caller.clone()), recipient_lookup, transfer_amount)
- verify {
+
+ #[extrinsic_call]
+ transfer(RawOrigin::Signed(caller.clone()), recipient_lookup, transfer_amount);
+
  assert_eq!(Balances::<T, I>::free_balance(&caller), Zero::zero());
  assert_eq!(Balances::<T, I>::free_balance(&recipient), transfer_amount);
  }
 
  // Benchmark `transfer_all` with the worst possible condition:
  // * The recipient account is created
  // * The sender is killed
- transfer_all {
+ #[benchmark]
+ fn transfer_all() {
  let caller = whitelisted_caller();
  let recipient: T::AccountId = account("recipient", 0, SEED);
  let recipient_lookup = T::Lookup::unlookup(recipient.clone());
@@ -188,13 +217,16 @@ benchmarks_instance_pallet! {
  let existential_deposit = T::ExistentialDeposit::get();
  let balance = existential_deposit.saturating_mul(ED_MULTIPLIER.into());
  let _ = <Balances<T, I> as Currency<_>>::make_free_balance_be(&caller, balance);
- }: _(RawOrigin::Signed(caller.clone()), recipient_lookup, false)
- verify {
+
+ #[extrinsic_call]
+ _(RawOrigin::Signed(caller.clone()), recipient_lookup, false);
+
  assert!(Balances::<T, I>::free_balance(&caller).is_zero());
  assert_eq!(Balances::<T, I>::free_balance(&recipient), balance);
  }
 
- force_unreserve {
+ #[benchmark]
+ fn force_unreserve() {
  let user: T::AccountId = account("user", 0, SEED);
  let user_lookup = T::Lookup::unlookup(user.clone());
 
@@ -208,15 +240,16 @@ benchmarks_instance_pallet! {
  assert_eq!(Balances::<T, I>::reserved_balance(&user), balance);
  assert!(Balances::<T, I>::free_balance(&user).is_zero());
 
- }: _(RawOrigin::Root, user_lookup, balance)
- verify {
+ #[extrinsic_call]
+ _(RawOrigin::Root, user_lookup, balance);
+
  assert!(Balances::<T, I>::reserved_balance(&user).is_zero());
  assert_eq!(Balances::<T, I>::free_balance(&user), balance);
  }
 
- impl_benchmark_test_suite!(
+ impl_benchmark_test_suite! {
  Balances,
  crate::tests_composite::ExtBuilder::default().build(),
  crate::tests_composite::Test,
- )
+ }
 }