Fuzz testing for nomination pools #12002

kianenigma · 2022-08-10T11:48:03Z

This PR is itself pretty insubstantial, there is not real code change other than adding a fuzz-style test for nomination pools.

Some notes about the approach: For now, I mainly did the fuzz-testing without using a proper fuzzer because it would allow for the mock to be reused. In the past we gated mock.rs behind some feature flag to make it available, and this time around I wanted to try the alternative approach.

Having done this, I can say that the main advantage of this type of testing is seamless setup, and the downside is reproducibility of failing tests.

For now, in order to have reproducible errors, you have to use a seeded randomness, and then re-use it. It requires a little bit of manual work, but it did allow me to solve every single test failure.

It is probably still better to move the fuzz-test use a proper fuzzer as follow-up. Nonetheless, it must be done such that the mock file is reused (a good first issue for a new team member or external contributor).

Some notes about the fuzzing outcome: As you see in the code, I randomly generate calls that may or may not be valid and bombard the pallet with these calls, and every so often I execute the entire set of sanity checks that are defined in the pallet.

Moreover, I inject a single agent into a random pool that is manually tracking the expected reward amount (see Agent in the code for details).

The tests generally get quite slow, particularly if you want to execute the sanity checks often. In the default setup, the state grows until 1000 pools and 10000 members, and with 1 million random submissions, this is the final outcome:

Aug 10 16:16:23.564  INFO runtime::nomination-pools: [1] 🏊‍♂️ running sanity checks at 1010000
Aug 10 16:16:23.640  INFO runtime::nomination-pools: [1] 🏊‍♂️ iteration 1010000, 1000 pools, 10000 members, 325223 ok 684777 err, events = [("Event::Created", 1000), ("Event::Bonded", 72300), ("Event::Unbonded", 41458), ("Event::PaidOut", 164750), ("Event::Withdrawn", 21622), ("Event::MemberRemoved", 1)]

All in all, this was super useful and I feel way more confident about nomination pools now.

follow-up Issue to migrate this to a proper fuzzer.

…ls-battle-testing-2

kianenigma · 2022-08-10T13:54:07Z

frame/support/src/lib.rs

@@ -428,6 +428,13 @@ macro_rules! parameter_types_impl_thread_local {
 					pub fn set(t: $type) {
 						[<$name:snake:upper>].with(|v| *v.borrow_mut() = t);
 					}
+
+					/// Mutate the internal value in place.
+					pub fn mutate<F: FnOnce(&mut $type) -> ()>(mutate: F) {


This is a unrelated change -- small enough to sneak in here, but I can also extract.

kianenigma · 2022-08-10T13:54:25Z

primitives/runtime/src/lib.rs

@@ -825,7 +827,7 @@ pub fn verify_encoded_lazy<V: Verify, T: codec::Encode>(
 macro_rules! assert_eq_error_rate {
 	($x:expr, $y:expr, $error:expr $(,)?) => {
 		assert!(
-			($x) >= (($y) - ($error)) && ($x) <= (($y) + ($error)),
+			($x >= $crate::Saturating::saturating_sub($y, $error)) && ($x <= $crate::Saturating::saturating_add($y, $error)),


This is a unrelated change -- small enough to sneak in here, but I can also extract.

sam0x17

Really cool to see some fuzzing. Looks good to me!

…ls-battle-testing-2

ggwpez · 2022-09-12T11:46:16Z

Some notes about the fuzzing outcome: As you see in the code, I randomly generate calls that may or may not be valid and bombard the pallet with these calls, and every so often I execute the entire set of sanity checks that are defined in the pallet.

We could create a tool which generates a set of calls from the metadata of a pallet. Could even be done in PolkadotJS, as general fuzzy testing framework.