Configuring fuzzer parameters per test and per contract

[mds1]

Component

Forge

Describe the feature you would like

There are multiple ongoing improvements to the fuzzer, which will result in additional fuzzer settings that should be exposed to the user. These include:

• number of fuzz runs
• max_local_rejects and max_global_rejects (Added assume cheat code #725)
• frequency of dictionary values vs. random values (feat(forge): fuzz dictionary #731)
• the ability to manually specify values that should be added to the dictionary
• etc.

It's common to want to vary the values of these settings by test or by contract, instead of just globally. For example, see the Uniswap V3 echidna tests where they have multiple config files.

The scope of this issue is currently to just discuss approaches to specify separate fuzzing config for different tests. Below are some potential approaches, feel free to suggest others. @gakonst pointed out the cheat code based approaches may be tricky because currently proptest config lives outside of the EVM, so implementing that approach may require some refactoring

Approach 1: foundry.toml only

The foundry.toml config file has profiles so you can change the config easily. We can simply leverage these profiles and the --match-test and --match-contract flags to craft various combinations. For example:

[default]
fuzz-runs = 500

[shallow]
fuzz-runs = 100

[deep]
fuzz-runs = 25000

Then run tests with a tests.sh like this:

FOUNDRY_PROFILE=shallow forge test --match-contract ShallowTests
FOUNDRY_PROFILE=deep forge test --match-contract DeepTests
forge test --no-match-contract "ShallowTests|DeepTests"

This is nice because it works right now and requires no changes, but it's a bit clunky

Approach 2: Extend foundry.toml and a single cheat code

In this approach, the config file lets you specify fuzz profiles, and there's a cheat code you can use at the contract or test level to specify a profile name. For example:

[default]
# default fuzzer config for this profile
fuzz-runs = 100

# additional fuzzer configs that can be used with the `default` profile
[default.fuzz-shallow]
fuzz-runs = 10

[default.fuzz-deep]
fuzz-runs = 10000


[ci]
# default fuzzing config for this profile
fuzz-runs = 10000

# additional fuzzer configs that can be used with the `ci` profile
[ci.fuzz-shallow]
fuzz-runs = 1000

[ci.fuzz-deep]
fuzz-runs = 100000

Then in a test of contract you'd use

contract MyTests is DSTest {
  vm.fuzz_profile("fuzz-deep");

  function testSomething() public {
    // This (and other tests in this contract) will be fuzzed with the
    // `fuzz-deep` profile, which has 10,000 runs with the `default`
    // profile and 100,000 runs with the `ci` profile
  }

  function testSomething() public {
    vm.fuzz_profile("fuzz-shallow");
    // This will be fuzzed with the `fuzz-shallow` profile, which has 10 runs with
    // the `default` profile and 1000 runs with the `ci` profile
  }
}

contract OtherTests is DSTest {
  // Since we don't specify a fuzz profile, this will use the default of 100 runs with the
  // `default` profile and 10,000 runs in CI
}

Aside from requiring refactoring for cheat code support, another downside of this approach is that you need a new profile for every specific fuzzer configuration (i.e. there's no way to just change one fuzz setting in one test without creating a new profile specific for that test)

Approach 3: Extend foundry.toml and multiple cheat codes

This extends approach 2 by adding cheat codes for each fuzzer setting to remedy the downside mentioned. For example:

contract MyTest is DSTest {
  vm.fuzz_profile("fuzz-deep");

  function testSomething() public {
    // This test will use the fuzz-deep profile
  }

  function testSomethingElse() public {
    // This test will use all the settings of the fuzz-deep profile, but overrides number of runs
    vm.fuzz_runs("default", 250);
    vm.fuzz_runs("ci", 2500);
  }
}

Approach 4: Cheat codes only

Remove all fuzzing config from foundry.toml, and specify everything as cheat codes. For example:

contract MyTestHarness is DSTest {
  // Define the default fuzz settings for all tests
  vm.fuzz_runs("default", 100);
  vm.fuzz_runs("ci", 10000);
}

contract MyTest is MyTest Harness {
  function testSomething() public {
    // This uses the defaults specified in MyTestHarness
  }

  function testSomethingElse() public {
    // This uses 100 runs in the default profile and 50,000 runs in CI
    vm.fuzz_runs("ci", 50000);
  }
}

[mds1]

Closing as replaced by #3062 which has a concrete proposal to discuss