Limit fuzz runs to possible inputs

[0xmikko]

Component

Forge

Have you ensured that all of these are up to date?

• Foundry
• Foundryup

What version of Foundry are you on?

forge 0.2.0 (6130af7 2022-04-07T00:08:44.543037+00:00)

What command(s) is the bug in?

forge test

Operating System

macOS (Apple Silicon)

Describe the bug

The problem is connected with fuzzing testing.
We use bool parameter to cover possible test. So, some of our tests looks so:

function test_ACV1_2_05_remove_liquidity_works_as_expected(bool multicall)
        public;

So, as you can see there are only 2 options to check: true or false, btw, it passes through 256 test or the quantity is set in env var.

Please, check the max possible options to limit fuzzing.

[mattsse]

you're of course right

it should be possible to implement full coverage for arguments that allow it

[mds1]

it should be possible to implement full coverage for arguments that allow it

This is true in the case of bool here, as well as other cases like a single uint8 or with two bool inputs. But this doesn't scale well because you'd basically need to hardcode a set of signatures for which foundry says "don't use regular fuzz rules, instead enumerate all inputs if the user's number of fuzz runs allow it".

Ideas for approaches that might scale better:

• Use a table test format: Table tests #858
• Use a library like solidity-generators to generate your test arrays and loop over it as a concrete test instead of a fuzz test
• Architect your tests differently, such as:

// This is the actual test foundry executes.
function test_ACV1_2_05_remove_liquidity_works_as_expected() public {
  _test_ACV1_2_05_remove_liquidity_works_as_expected(true);
  _test_ACV1_2_05_remove_liquidity_works_as_expected(false);
}

// Actual test logic and assertions here.
function _test_ACV1_2_05_remove_liquidity_works_as_expected(bool) internal {
    // --- snip ---
}

[OliverNChalk]

Would it not be possible to check the number of possible combinations basis the byte-size of the function arguments? E.g. if you have a signature of:

func(bool,bool)

you have 2 bits of possible inputs, ergo 4 cases. Similarly, if you have:

func(uint8,uint8)

you have 16 bits = 65536 possible combinations. So in this case, 256 fuzz runs can only cover 8 bits of input information. Assuming most people won't fuzz more than 10_000 times in their CI runs, it would only be possible to exhaustively cover ~13 bits of input. The question would then be is it worth implementing a secondary strategy that generally won't cover more than 13-16 bits of input.

[onbjerg]

We just don't have a way currently to limit the fuzz runs based on input size. As to feasibility with our current setup I couldn't tell, we'd need to check the proptest docs :) But yes, the general idea you outlined applies

[onbjerg]

I've investigated more and it is not possible to do with proptest. We would essentially need to manually calculate the input space and forego using proptest all together if the number of possible inputs is less than the number of fuzz runs and just do a loop instead.

[mds1]

Going to close this since it would be difficult to implement with the current fuzz architecture, afaik it's not a feature of most other fuzzers, and there are workarounds described above to get similar behavior