Cannot run fuzzer for a fixed amount of time

[wuestholz]

Component

Forge

Describe the feature you would like

I would like to run the fuzzer for a fixed amount of time (for instance, one hour). It would be great if this was possible (just like for Echidna or other fuzzers). Unfortunately, the runs configuration option results in different running times depending on the contracts that are tested and how much time is spent on executing inputs.

Additional context

No response

[mds1]

Closing this as a duplicate of #990. Until implemented, you of course can work around this by setting runs to a very high value and just terminating the forge process after an hour

[wuestholz]

@mds1 Thanks for the quick reply! I don't quite see how the workaround you describe would work. When I terminate the process early I won't see the output (incl. any property violations that were detected). Or am I missing something?

(For context, I'm trying to compare Echidna and Foundry to see which fuzzer finds more bugs in a given amount of time. This feature would really help to ensure that the comparison is fair.)

[mds1]

I think implicit in this issue and #990 is: when the fuzzer is run for a duration, collect all violations found and report all of them at the end of the time.

Whereas the current behavior of forge is is: As soon as a fuzz test has a failure, it gets printed to the terminal immediately and that test stops executing.

In other words, if you set the fuzzer to a billion runs as a workaround for this feature request, and run 100 has a failure in the test_Add test, test_Add stops running and you only get told about that one failure. There is currently no way to collect multiple failures from the test, so to be fair that makes this a partial workaround.

I'm not sure how you plan to structure the benchmark (or how fuzzers are typically benchmarked 😅), but one way to structure a benchmark to work around this limitation is: come up with n different contracts/test cases, run the echidna and foundry test suites across all contracts/tests multiple times, and report the median time to find each/all bugs. That way you only need to find a single failure in each test case

[wuestholz]

@mds1 Thanks! Unfortunately, I can't observe the behavior that you describe. Perhaps it's because I'm using the invariant testing (i.e., stateful fuzzing) mode. Here's a simple repro:

• test/bug.t.sol:

pragma solidity ^0.8.19;
contract X {
  function foo() external {}
}
import "forge-std/Test.sol";
contract TestX is Test {
  X x;
  function setUp() external {
    x = new X();
  }
  function invariant_0() external { fail("0"); }
  function invariant_1() external { }
}

• foundry.toml:

[fuzz]
runs = 10000
max_test_rejects = 1000000000
seed = 0x00
dictionary_weight = 40
include_storage = true
include_push_bytes = true

[invariant]
runs = 10000
depth = 30
fail_on_revert = false
call_override = false
dictionary_weight = 80
include_storage = true
include_push_bytes = true

I use the following command to run the fuzzer:

$ forge test --match-path test/bug.t.sol --fuzz-seed 0x00

It should find the violation immediately, but nothing is shown when I abort the process early.

The current benchmarks contain several invariants just like the contract above. Some of them can be violated. I'm happy to share an example if you're interested.

[mds1]

Ah, interesting. What if you change that to a fuzz test, e.g. function test_0(uint256 x) external { fail("0"); }, does it find and log the violation immediately in that case? I think it should, so I assumed invariant tests do also (if fuzz tests do and invariant tests don't, I'd consider that a bug for the current architecture)

[wuestholz]

@mds1 Thanks! I tried what you suggested with the following files:

• test/bug.t.sol:

pragma solidity ^0.8.19;
contract X {
  function foo() external {}
}
import "forge-std/Test.sol";
contract TestX is Test {
  X x;
  function setUp() external {
    x = new X();
  }
  function test_0(uint256 y) external { fail("0"); }
  function test_1(uint256 y) external { }
}

• foundry.toml:

[fuzz]
runs = 1000000
max_test_rejects = 1000000000
seed = 0x00
dictionary_weight = 40
include_storage = true
include_push_bytes = true

[invariant]
runs = 10000
depth = 30
fail_on_revert = false
call_override = false
dictionary_weight = 80
include_storage = true
include_push_bytes = true

Note that I increase runs from 10000 to 1000000 to make sure it doesn't terminate too quickly.

When I abort the process early, nothing is shown either. So, the behavior for the two modes seems to be consistent. However, this means the suggested workaround doesn't work for either mode.

Is there a chance that you could change the behavior to report violations as soon as they are found (provided that this is easier than adding support for the time limit)?

[mds1]

Adding support for time based fuzzing requires a decent amount of changes since proptest doesn’t support it, more discussion in #990

Deferring to @mattsse on if it’s easy to report/save failures right away, so we can support using very high runs as a workaround. Potentially could implement #2551 as an easy solution here by just saving failures to a file

[wuestholz]

@mds1 Thanks! I had a idea for how to work around some of the issues with proptest. I shared it in that thread (#990 (comment)), but I'm not familiar with the Forge implementation.

For my purposes, it would also be sufficient to have some log output when a failure is reported, but reporting the failure immeditately seems like a nice usability improvement.