fix: fuzz console log

[Azleal]

Motivation

**Closes #7928 #3843 **
when using fuzz test, console.log doesn't work as expected(display console log info). this PR is to fix console.log in fuzz test

Solution

current version of foundry only keep the logs for failed fuzz test case. so in this PR, added logs to keep logs in every CaseOutcome and display logs for every run case in a fuzz test.

[Azleal]

I tried to figure out how to add a test as instructed here, but I didn't figure it out. appreciate if anyone could kindly point out how to add one, i'll recommit this PR. thanks

[mattsse]

wdyt @grandizzy ?

[grandizzy]

I tried to figure out how to add a test as instructed here, but I didn't figure it out. appreciate if anyone could kindly point out how to add one, i'll recommit this PR. thanks

I think a test for this could be done similar to https://github.com/foundry-rs/foundry/pull/7544/files#diff-815e5f4f8a07901e1a8da4960e8334a0ba54a5bbd94ce5c620a7bd24ec1f36e9R530-R578 where it basically runs forge test -vv

/// forge-config: default.fuzz.runs = 1
function testConsoleLogFromFunction(uint256 x) public {
    console.log("logging from function");
}

and then make sure it prints logs properly by checking output for

Logs:
  logging from function

sequence

[Azleal]

I tried to figure out how to add a test as instructed here, but I didn't figure it out. appreciate if anyone could kindly point out how to add one, i'll recommit this PR. thanks

I think a test for this could be done similar to https://github.com/foundry-rs/foundry/pull/7544/files#diff-815e5f4f8a07901e1a8da4960e8334a0ba54a5bbd94ce5c620a7bd24ec1f36e9R530-R578 where it basically runs forge test -vv

/// forge-config: default.fuzz.runs = 1
function testConsoleLogFromFunction(uint256 x) public {
    console.log("logging from function");
}

and then make sure it prints logs properly by checking output for

Logs:
  logging from function

sequence

@grandizzy Thank you for your reference. I added a test for fuzz test console log.

[Azleal]

Hi @mattsse @grandizzy , I'm wondering if my updated PR meets the requirements for testing. If not, I'd like to discuss further and make improvements. hope to see this feature included in the next release soon.

[grandizzy]

I think this can be one liner as logs.borrow_mut().extend(_counterexample.1.logs.clone());

[Azleal]

@grandizzy thank you for pointing it out, one liner is much cleaner.

[grandizzy]

Hi @mattsse @grandizzy , I'm wondering if my updated PR meets the requirements for testing. If not, I'd like to discuss further and make improvements. hope to see this feature included in the next release soon.

I think that's a really cool addition, would be nice to have such for invariant tests as well, but out of the scope of this PR. @mattsse @DaniPopes pls have a look 🙏

[Azleal]

Hi @mattsse @grandizzy , I'm wondering if my updated PR meets the requirements for testing. If not, I'd like to discuss further and make improvements. hope to see this feature included in the next release soon.

I think that's a really cool addition, would be nice to have such for invariant tests as well, but out of the scope of this PR. @mattsse @DaniPopes pls have a look 🙏

I'll dig into invariant tests console logging after this PR closed

[Azleal]

Hi @mattsse @grandizzy , I'm wondering if my updated PR meets the requirements for testing. If not, I'd like to discuss further and make improvements. hope to see this feature included in the next release soon.

I think that's a really cool addition, would be nice to have such for invariant tests as well, but out of the scope of this PR. @mattsse @DaniPopes pls have a look 🙏

hi @grandizzy @mattsse @DaniPopes , it's been a while since this PR was committed. i'm using this feature by building foundry locally. and i noticed many others need this feature too. so i wonder if there's anything that i can do to push this forward? please don't hesitate to comment if anything that i can do. thanks for your time.

[mattsse]

sgtm,

only have one smol nit, ptal @grandizzy

[mattsse]

this clone can be avoided by tmp converting into_inner and borrow from there.

[Azleal]

this clone can be avoided by tmp converting into_inner and borrow from there.

how about this? @mattsse

[grandizzy]

lgtm, pls address the nit above

[j-vp]

I believe this commit causes the more verbose test runs to have extremely large outputs, especially on high-fuzz github actions (ie: 2000 runs, 500 tests, 2-3 bounds/test = a lot of logging).

Even on local tests, using -vvv to get failing test traces, this now becomes unworkable. Example: https://github.com/arcadia-finance/accounts-v2/actions/runs/9116420829/job/25065457315?pr=209

[grandizzy]

that's correct, it is because bounds are logged in forge std
https://github.com/foundry-rs/forge-std/blob/d44c4fbbb9ff054fb334babbdd34f9b6e899b3d6/src/StdUtils.sol#L62
https://github.com/foundry-rs/forge-std/blob/d44c4fbbb9ff054fb334babbdd34f9b6e899b3d6/src/StdUtils.sol#L87

@mattsse I am going to revert this commit for now as it looks it needs some more work
@mds1 do you know any reason for Result bound to be logged in forge-std while others are not?

[mds1]

So this PR now print console logs for every iteration of a fuzz set? I think previously we only printed them for the last successful run? Printing for every run feels extremely noisy

But the reason we log the bound result is to make debugging failed fuzz tests easier to reason about, and to simplify creating test cases to reproduce the failure with

[grandizzy]

So this PR now print console logs for every iteration of a fuzz set? I think previously we only printed them for the last successful run? Printing for every run feels extremely noisy

But the reason we log the bound result is to make debugging failed fuzz tests easier to reason about, and to simplify creating test cases to reproduce the failure with

gotcha, we have to re evaluate the proper way to implement such to avoid all the noise