Fuzz testing only catches console.log in setUp() and in failing test functions, not passing functions

[webthethird]

Component

Forge

Have you ensured that all of these are up to date?

• Foundry
• Foundryup

What version of Foundry are you on?

forge 0.2.0 (2e99f7f 2022-12-03T00:08:28.619060629Z)

What command(s) is the bug in?

forge test

Operating System

Linux

Describe the bug

Similar to issue #655, which was closed as stale (creating a new issue since there have been changes since that issue, and mine may be subtly different).

Only console logs in failing functions and setUp() are showing in the Logs section of the results, above the traces, but not the logs in passing functions. I can see the other calls to console::log in the traces, but they're not getting formatted.

Running 1 test for src/test/compound/Comptroller-fork-diff.sol:TestComptroller
[PASS] test_claimComp_diff_before_after(uint8) (runs: 1, μ: 2924860, ~: 2924860)
Logs:
  Number of addresses: 74

Traces:
  [4852548] TestComptroller::setUp() 
    ├─ [0] console::log(Number of addresses: %s, 74) [staticcall]
    │   └─ ← ()
...
  [2924860] TestComptroller::test_claimComp_diff_before_after(5) 
    ├─ [0] console::log(Address %s, 0x000008599eB841e58b3e13305Da3dA5df5Ed8fFD) [staticcall]
    │   └─ ← ()

This would be fine if I were only trying to do one run, but since I'm fuzzing, I'd like to see all of the logs without printing all of the traces.

I have added a minimized test contract in my repo for reproducing this bug:
https://github.com/webthethird/solidity-diff-fuzz-upgrades/blob/main/src/test/test_logging.sol

[tommyrharper]

Would be great to fix this. Any updates?

[mds1]

Simple way to reproduce:

import "forge-std/Test.sol";

contract Test5 is Test {
  function setUp() public {
    console2.log("Test5.setUp()");
  }
  function test_FuzzVals(uint x) external {
    console2.log("Test5.test_FuzzVals()");
  }
}

As shown in the screenshot below, forge test with any verbosity shows the Test5.setUp() log, but never shows the Test5.test_FuzzVals() log. However, in the trace, both are shown.

When all runs of a fuzz test are successful, the trace shows the values from the last run. The console.logs should do the same

[upalchowdhury]

running into the same issue. tried both console and console2 but its not outputting . only prints if it is in setUp()

[mds1]

cc @Evalir, I just verified this bug is still occurring on the latest nightly using the repro in #3843 (comment). Guessing this is a simple fix

[Evalir]

So, I can confirm this is an issue—even though just fixing it could make everyone's CI or local runs extremely noisy as a default. @mds1 this is somewhat easy fix, but I wonder if we should put this behind some kind of flag or increase the verbosity level needed for printing fuzz logs?

[mds1]

even though just fixing it could make everyone's CI or local runs extremely noisy as a default

We already would show the logs if it was a regular test and not a fuzz test, so I think this is ok? Unclear why we'd want to handle the console.logs differently between the two, it seems unintuitive. To hide the logs you'd just run with -v as the verbosity

[danielpham765]

Any update on this issue? I really need to show logs in fuzz tests for both failed and success results.

[brucdarc]

Bump to this, also looking to get output on each fuzz test, not just the failing ones

[nicovalencia]

+1 to this use case being valuable

[d3mage]

still have it ;(

[sujithsomraaj]

+1 to this is

[deepdfs]

+1
Let me know if you need any help with adding the fix.

[yenchihliao]

• 1 to this

[Zartaj0]

having this feature would be great.

I think the only readability issue will be there because fuzzing runs the function too many times and logging every time will spam the terminal

[0xNilesh]

+1 to this

[m-waqas88]

+1 to this

This feature is immensely required.

[grandizzy]

#7781 is now merged

[Azleal]

exciting...

[louislung]

#7781 is reverted in #7935
Should this issue be reopened?

[grandizzy]

I think we could track it woth #3844

[Azleal]

I think we could track it woth #3844

how should this feature be implemented? i didn't take every situation into consideration in my last try. i'm still using locally built foundry when fuzz logs are needed 😅

[wagmi33]

+1 would love to see fuzzing results logged

[novaknole]

+1. when will this be available ?

[0xJuancito]

+1

[Azleal]

+1

It's already implemented, see here for fuzz config and here for fuzz inline config