forge test --debug uses huge amounts of memory for complex tests, spilling into swapfile and causing kernel to kill the task (macOS)

[emo-eth]

Component

Forge

Have you ensured that all of these are up to date?

• Foundry
• Foundryup

What version of Foundry are you on?

forge 0.2.0 (7545c7a 2024-03-12T00:17:55.654014000Z)

What command(s) is the bug in?

forge test --debug

Operating System

macOS (Apple Silicon)

Describe the bug

The Forge debugger uses huge amounts of RAM, which macOS has no problem compressing, for what it's worth, but it still spills over into swapfile.

Having poked around the debugger and having a vague understanding of its mechanisms, I believe this is because the entire test is executed first, and the VM's state at each step during execution is saved. Intuition tells me this might also include sourcemaps, etc (perhaps tests should make more use of references vs clones/copying, if they aren't? not my wheelhouse).

Problems arise when the swapfile grows large enough that the kernel decides it's "too big" (in my case, with 500+GB free, ~41GB, when OS-reported memory usage of forge is ~384GB) and kills the process.

Repro:

Clone this repo: https://github.com/ProjectOpenSea/seaport-1.6/tree/4240dc6b30fa73f4b6b437fb588ebe583e796e3a
Edit this file: https://github.com/ProjectOpenSea/seaport-1.6/blob/main/test/foundry/new/FuzzMain.t.sol#L42

Change xtest_concrete to the following:

function test_concrete2() public {
        (
            uint256 seed,
            uint256 orders,
            uint256 maxOfferItemsPerOrder,
            uint256 maxConsiderationItemsPerOrder
        ) = (
            3,
            1,
            34284977338132815718688868832803618110488078917419162154611084291530410,
            10
        );
        bytes memory callData = abi.encodeCall(
            this.test_fuzz_generateOrders,
            (seed, orders, maxOfferItemsPerOrder, maxConsiderationItemsPerOrder)
        );
        (bool success, bytes memory result) = address(this).call(callData);
        if (!success) {
            if (result.length == 0) revert();
            assembly {
                revert(add(0x20, result), mload(result))
            }
        }
    }

Run the following: forge test -vvv --debug test_concrete2

[mattsse]

#7383 and #7385 only target the debugger

I tried running this forge test -vvv --debug test_concrete2
but it never entered the debugger because it failed

 Error: Mutation UnusedItemParameters_Token did not revert with the expected reason
  Error: a == b not satisfied [bytes]
        Left: 0x8baa579f
       Right: 0x6ab37ce7

though I saw it spike to > 100GB, so I think the issue is related to fuzzing.
will investigate a bit more

[emo-eth]

@mattsse failure is expected – the debugger wouldn't be much use if you could only debug tests that pass. It's also not a fuzz test, so if Forge is fuzzing things, it shouldn't be.

Building latest master, will give that a shot!

[mattsse]

@emo-eth I think I found the issue, it was indeed debug info collecting I believe.

could you please try main?

it's down to 15GB on my machine when it launches the debugger, with a few more fixes I think we should be able to improve this a lot more

[emo-eth]

@mattsse Totally works! 23gb on my machine, but only takes a second to enter the debugger – would sometimes take a minute plus for complex tests. Awesome work 🙏

Should we close this issue or leave open for remaining optimizations?

[mattsse]

thanks!

I'll close this and track followups separately.