More flexibility to measure gas usage for tests

[RickVM]

Component

Forge

Describe the feature you would like

I would like a way to do specific pre-test setups without it counting towards the gas-usage for that test.
Setup() is fine for a lof of cases, but I require a bit more flexibility to effectively write various test scenarios.

I'm open for ideas but can propose a solution:
A cheatcode that allows us to reset gas measurements from that point on during a test.
This will give users the most flexibility regarding what they want to measure.

    function testRemoveStates() public {
        uint256 amount = 100;
        States[] memory states = fillStates(100);
        vm.resetGas(); // I only want to measure the performance of removeStates in this scenario
        removeStates(states);
    }

Context:
I have a contract that keeps track of data, initialized with an empty state and various general parameters during setup(). (Struct[] with specific data)

On this contract, I want to run various tests to measure gas performance, e.g. for data retrieval and deletion.
Some tests would contain more values, some less. I do not want the part where I fill the state to be part of the gas measurements. Right now I'm finding myself needing to write abstract contracts with helper functions and extending that contract multiple times for each Test case (eg filled with 100, 1000, 10000 states), far from ideal

Additional context

No response

[RickVM]

I just noticed that the skip and vm.prank functions also add to the measured gas usage.
That's an additional reason to implement some kind of reset feature to allow for more fine-grained measurements :)

function testRemoveStates() public {
        skip(900000);
        vm.prank(theMint);
       // registry.removeOldData(removeKeys);
    }```

[onbjerg]

skip is a helper function in Forge Std, not an actual cheatcode, so it will impact gas consumption. vm.prank only impacts gas consumption because of Solidity opcodes to perform the call, the actual cheatcode logic itself does not add to gas consumption - not easy to remove since we can't reasonably have perfect knowledge of the emitted bytecode for the call

Currently this is not really possible with the EVM implementation we use since it's not just setting some gas used variable to 0, there is a lot of other considerations about cold/warm accoutns and storage slots. Warm storage slots and accounts also change other state of accounts that is used to figure out what to persist and so on, so it's definitely a hard challenge. I know it's been requested a lot though, so definitely worth exploring a lot more at some point

Currently the best we have is forge test --gas-report which will give you the gas cost of individual function calls, but again it may vary dependign on cold/warm accounts and storage slots.

Edit: A note on the API, I don't think vm.resetGas makes a lot of sense since you might have multiple points in your test you want to measure, so I think better UX would be something like vm.startGasMeasurement("some span name") and vm.stopGasMeasurement() where the span name would be displayed in a report (names suck but that's a tomorrow problem)

[mds1]

Going to close this per the above comment. If there is a need for better ways to measure gas usage than what's currently available, we'll need to come up with a good UX and implementation and create a new issue with that proposal