Test function early returns after `vm.expectRevert`

[0xbok]

Component

Forge

Have you ensured that all of these are up to date?

• Foundry
• Foundryup

What version of Foundry are you on?

forge 0.2.0 (d7733ee 2022-10-03T00:06:06.841223Z)

What command(s) is the bug in?

forge test

Operating System

macOS (Apple Silicon)

Describe the bug

pragma solidity 0.8.4;

import "forge-std/Test.sol";

contract ContractTest is Test {
    function setUp() public {}

    function rever() internal {
        revert();
    }

    function testExample() public {
        vm.expectRevert();
        rever();

        rever();
        console.log("Does not print");
    }
}

Running forge test -vvv passes this test and also doesn't print "Does not print" on console. In summary, after vm.expectRevert() the test just executes the call and does not run the remaining test.

[mattsse]

@DaniPopes there's something off here, do you have the bandwidth to look into this?

[DaniPopes]

This has something to do with reverting and expecting on the test contract itself since this works as expected:

// SPDX-License-Identifier: Unlicensed
pragma solidity ^0.8.4;

import "forge-std/Test.sol";

contract CustomContract {
    error CustomError();

    function rever() external {
        revert CustomError();
    }
}

contract ContractTest is Test {
    CustomContract internal c;

    function setUp() public {
        c = new CustomContract();
    }

    function testExample() public {
        // vm.expectRevert(CustomContract.CustomError.selector);
        vm.expectRevert();
        c.rever();

        c.rever();
        console.log("Does not print");
    }
}

Outputs:

$ forge test -vvv
[...]
[FAIL. Reason: CustomError()] testExample() (gas: 8795)
Traces:
  [8795] ContractTest::testExample() 
    ├─ [0] VM::expectRevert() 
    │   └─ ← ()
    ├─ [158] CustomContract::rever() 
    │   └─ ← "CustomError()"
    ├─ [158] CustomContract::rever() 
    │   └─ ← "CustomError()"
    └─ ← "CustomError()"

Test result: FAILED. 0 passed; 1 failed; finished in 244.88µs

Failing tests:
Encountered 1 failing test in test/a.t.sol:ContractTest
[FAIL. Reason: CustomError()] testExample() (gas: 8795)

Encountered a total of 1 failing tests, 0 tests succeeded

[0xbok]

This has something to do with reverting and expecting on the test contract

makes sense as that’s how I discovered this bug. I have an abstract contract that I’m testing by inheriting it from my test contract.

[DaniPopes]

I think what's happening is that the vm.expectRevert call makes the test expect a revert, so when the test contract itself reverts, stopping the execution of the test, it also succeeds since we expected a revert.
You can create a mock contract that just inherits the abstract contract and deploy and test with external calls on that

[mds1]

Does this happen (i.e. the test erroneously passes) if the reason the test contract reverts is due to a solidity panic like overflow or divide by zero? I don't see why you'd intentionally revert in a test contract like this, but I can imagine unintentional reverts due to math errors and want to make sure that doesn't cause tests to incorrectly pass.

I think what's happening is that the vm.expectRevert call makes the test expect a revert, so when the test contract itself reverts, stopping the execution of the test, it also succeeds since we expected a revert.

Hmm this seems odd though because my understanding is that expectRevert only expects reverts for calls made from the test contract, I don't think it should count reverts within the test contract, that feels like a bug

[0xbok]

I don't see why you'd intentionally revert in a test contract like this

I have an abstract contract that I’m testing by inheriting it from my test contract. The abstract contract has a function that can revert. The PoC that I showed in this issue is just a toy example replicating this behavior. But I'll be moving to @DaniPopes's workaround by creating a mock contract which just calls the abstract contract.

[mds1]

Got it, makes sense! I do still think we should verify the below, to make sure users can't have tests that are passing when you'd normally expect them to fail due to a revert in the test contract

Does this happen (i.e. the test erroneously passes) if the reason the test contract reverts is due to a solidity panic like overflow or divide by zero? I can imagine unintentional reverts due to math errors and want to make sure that doesn't cause tests to incorrectly pass.

[0xbok]

Yes, overflow passes as well:

pragma solidity 0.8.4;

import "forge-std/Test.sol";

contract ContractTest is Test {
    function setUp() public {}

    function rever(uint8 a) internal returns (uint8) {
        return a*2;
    }

    function testExample() public {
        vm.expectRevert();
        rever(254);

        rever(254);
        console.log("print");
    }
}

[mds1]

Thanks for testing! So IMO this is a bug that we should fix, cc @mattsse. According to the docs, expectRevert looks for a revert in the next call. In that example, there is never a call (just jumps to internal functions), so that test should fail from the overflow in rever(uint8)