Expecting an assertion, eventually

[nevillco]

• I have read CONTRIBUTING and have done my best to follow them.

What did you do?

Wrote the following test suite:

func assertNotThree(_ value: Int) {
    assert(value != 3)
}

func asyncAssertNotThree(_ value: Int) {
    DispatchQueue.main.asyncAfter(deadline: .now() + 0.1) {
        assertNotThree(3)
    }
}

class NimbleExperimentTests: XCTestCase {

    func testAssertion() {
        let expression = { assertNotThree(3) }
        expect(expression: expression).to(throwAssertion())
    }

    func testAssertionAsync() {
        let expression = { asyncAssertNotThree(3) }
        expect(expression: expression).toEventually(throwAssertion())
    }
    
}

What did you expect to happen?

I would expect both tests to pass.

What actually happened instead?

The assertion inside func testAssertionAsync() does not get caught; I get the console message:

Assertion failed: file /Users/connorneville/Documents/Projects/NimbleExperiment/NimbleExperimentTests/NimbleExperimentTests.swift, line 14

Then, after a short delay, the test suite seems to reboot itself, and I get this console message:

Restarting after unexpected exit or crash in NimbleExperimentTests/testAssertionAsync(); summary will include totals from previous launches.

After this, the tests complete, and func testAssertionAsync() fails, having not caught the assertion.

Environment

• Quick: 1.2.0
• Nimble: 7.0.3
• Xcode Version: 9.2 (9C40b)
• Swift Version: 4.0
• Cocoapods: 1.5.0

[kaganriedel]

I am not seeing this issue on Xcode 9.2 / Swift 4.0, but after upgrading to Xcode 9.4 / Swift 4.1 I am experiencing the same issue described by @nevillco. Any updates on this?

[ikesyo]

throwAssertion() matcher expects that the given expression will throw assertion. But in the case of let expression = { asyncAssertNotThree(3) }, the expression which actually throws is not the closure ({ asyncAssertNotThree(3) }) nor asyncAssertNotThree function but the closure passed to DispatchQueue.asyncAfter API. Throwing an assertion is happening outside expect(...) body (context). So it seems to me that it is correct behavior.

[kaganriedel]

Thanks for the response @ikesyo. That makes sense!

I am seeing another issue as well. Ever since upgrading to Xcode 9.4 / Swift 4.2 tests stop program execution when hitting an assertion. The tests do pass, but I have to hit "Continue program execution" every time an assertion is thrown. I don't know if this is due to a change in Xcode, Swift, or Nimble.

I've included a quick example, cribbed from the code above. Note that this is not related to the async tests. Let me know if you need more information or if further discussion should be moved to a separate issue.

Quick example cribbed from above:

What to do?
Run this test from the above example.

    func testAssertion() {
        let expression = { assertNotThree(3) }
        expect(expression: expression).to(throwAssertion())
    }

What did you expect to happen?
Expect the tests to run without being stopped by Xcode when as assertion is thrown when using expect(expression: expression).to(throwAssertion()).

What actually happened instead?
Xcode stops program execution when the assertion is hit.

[ikesyo]

@kaganriedel That is #478 and it's fortunate that it will be resolved in a next patch version (#545).

[kaganriedel]

Wonderful! Thank you. 👍

[ikesyo]

Closing.