Tests don't run and crash with error

[mrackwitz]

Describe the bug

The test crash with the following assertion failing:

xctest[59333:2879569] *** Assertion failure in -[XCTestObservationCenter addTestObserver:], XCTestObservationCenter.m:101
xctest[59333:2879569] *** Terminating app due to uncaught exception 'NSInternalInconsistencyException', reason: 'Test observers can only be registered and unregistered on the main thread.'

Expected behavior

Expected tests to run.

Screenshots
If applicable, add screenshots to help explain your problem.

Environment

• swift-snapshot-testing version: 1.10.0
• Xcode: 14.0.1
• Swift: Swift 5.
• Test OS: 5.7.1
• Dev OS: macOS 12.6

Additional context
Add any more context about the problem here.

[rbsgn]

Same issue happens at the attempt to verify cURL snapshot produced by async networking method.

[rbsgn]

This is where it crashes.

[Descartess]

This issue was introduced by this PR in release 1.10.0.

You can downgrade to release 1.9.0 till a fix is published.

[JackYoustra]

This is part of a broader problem, it seems like verifySnapshot only works on the main thread. Try using the main thread whenever calling assertSnapshot instead.

[mrackwitz]

This definitively does occur for us with Swift's actor-based concurrency. For UI-based tests e.g., I could see that there could be a main-thread dependency. But our tests in this case only use JSON-serialization, so I think it's rather odd that this would be confined to the main-thread. If there was an actual dependency, then shouldn't the API be annotated with @MainActor to indicate that to compiler and make sure this is used correctly?

[AlexKobachiJP]

This issue's been open for quite a while but I don't seem to find any PR to address it. Makes me wonder wether maybe I am holding it wrong?

For my use, I am adding async overloads for both of the assertSnapshot functions and use those in any async test method. The overloads just call out to the existing functions on the main actor.

Or is there a way do get around this without these overloads (and without having to dispatch calls to assertSnapshot individually at the call site)?

For reference, here are my overloads:

@MainActor
public func assertSnapshotAsync<Value, Format>(
    matching value: @autoclosure () throws -> Value,
    as snapshotting: Snapshotting<Value, Format>,
    named name: String? = nil,
    record recording: Bool = false,
    timeout: TimeInterval = 5,
    file: StaticString = #file,
    testName: String = #function,
    line: UInt = #line
) async {
    assertSnapshot(
        matching: try value(),
        as: snapshotting,
        named: name,
        record: recording,
        timeout: timeout,
        file: file,
        testName: testName,
        line: line
    )
}

@MainActor
public func assertSnapshotsAsync<Value, Format>(
    matching value: @autoclosure () throws -> Value,
    as strategies: [String: Snapshotting<Value, Format>],
    record recording: Bool = false,
    timeout: TimeInterval = 5,
    file: StaticString = #file,
    testName: String = #function,
    line: UInt = #line
) async {
    assertSnapshots(
        matching: try value (),
        as: strategies,
        record: recording,
        timeout: timeout,
        file: file,
        testName: testName,
        line: line
    )
}

[ianthetechie]

Same issue here. This is extremely confusing and annoying since many of our tests involve async processing for things like network requests or parsing.

[NachoSoto]

This is happening with Xcode 15.2 as well.
This is the culprit:

private class CleanCounterBetweenTestCases: NSObject, XCTestObservation {
    private static var registered = false
    private static var registerQueue = DispatchQueue(label: "co.pointfree.SnapshotTesting.testObserver")

    static func registerIfNeeded() {
      // Running on background thread, which is not allowed on XCTest with iOS 17.2
      registerQueue.sync {
        if !registered {
          registered = true
          XCTestObservationCenter.shared.addTestObserver(CleanCounterBetweenTestCases())
        }
      }
    }

    func testCaseDidFinish(_ testCase: XCTestCase) {
      counterQueue.sync {
        counterMap = [:]
      }
    }
}