Try to prevent app crashes from running Subliminal test cases

[justinseanmartin]

Subliminal should try its best to not crash in the middle of a test run, regardless of if it is caused by a test writer or the test framework itself.

Ideally, Subliminal and the app's integration test code would have no bugs and so only expected asserts would fire. However, if there are random failures in the framework that lead to exceptions being thrown, we'd be better off handling them and attempting to recover than bailing out of the entire automation run.

One approach would be to expose a GCD utility wrapper that catches exceptions and then proxies them back to the test thread. This would then be used internally for any methods expected to be called from test setup, execution or teardown. It would also be exposed for test case writers to safely invoke code on the main thread and not take down the entire app.

One thing that I'm thinking though is that if the app throws a legit NSException for doing something improper on the main thread, would you really want to keep the app up and running? I think the answer is yes, and it is expected that setup testCase should reset to a good state if possible, but I think it is debatable.

[wearhere]

I like the idea of a sl_dispatch_sync_to_main_queue a lot. I feel like we've had at least one Stack Overflow question and/or GitHub issue that's like "why do the tests just abort?" and I haven't made the connection until now.

One thing that I'm thinking though is that if the app throws a legit NSException for doing something improper on the main thread, would you really want to keep the app up and running?

I think that such a wrapper should catch all manner of exceptions. IMO an assertion doesn't necessarily mean "abort the app" but rather "halt this line of execution". (If a developer really wanted the former behavior, they could use assert rather than NSAssert.) Especially if the assertion occurs in response to some manipulation performed by a test, it like any other assertion occurring within tests (on the test thread) should not abort the tests. And if the wrapper rethrows the exceptions, then if the developer really did want to abort in response to a particular exception they could do so.