[ios, build] Expand iOS tests to more versions

[friedbunny]

• Runs tests with multiple concurrent simulators across all supported iOS versions. Fixes [ios] Run test suite against more versions of iOS on CI #10708.
  • Tests currently fail on iOS <11 — will address those here before merging.
  • Installs a forked xcpretty (technology-ebay-de/xcpretty@2b68127) that partially fixes Test results are missing when using parallel simulators on Xcode 9 xcpretty/xcpretty#295, a bug preventing test results from appearing when multiple simulators are in use.
• Adds make ios-static-analyzer and runs it on CircleCI. Fixes Run Xcode static analyzer on CircleCI for iOS builds #1549.
• Adds make ios-sanitize with thread sanitizer and undefined behavior sanitizer.
  • Replaces make ios-sanitize-thread.
• Address sanitizer tests (make ios-sanitize-address) are still disabled, waiting on a fix for ASan heap-buffer-overflow in MGLPolyline #10741.

/cc @akitchen @1ec5 @fabian-guerra

[friedbunny]

Looks like CircleCI’s stock macOS image doesn’t have iOS 8 or 9 simulators installed, which is a bummer — I’ll look into ways to get them installed.

https://discuss.circleci.com/t/please-add-simulators-for-ios-9-10-to-xcode-9-image/16530

[friedbunny]

OK, I’ve got this to a point where it’s running more or less properly. Remaining:

• Fix unrelated failing tests on iOS 10 → [ios] Fix and expand accessibility label transliteration #10881
• Save raw logs as CI artifacts.

[friedbunny]

I rebased this to bring in the new iOS integration tests, but they’re not playing nicely with multiple simulators — even with a reduced set, most of the simulators fail to boot:

2018-02-14 10:08:27.057 xcodebuild[2641:24307]  iOSSimulator: task_name_for_pid(mach_task_self(), 3063, &task) returned 5
2018-02-14 10:08:27.061 xcodebuild[2641:24307]  iOSSimulator: Falling back to DISPATCH_SOURCE_TYPE_PROC
2018-02-14 10:08:27.067 xcodebuild[2641:23586]  iOSSimulator: task_name_for_pid(mach_task_self(), 3062, &task) returned 5
2018-02-14 10:08:27.068 xcodebuild[2641:23586]  iOSSimulator: Falling back to DISPATCH_SOURCE_TYPE_PROC
2018-02-14 10:08:27.729 xcodebuild[2641:19676] Error Domain=IDETestOperationsObserverErrorDomain Code=6 "Early unexpected exit, operation never finished bootstrapping - no restart will be attempted" UserInfo={NSLocalizedDescription=Early unexpected exit, operation never finished bootstrapping - no restart will be attempted}
2018-02-14 10:08:27.738 xcodebuild[2641:23596] Connection peer refused channel request for "dtxproxy:XCTestManager_IDEInterface:XCTestManager_DaemonConnectionInterface"; channel canceled <DTXChannel: 0x7f8ba6be05d0>
2018-02-14 10:08:27.747 xcodebuild[2641:19676] Error Domain=IDETestOperationsObserverErrorDomain Code=6 "Early unexpected exit, operation never finished bootstrapping - no restart will be attempted" UserInfo={NSLocalizedDescription=Early unexpected exit, operation never finished bootstrapping - no restart will be attempted}
2018-02-14 10:08:27.747 xcodebuild[2641:24306] Connection peer refused channel request for "dtxproxy:XCTestManager_IDEInterface:XCTestManager_DaemonConnectionInterface"; channel canceled <DTXChannel: 0x7f8ba728b310>

Testing failed:
	Test target integration encountered an error (Early unexpected exit, operation never finished bootstrapping - no restart will be attempted)
** TEST FAILED **

/cc @akitchen

[akitchen]

I also encountered this failure locally on Xcode 9.2, on a late model MBP 13"

[friedbunny]

Everything not related to getting multiple simulators to run is now moving in #11529.

[1ec5]

#11776 removes support for deploying back to iOS 8.x, due in part to the challenges associated with this PR. Does this PR contain any work that’ll come in handy the next time we need to drop support for an iOS version?

[stale]

This pull request has been automatically detected as stale because it has not had recent activity and will be archived. Thank you for your contributions.