Skip to content
This repository has been archived by the owner on Aug 8, 2023. It is now read-only.

reenable iOS tests #1452

Closed
jfirebaugh opened this issue May 6, 2015 · 7 comments
Closed

reenable iOS tests #1452

jfirebaugh opened this issue May 6, 2015 · 7 comments
Assignees
@jfirebaugh
Labels
build iOS Mapbox Maps SDK for iOS tests
Milestone
iOS Beta 2

Comments

@jfirebaugh
Copy link
Contributor

We disabled iOS tests because they were hanging with 100% reproducibility on Travis while passing locally.

Revert a6dda07 once CI infrastructure stabilizes.
@incanus incanus added iOS Mapbox Maps SDK for iOS build labels May 6, 2015
@incanus incanus mentioned this issue May 7, 2015
Closed
@incanus incanus added this to the iOS Beta 2 milestone May 9, 2015
@incanus
Copy link
Contributor

incanus commented May 9, 2015

We should check out the new iOS 8 XCTestExpectation for the delegate callbacks. #conferencelearning

@incanus
Copy link
Contributor

incanus commented May 12, 2015

We are starting to use XCTestExpectation over in #1517. This will likely work well for the delegate stuff.

This was referenced May 12, 2015
Closed
Merged
@incanus incanus added the tests label May 14, 2015
@bleege bleege mentioned this issue May 18, 2015
Closed
@1ec5 1ec5 mentioned this issue May 21, 2015
Closed
@jfirebaugh jfirebaugh mentioned this issue May 29, 2015
Closed
jfirebaugh added a commit that referenced this issue May 29, 2015
@jfirebaugh
Run iOS tests again
8879dcd 
Fixes #1452
@jfirebaugh
Copy link
Contributor Author

I've been doing some debugging to try to pinpoint why Travis hangs. I created a test repo with a minimal iOS test and replicated the hang on Travis.

On Travis's Xcode 6.3 beta image, we get the following ouput from diagnostic commands:

$ xcodebuild -showsdks
OS X SDKs:
    OS X 10.9                       -sdk macosx10.9
    OS X 10.10                      -sdk macosx10.10
iOS SDKs:
    iOS 8.3                         -sdk iphoneos8.3
iOS Simulator SDKs:
    Simulator - iOS 8.3             -sdk iphonesimulator8.3
The command "xcodebuild -showsdks" exited with 0.
3.00s$ instruments -s devices
Known Devices:
Travis’s Mac (4) [00000000-0000-1000-8000-005056A677FC]
Resizable iPad (8.1 Simulator) [6F64182A-B90B-4939-A1A2-FB20F473A380]
Resizable iPad (8.2 Simulator) [9FAB1C4F-A68B-4398-A4A8-D0BF07AD16D6]
Resizable iPad (8.3 Simulator) [25405AE0-F581-4D6A-8F54-56848453106B]
Resizable iPhone (8.1 Simulator) [90B79A99-CC8F-46A1-BA22-71133DDFA40F]
Resizable iPhone (8.2 Simulator) [2DAEC6F6-C98C-4B2C-B26F-194C77FA9253]
Resizable iPhone (8.3 Simulator) [CA738A26-5EF8-473A-AE1C-1A9418F214A5]
iPad 2 (7.1 Simulator) [F0199676-B5DE-4D08-B79B-870B02C05A18]
iPad 2 (8.1 Simulator) [F3B0AE22-B18B-402F-BD92-1836388DF174]
iPad 2 (8.2 Simulator) [F221AE13-D0CD-4A42-AAC1-B8CB4AF4DC6F]
iPad 2 (8.3 Simulator) [6E7F740B-8FC0-4E7E-A999-532C369FB293]
iPad Air (7.1 Simulator) [8FDFE5EF-4221-4101-B03F-4792E1ECF3DE]
iPad Air (8.1 Simulator) [993A542B-3170-412C-B308-78C8A346799C]
iPad Air (8.2 Simulator) [ED565A20-1A62-48EC-92B9-E53E4F528E01]
iPad Air (8.3 Simulator) [007455C5-3963-41CF-A50E-3F73DC11DCE4]
iPad Retina (7.1 Simulator) [69E93454-0DBF-4130-9C79-D715266D2854]
iPad Retina (8.1 Simulator) [80E7B97E-0F05-43B7-89BB-59C324F44DF1]
iPad Retina (8.2 Simulator) [1A761A07-DB90-4C12-86BB-EA5BF8166A87]
iPad Retina (8.3 Simulator) [71A0CE79-077D-45D3-B5AE-6897B6EC5514]
iPhone 4s (7.1 Simulator) [27C30F21-56EF-46A0-9BF3-29F7638E0F94]
iPhone 4s (8.1 Simulator) [5E72128E-B078-45B6-9860-B761CA88CE65]
iPhone 4s (8.2 Simulator) [46FEA86C-934B-4E97-9112-48A2510285B2]
iPhone 4s (8.3 Simulator) [8E7FD3EF-20B6-4BDE-B41B-6540997121AD]
iPhone 5 (7.1 Simulator) [1597A9EE-56F3-443A-960A-5EB7A668F33A]
iPhone 5 (8.1 Simulator) [4DAA2E47-3536-494E-A50F-4A901DB179F3]
iPhone 5 (8.2 Simulator) [5726DEB2-A7EF-48A3-9223-DE38104C4104]
iPhone 5 (8.3 Simulator) [93243A9F-2768-4979-9A41-86BC75EBC18C]
iPhone 5s (7.1 Simulator) [81C46931-BFAB-4669-87FC-9E45E4634196]
iPhone 5s (8.1 Simulator) [38413F39-5A5C-474F-AFF6-5556916EB267]
iPhone 5s (8.2 Simulator) [D0CC7592-B18B-489E-9BF3-D293663055D6]
iPhone 5s (8.3 Simulator) [1005D367-4809-4B61-A33E-31B3A1FE6B71]
iPhone 6 (8.1 Simulator) [397EFF88-9A7C-4889-91E1-6F1BCF6AB24F]
iPhone 6 (8.2 Simulator) [43D1FD5C-E7F3-4922-AEEC-B23F8606CE29]
iPhone 6 (8.3 Simulator) [A4D8DEA5-6009-47E7-925F-A1912EBDA005]
iPhone 6 Plus (8.1 Simulator) [69277837-90F0-4FE8-B654-7AC9BB880565]
iPhone 6 Plus (8.2 Simulator) [6D7DBD9D-FFA4-4554-AF3C-F4F7008E9A61]
iPhone 6 Plus (8.3 Simulator) [BFF8CAB0-371E-44CF-874A-647C3A8B641D]

Based on this, I tried xcodebuild -project travis-ios-test.xcodeproj -scheme travis-ios-test -sdk iphonesimulator8.3 -destination 'platform=iOS Simulator,name=iPhone 5s' test and it works. Other permutations of -sdk and -destination that I have tried fail. Using xctool rather than xcodebuild works. I haven't investigated whether xctool can do multi-destination builds.

Next step here is either locating the intersection of the permutations that we'd like to test with the permutations that actually work on Travis, or investigating a wholesale switch to xctool.

@jfirebaugh
Copy link
Contributor Author

Tried a few more things, narrowed the failure case down, emailed Travis support:

Hi,

I'm trying to get iOS tests running on Travis. I need to test specific combinations of hardware and iOS version, so I'm using the -destination option for xcodebuild, which is also supported by xctool.

However, this does not work on Travis. I've created a minimal test project to demonstrate the issue. Using xcodebuild, the command simply hangs until killed by Travis's 10 minute timeout:

https://travis-ci.org/mapbox/travis-ios-test/builds/64958824

Using xctool, I get some debugging output that points to a test environment issue that causes the simulator to be unable to launch:

https://travis-ci.org/mapbox/travis-ios-test/builds/64997691

The only way I've been able to get xcodebuild or xctool to work on Travis is by omitting the "OS" option for -destination entirely, but this doesn't satisfy my use case:

https://travis-ci.org/mapbox/travis-ios-test/builds/64657039

Can you provide any information about these errors, or in general about how to test against specific hardware / iOS profiles?

Thanks,
John

@jfirebaugh jfirebaugh self-assigned this Jun 2, 2015
@incanus
Copy link
Contributor

incanus commented Jun 15, 2015

@jfirebaugh Have you heard back from Travis on this? Any leads?

@jfirebaugh
Copy link
Contributor Author

Yeah, got this back a few days ago:

Hi John,

I apologize for the big delay. We've had some unexpected team availability shortages this past week.

I must admit that this is the first time I've encountered your use case, and I fear that all of my questions about how to proceed are very naïve. I've been unable to find any information internally about how to test against specific hardware / iOS profiles. Given the fact that your builds are succeeding locally, I have to assume there's something specific about our build VM environment that's causing this, which may be because it's a virtualized environment, or perhaps the stalling behavior is because of a window prompt awaiting input. Such problems typically require flipping all of the debugging and tracing flags and triggering screen captures in the background.

Assuming you're still having issues with -destination, I'd like to fork your project and do some debugging with a background screencapture tool loop. What do you think?

I replied affirmatively.

It seems like our best bet right now is:

  • Use xctool instead of xcodebuild, as the former seems to have better failure modes
  • Test against a default -destination for now rather than a hardware / iOS version matrix

jfirebaugh added a commit that referenced this issue Jun 15, 2015
@jfirebaugh
Reenable iOS tests, default hardware/iOS version only for now
69cf6ed 
Use of the -destination argument to either xcodebuild or xctool
fails on Travis.

Refs #1452
@jfirebaugh jfirebaugh mentioned this issue Jun 15, 2015
Closed
@jfirebaugh
Copy link
Contributor Author

Done. Followup in #1727.

@jfirebaugh jfirebaugh removed the ready label Jun 15, 2015
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@jfirebaugh jfirebaugh

Labels
build iOS Mapbox Maps SDK for iOS tests
Projects
None yet
Milestone
iOS Beta 2
Development

No branches or pull requests
2 participants
@incanus @jfirebaugh