fixes #830, #880: bring back iOS tests; split OS X & iOS tests

[incanus]

Bring it back!

• make itest at the command line to run xcodebuild test in console
• builds static library using package_os.sh to run inside test harness app
  • This is basically impossible to integrate with GYP, so instead we test our framework as a user would, by using libMapboxGL.a in a ready-to-go app. This app has a KIF-based test bundle which you can run with Command-U in Xcode.
• Updates the old gl-cocoa-based project for the latest setup & API.
• Adds a few more tests like testing rotation now that KIF supports the gestures needed.

[incanus]

Thinking about punting on this for a while. I am getting various tests failing on the same build — e.g. simple gesture tests that pass on iOS 8.1 but fail on iOS 7, varying between builds. Always works locally. I think it's a Travis problem. I don't know if it's KIF or command-line simulation or what.

[incanus]

Holy crap, a620eee actually passed.

Re-squashing and force-pushing here.

[incanus]

I'm aiming to support automated testing on the following:

• iPhone 5s iOS 7.1
• iPhone 5s iOS 8.1
• iPad 2 iOS 7.1
• iPad 2 iOS 8.1

That gives us a fair coverage of iPhone/iPad, iOS 7/8, and retina/non-retina.

This is what both Travis and make itest use.

Manual testing (Command-U on test/ios/ios-tests.xcodeproj in Xcode) will use the currently-selected device configuration.

[incanus]

iOS tests all seem to be passing reliably now; we've also got the latest master merged in. I'm only seeing one device run, though, not the four we'd like that should be handled by xcodebuild test -destination with multiple flags. Will investigate.

[friedbunny]

xcpretty doesn't appear to be available on Homebrew, their suggested install is gem install xcpretty.

[incanus]

Oh, whoops. Yeah, we do actually already do it here in master:

mapbox-gl-native/scripts/travis_before_install.sh

Line 51 in 36cee94

mapbox_time "install_xcpretty" \

But I'll patch up the docs.

[friedbunny]

I got this working locally after swapping to iOS 8.2 in test_ios.sh and package_ios.sh.

Despite having 7.1, 8.1, and 8.2 installed and usable as simulators in Xcode 6.2, xcodebuild doesn't see them.

jason@keaton ~/mapbox-gl-native $ xcodebuild -showsdks
OS X SDKs:
    OS X 10.9                       -sdk macosx10.9
    OS X 10.10                      -sdk macosx10.10

iOS SDKs:
    iOS 8.2                         -sdk iphoneos8.2

iOS Simulator SDKs:
    Simulator - iOS 8.2             -sdk iphonesimulator8.2

[incanus]

Yep, these are the SDKs, not necessarily the runtime simulations. They are separate. A given version of Xcode (6.2 in the case) is going to build against the latest SDK at time of its release. You'll see only 8.3 for 6.3b, or 8.1 for 6.1.1.

I too am having problems with simulator versions, but mostly on Travis. They say they have 7.0, 7.1, and 8.1 currently. My problems are mostly with tests failing for unknown reasons despite never failing locally, or simulators timing out.

Peeled off some useful work into #1219, which will merge soon, then I'll rebase this and keep picking at it.

[incanus]

Just split off the packaging phase of this as part of #1219, so this needs some quick rebasing and can be picked up again.

I spent some time today trying to get screenshot uploads to S3 working without luck in the hopes of seeing what state things were in when tests mysteriously fail.

I may consider just leaving this as make itest and not yet worrying about Travis automatic run of those tests for now.

[incanus]

In some other tests, we've started to have good results with Bitrise for CI. I'm thinking the next steps here are to get these tests merged in, but run manually in Xcode / Makefile for the time being, then when we have good luck with CI, automate on a per-commit basis. I will pick this up early next week.

[friedbunny]

For those following along at home, with the move to Xcode 6.3 I had to also do:

make clean && make distclean && make itest

Otherwise I'd get variations on libuv linking errors:

Undefined symbols for architecture x86_64:
  "_alphasort$INODE64", referenced from:
      _uv__fs_readdir in libMapboxGL.a(fs.o)
  "_scandir$INODE64", referenced from:
      _uv__fs_readdir in libMapboxGL.a(fs.o)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

[friedbunny]

@incanus Derailing your thread here a bit, but when and how would you like tests contributed? I've got a mock CLLocationManager working and couple tests around user tracking modes written.

There is a potential issue with the Travis simulators not liking the location authorization override, but there are several ways to attack that if my preferred way doesn't cut it.

[incanus]

I've now got tests working locally across a matrix for both iPhone & iPad as well as iOS 7.1 and iOS 8.x. https://travis-ci.org/mapbox/mapbox-gl-native/jobs/59800251 indicates a failure with the compass going away when the map has north reset, which we've seen before (is #1064 related?). Working on that bug now.

MapViewTests
    ✓ testBottomLayoutGuide (1.643 seconds)
    ✓ testCenterSet (6.148 seconds)
    ✗ testCompassTap, ((tester.compass.alpha) equal to (0)) failed: ("1.000000") is not equal to ("0") - compass should not be visible when map is unrotated
    ✓ testDelegateRegionDidChange (3.366 seconds)
    ✗ testDirectionReset, ((tester.compass.alpha) equal to (0)) failed: ("1.000000") is not equal to ("0") - compass should not be visible when map is unrotated
    ✓ testDirectionSet (1.412 seconds)
    ✓ testPan (2.004 seconds)
    ✓ testPanDisabled (0.431 seconds)
    ✓ testRotate (4.461 seconds)
    ✓ testRotateDisabled (0.563 seconds)
    ✓ testTopLayoutGuide (1.562 seconds)
    ✓ testZoom (3.108 seconds)
    ✓ testZoomDisabled (0.543 seconds)
    ✓ testZoomSet (0.002 seconds)

[incanus]

Finally, success!!

I'm inclined to stick to Travis right now since load is down and the rest of our build matrix is there. If in the future we decide to look elsewhere, we can consider maybe putting the KIF tests on Bitrise of Greenhouse CI, both of which we've had success with elsewhere.

Sound good @bleege @1ec5?

Going to squash this now and prepare for merge.

[incanus]

To clarify one point about this whole setup:

• Tests run from a separate test/ios/ios-tests.xcodeproj instead of the dev iOS app for two reasons:
  • Making a unit testing bundle by way of GYP is a nightmare/maybe impossible.
  • We should test integration by way of static library as opposed to the "inline" method that the dev app uses.

[bleege]

I'm inclined to stick to Travis right now since load is down and the rest of our build matrix is there.

Am I catching things out of order now (likely) as I thought we were on to Bitrise now? Either way we should settle on one soon (today if possible) so that we can gain team confidence in the setup over a period of time.

[incanus]

Sorry for the confusion @bleege. Downstream, user-facing apps are on Bitrise, but the GL framework itself is on Travis with the rest of GL. Reasoning above in #1168 (comment).

The separation is:

• GL tests things like Do the map view constraints work? Does the compass work properly?
• App tests things like Do my uses of annotations work? Do my other services integrate well?