Continuous Integration

You can run Subliminal tests in continuous integration using the subliminal-test script, which takes care of building your application and running the tests on the appropriate simulator or device. To get started, choose a CI platform below.

CI Platforms

Self-hosted (Jenkins or Buildbot)

To use subliminal-test on CI servers like Jenkins or Buildbot, you must first:

1. Install Subliminal on the server:
   • If you added Subliminal to your project using Git submodules or manually, initialize your project's submodules (as applicable). Then, use Terminal to execute rake install DOCS=no from Subliminal's directory.
   • If you added Subliminal to your project using Cocoapods, use Terminal to execute pod install from the root directory of your project.
2. "Share" your "Integration Tests" scheme to make it available to your CI server: in Xcode, click Product > Schemes > Manage Schemes…, click the "Shared" checkbox next to the scheme, and check the resulting file into source control.
3. "Pre-authorize" Apple's instruments tool as described here.

You then can invoke subliminal-test using a script like the following:

#!/bin/bash

# Run the tests on the 3.5" Retina iPhone Simulator
DEVICE="iPhone Retina (3.5-inch)"

# Run the tests on iOS 7.0
VERSION=7.0

OUTPUT_DIR=reports
mkdir -p "$OUTPUT_DIR"

# Returns 0 on success, 1-4 on failure
# Log output and screenshots will be placed in $OUTPUT_DIR
"$PROJECT_DIR/Integration Tests/Subliminal/Supporting Files/CI/subliminal-test" \
    -project "$YOUR_PROJECT" \
    -sim_device "$DEVICE" \
    -sim_version "$VERSION" \
    -output "$OUTPUT_DIR"

You can process test logs into JUnit reports using the subliminal_uialog_to_junit script:

"$PROJECT_DIR/Integration Tests/Subliminal/Supporting Files/CI/subliminal_uialog_to_junit" \
    -i "$OUTPUT_DIR/Run\ Data/Automation\ Results.plist" \
    -o "$OUTPUT_DIR/junit.xml"

Travis CI

Travis CI is a CI service that runs OS X and is free for open-source projects. To use Travis to run subliminal-test:

1. Sign in and activate the GitHub webhook for your repository, as described here.

2. Make sure you've chosen to "Share" your "Integration Tests" scheme to make it available to Travis: in Xcode, click Product > Schemes > Manage Schemes…, click the "Shared" checkbox next to the scheme, and check the resulting file into source control.

3. Add a .travis.yml file to the root of your repository. Here's a minimal .travis.yml:

   language: objective-c
   
   env:
       global:
           - PROJECT_PATH="TestTravis.xcodeproj"
           - DEVICE="iPhone Retina (3.5-inch)"
           - VERSION="7.0"
   
   install:
       # Travis will automatically initialize git submodules & install Cocoapods.
       # Skip this step if using Cocoapods--the Podspec installs Subliminal automatically.
       - rake install --rakefile Integration\ Tests/Subliminal/Rakefile DOCS=no
   
   # Use Pods/Subliminal/.../subliminal-test if using Cocoapods
   script: >
       "Integration Tests/Subliminal/Supporting Files/CI/subliminal-test"
       -project "$PROJECT_PATH"
       -sim_device "$DEVICE"
       -sim_version "$VERSION"

   Subliminal runs integration tests against itself using Travis. Take a look at its configuration file for a more in-depth example.

   The only crucial elements are that your .travis.yml install Subliminal in the install step, unless it's using Cocoapods; and that it run subliminal-test in the script step.

4. Trigger a build by pushing to your repository.

FAQ

• What is "pre-authorization" and why is it necessary to run subliminal-test in a self-hosted CI environment?

  As a security feature, instruments asks the user for permission to take control of other processes. Nevertheless, permissions dialogs prevent scripts from running un-attended as is desirable for continuous integration.

  instruments can be "pre-authorized" to take control of your application by modifying a configuration file that determines the privileges granted to OS X applications. To pre-authorize instruments, first execute the following command(s) in Terminal:

  # If you are running OS X 10.8:
  sudo /usr/libexec/PlistBuddy -c "Set rights:system.privilege.taskport:allow-root true" /etc/authorization
  
  # If you are running OS X 10.9:
  security authorizationdb read system.privilege.taskport > /tmp/system.privilege.taskport.plist
  /usr/libexec/PlistBuddy -c "Set :allow-root true" /tmp/system.privilege.taskport.plist
  sudo security authorizationdb write system.privilege.taskport < /tmp/system.privilege.taskport.plist

  Then, try to run subliminal-test as you would normally (using a script like the example given in the self-hosted instructions above). You'll be prompted to enter your username and password once, but after that instruments will no longer require authorization.

  Obligatory warning: Pre-authorization will make your machine less secure. However, it is currently the only way to run UIAutomation scripts un-attended. Please dupe http://openradar.appspot.com/14135016 to encourage Apple to provide a true solution.

• I'm running a self-hosted CI environment and subliminal-test won't run the tests (it says that instruments is "hiccuping"). What's going on?

  Your CI server is probably trying to run subliminal-test as the root user rather than as the logged-in user. Moving the launchd plist file from /Library/LaunchDaemons to /Library/LaunchAgents or ~/Library/LaunchAgents should solve the problem. See this thread for more information.