Skip to content
This repository has been archived by the owner on Feb 13, 2020. It is now read-only.

Concurrent Testing

callahat edited this page Mar 17, 2016 · 14 revisions

Concurrent Testing with Sauce

Terms

  • 'Platform' here means a Sauce Labs platform
  • 'Sauce Concurrencies' here means the number of concurrent test slots on Sauce Labs' infrastructure
  • 'tests' means specs, tests and features interchangeably

The Options

Parallel Tests Gem

parallel_tests is a sweet gem that lets you run your specs, tests or features in parallel. It's highly configurable and takes away a lot of the concurrency pain on your behalf. You can use the parallel_tests gem as it stands.

Sauce Gem 3.0+ Rake Tasks

The Sauce gem also integrates a modified version of the parallel_tests gem, to take full advantage of the concurrency and multiple platform support Sauce Labs provides. These are accessible through Rake tasks sauce:spec and sauce:features.

##The Differences The Sauce integration is currently targeted at RSpec and Cucumber, running on a local server you can spin up multiple copies of. It runs a copy of each test for each platform, and divides them up across all the concurrency available to your Sauce Labs account by default

The parallel_tests gem allows you to run TestUnit, RSpec and Cucumber, running on a local server you can spin up multiple copies of. It divides your test files across the cores available to your local machine, by default. It then runs each test file across each platform in serial.

Usually, the limiting factor to how fast your tests perform is your local computational power, so running more tests at once won't provide a speed increase. When testing against a remote server like Sauce, however, the limited factor is the network latency. This means the speed of your tests increases as you run more of them, until you exceed your network bandwidth.

An Example

Your local machine has 4 cores, and you've 4 concurrent test slots. You've created 3 tests, Alpha, Beta and Delta. Then, you've specified 4 platforms:

  1. Chrome on Win7
  2. Chrome on Linux
  3. Firefox 18 on Win 8
  4. Firefox 18 on OS X
Tool Core 1 Core 2 Core 3 Core 4 Sauce Concurrencies Used
parallel_tests Alpha on 1, Alpha on 2, Alpha on 3, Alpha on 4 Beta on 1, Beta on 2, Beta on 3, Beta on 4 Delta on 1, Delta on 2, Delta on 3, Delta on 4 3
sauce parallel tasks Alpha on 1, Beta on 1, Delta on 1 Alpha on 2, Beta on 2, Delta on 3 Alpha on 3, Beta on 3, Delta on 3 Alpha on 4, Beta on 4, Delta on 4 4

Using the parallel_test gem

If you'd prefer to simply run each test sequentially across each platform, you can use the parallel_tests as it stands.

The parallel_tests gem provides two executables, parallel_rspec and parallel_cucumber, which use the gem's parallelization magic to invoke rspec or cucumber respectively.

If for example, you had a dual core computer, 3 platforms specified, and a features/ directory with 10 .feature files in it, you could run:

% bundle exec parallel_cucumber features

And the command would run two Cucumber processes at the same time, each executing one of the 10 feature files. In those files, each feature would run 3 times, once for each platform.

Don't forget to Check the Parallel Tests documentation.

Using the Sauce gem's built in parallel_tests support

Concurrency is built into the gem as of version 3.0+

How it works

The rake tasks check your account for your concurrency limit, load a helper file to configure the desired platforms, then divide your tests up among as many threads as you have concurrent Sauce sessions. It runs each test once for each configured platform, then collates the results.

Platform Configuration

Configuration is done the same way as for standard test runs, with a Sauce.config block.

Platforms are read from the "Browsers" option, and each platform is used once for each test:

  # This should be in a specific location (see below)
  # This will run each test 3 times
  Sauce.config do |c|
    c[:browsers] = [
      ["OSX 10.6", "Firefox", 17],
      ["Windows 7", "Opera", 10],
      ["Windows 7", "Firefox", 20]
    ]

See the configuration guide for more details

RSpec will read configuration details from spec/sauce_helper.rb

Cucumber will read configuration details from features/support/sauce_helper.rb

Note that the Browsers array will function normally during non-parallel tests, eg, by running each test in each platform in turn.

Rake Tasks

Set up

$ rake sauce:install:spec     # Create spec/sauce_helper.rb, add a require for it to spec/spec_helper.rb
$ rake sauce:install:features # Create features/support/sauce_helper.rb

RSpec

$ rake sauce:spec                                               # Run all specs in spec at max concurrency
$ rake sauce:spec test_files="spec/dynamic_email.spec"               # Run only dynamic_email.spec at max concurrency
$ rake sauce:spec test_files="spec/dynamic_email.spec" concurrency=8 # Run only dynamic_email.spec, at most 8 parallel instances

Cucumber

$ rake sauce:features                                                      # Run all features at max concurrency
$ rake sauce:features test_files="spec/dynamic_email.feature"                # Run only dynamic_email.feature at max concurrency
$ rake sauce:features test_files="spec/dynamic_email.feature" concurrency=8  # Run only dynamic_email.feature, at most 8 parallel instances

Task options

test_files                 Files or locations containing features/specs to run      (default: ./features or ./spec)
concurrency              How many concurrent tests to run                   (default:  Your account's max)
test_options             Arbitrary options to pass to cucumber or rspec 
                             eg: test_options='--fail-fast'
parallel_test_options    Arbitrary options to pass to the parallel_tests gem 
                             eg: parallel_test_options='--group-by scenarios'

See the Parallel Tests Readme for all available options.

Changes required to your application

Any behaviour that takes place during a non-concurrent run (say, with $ rspec) will occur during concurrent tests. This means you may need to change your test code or application config somewhat to accommodate parallel testing.

Knowing what test environment you're in

The parallel_tests gem sets the TEST_ENV_NUMBER environment variable for each parallel test group. You can use this to index arrays, make data a bit more unique, keep your logs cleaner and other fun stuff.

Behaviour that can only happen once, regardless of threading

For example, sending an informational email informing someone that a test has started and finished

ParallelTests.first_process? ? send_the_email(:start) : sleep(1)

at_exit do
  if ParallelTests.first_process?
    ParallelTests.wait_for_other_processes_to_finish
    send_the_email(:stop)
  end
end

(*NB: Don't send emails at the start and end of tests from within the tests. CI, baby!)

Databases

If your tests are reliant on database state, you may need to have multiple instances of your database.

One way of doing this is to create a database for each concurrent test you'll run, and use the TEST_ENV_NUMBER environment variable to refer to them.

For rails, there's a built in rake task in the parallel_tests gem, rake parallel:create, and to keep your schema up to date after migrations, rake parallel:migrate.

You'll need to edit your database.yml:

test:
  database: yourproject_test<%= ENV['TEST_ENV_NUMBER'] %>

Application Server Port

Rails

If the Rails process is started automatically by the Sauce gem (eg, :start_local_application is set to true in your config), it will use the TEST_ENV_NUMBER to start the server on its own Port.

Other application servers

Figure out how to use a unique port -- Turning ENV["TEST_ENV_NUMBER"] into an int and adding it to a base port number is a reasonable approach.

If your tests use Sauce Connect, you'll need to use the Sauce Connect compatible ports. They're available as Sauce::Config::POTENTIAL_PORTS. Again, using ENV["TEST_ENV_NUMBER"] as an index is a good approach.

Sauce Connect

If your tests start Sauce Connect by themselves (eg :start_tunnel or :application_host are set to a truthy value in Sauce.config and your tests use the standard integration), only one instance will be started.

If you want to make sure only one instance of Sauce Connect is started per test:

Sauce::Utilities::Connect.start # All options are passed to the Sauce Connect gem - Try {:quiet => true}

Sauce::Utilities::Connect.close # Closes the tunnel once all other threads are finished

These methods will only perform actions for the first thread, so you can safely call them from all test threads without checking if they're the first process.

Environment variables

  • TEST_ENV_NUMBER -- The number of the environment of the current thread. Starts with "", and then carries on up from 2, eg "", 2, 3
  • SAUCE_PERFILE_BROWSER -- A JSONified hash listing what environment this process should use for which file under test

Parallel Testing Tips

In order to run parallel tests effectively, your test code must be able to run at the same time as other tests.

  • Avoid shared or hard-coded data in your tests, if you need an email address or some other form of fake data, generate it yourself (check out the Faker gem).
  • If you can use a remote staging server and not have test data collisions, do it - ideally one that closely represents production and can handle many clients simultaneously. Spinning servers up and down is a sucker's game.
  • If you have to share a staging server, consider using ENV[TEST_ENV_NUMBER] to make unique test data; Appending it to strings or using it to index into arrays.
  • Make sure your tests pass on a single platform first. When testing with Sauce, Chrome is the fastest browser, so if you run your tests with Chrome first then you can confidently say that the app isn't fundamentally broken before spending more time running the tests on slower browsers.
  • Are these docs wrong? Can you explain better? Open an issue, we love feedback!

References

  • Parallel Tests gem -- The gem we're driving everything with (with judicious monkey patching)

Terminate zombie processes

The sauce_ruby gem will refuse to run in parallel if there are zombie processes from previous parallel test runs. On OSX and Linux, an easy way to terminate these is with the following shell command:

for pid in $(ps -ef | awk '/[T]EST_ENV_NUMBER=/ {print $2}'); do kill -9 $pid; done