Vagrant Spec

Work-in-progress: This library is not ready for general use yet and is under active development. The documentation will become much better once this is more easily usable.

vagrant-spec is a both a specification of how Vagrant and its various components should behave as well as a library of testing helpers that let you write your own unit and acceptance tests for Vagrant.

The library provides a set of helper methods in addition to RSpec matchers and expectations to help you both unit test and acceptance test Vagrant components. The RSpec components are built on top of the helper methods so that the test library can be used with your test framework of choice, but the entire tool is geared very heavily towards RSpec.

Table of Contents

• Prerequisites
• Installation
  • Standalone
  • Plugin Development Environment
• Running Acceptance Tests
  • Configuration
  • Running
  • Provider's Acceptance Specs
• Writing Acceptance Tests
  • Configuring Component Paths
  • Standalone Components
  • Provider Parameterized Components
  • Helpers
    • Context: "acceptance"
    • Matcher: "exit_with"

Prerequisites

• Ruby 2.0.0 or greater

Installation

Standalone

vagrant-spec can be installed as a standalone tool to black-box test existing Vagrant installations (potentially with a combination of plugins). Vagrant plugin developers should see the next section "Plugin Development Environment" to see the proper way to test plugins in development.

To install vagrant-spec as a standalone tool, simply gem install it:

$ gem build vagrant-spec.gemspec
$ gem install vagrant-spec

Plugin Development Environment

If you're developing a Vagrant plugin and wish to use vagrant-spec to help test that plugin, then you should add vagrant-spec as a development dependency to your plugin by modifying the gemspec.

The version of vagrant-spec follows the version of Vagrant. So if you're writing your plugin to target Vagrant 1.4, then you should constrain the plugin to that version:

spec.add_development_dependency "vagrant-spec", "~> 1.4.0"

If you want to use the latest (from git) of vagrant-spec, you should add the git dependency to your Gemfile:

gem 'vagrant-spec', git: "https://github.com/hashicorp/vagrant-spec.git"

vagrant-spec can then be executed by being prefixed with bundle exec. Since your Gemfile should also contain "vagrant", vagrant-spec can then execute Vagrant in the context of the bundler environment in order to test your plugin.

Running Acceptance Tests

This section documents how acceptance tests can be configured, run, and written. Acceptance tests test Vagrant as a black box: they configure some Vagrant environment (by creating a Vagrantfile and perhaps other files), execute Vagrant, and assert some sort of result. They have zero access to the code internals.

vagrant-spec contains a set of acceptance tests that can easily verify the behavior of Vagrant. It also provides a framework for writing new acceptance tests that can easily be executed in the context of Vagrant.

Configuration

Before running the acceptance tests, you must first configure vagrant-spec. Configuration is done by creating the file vagrant-spec.config.rb. It looks like the following:

Vagrant::Spec::Acceptance.configure do |c|
  # ...
end

The available options for c and their defaults are fully documented in the configuration source file. Please read through the documentation thoroughly and implement what you need.

Running

Running the full suite of acceptance tests can be horribly long. It is recommended you only run a component at a time (or a handful of components). To view the list of testable components based on your configuration, run vagrant-spec components. You might see something like the following:

$ vagrant-spec components
cli
provider/virtualbox/basic
provider/virtualbox/provisioner/shell
provider/virtualbox/synced_folder

Choose a component to run, and run it:

$ vagrant-spec test --components cli
...

That component will be tested.

Provider's Acceptance Specs

vagrant-spec ships with a handful of provider acceptance tests used to test Vagrant's built in VirtualBox provider that you can reuse to test your custom plugins.

All that it needs is the following block on your vagrant-spec.config.rb:

Vagrant::Spec::Acceptance.configure do |c|
  # ... other stuff ...
  c.provider 'foobar', box: '<PATH TO A BOX>'
end

With that in place, you should be able to run provider's components with vagrant-spec test. For the complete list, please run vagrant-spec components.

The code example might get outdated while we don't have a stable release, please refer to Vagrant's configuration in case things are not working for you

Writing Acceptance Tests

In addition to running the tests that already exist, vagrant-spec makes it trivial to write new acceptance tests. This is especially useful if you're writing your own plugin.

All new tests must go within their own folder in your project. For the sake of these docs, put your tests in the vagrant-spec folder.

Configuring Component Paths

For vagrant-spec to know about new tests you write, you must add the path that will hold the Ruby files to the configuration "component_paths". These are the paths where the tests are automatically loaded from by vagrant-spec. Example:

Vagrant::Spec::Acceptance.configure do |c|
  c.component_paths << "vagrant-spec"
end

Within the component paths, any files matching *_{output,spec}.rb within any directory are loaded. Components are found based on the "component" key as shown in the sections below.

If the path is relative, it will be relative to the pwd when vagrant-spec is executed.

Standalone Components

If your plugin doesn't depend on a specific provider, then it is known as a component (specifically a standalone one) of vagrant-spec. Create a new file in the vagrant-spec folder of your project with something that looks like the following:

describe "my test", component: "my-test" do
  include_context "acceptance"

  it "can output the version" do
    expect(execute("vagrant", "-v")).to exit_with(0)
  end
end

This should be fairly straightforward RSpec code. The key things are:

• component: "my-test" in the describe definition. This is read by vagrant-spec and will be what is outputted in by vagrant-spec components. Set this to something unique. The actual name of the describe block (in this case "my test") doesn't matter except for output.

• include_context "acceptance" exposes important helpers such as the execute method used. These helpers allow you to execute Vagrant in a completely isolated environment so it doesn't mess up any actual Vagrant state that may be setup on your computer.

And that's it for writing a custom component. You should be able to see it when you run vagrant-spec components and you should also be able to run it.

Provider Parameterized Components

Tests that require a provider (basically any test that executes vagrant up) are known as provider parameterized components. They will be executed for every provider under test.

These are useful, for example, for provisioners. Provisioners should work regardless of what provider is running them. Therefore, if you're testing VirtualBox and VMware, you want to be sure that the "shell" provisioner works, for example.

Provider parameterized components are implemented as RSpec shared examples:

shared_examples "provider/foo" do |provider, options|
  it "does things" do
    puts "Executing for provider: #{provider}"
  end
end

The key things in this are:

• The provider/ prefix in the name of the shared example group. vagrant-spec treats this specially as a provider parameterized component. A new component will automatically be created by vagrant-spec for every provider.

• The |provider, options| parameters to the shared example group. Parameterized components are, as they're named, parameterized. The two parameters they receive are the provider under test as well as any options for that provider.

If you have vagrant-spec configured to test a provider, then when you run vagrant-spec components, you should see your component listed:

$ vagrant-spec components
...
provider/virtualbox/foo
...

And you can now run the tests!

Helpers

Context: "acceptance"

The "acceptance" context defines many helpers for executing Vagrant within an isolated environment and setting up that environment. See the context source file for detailed docs.

The key features of the context are:

• execute is a method for executing commands within the isolated environment. The isolated environment prepares a empty working directory as well as changes the home directory from the point-of-view of executing processes.

• environment is the actual instance of the "IsolatedEnvironment" class. It most importantly allows for skeletons to be used to create complex directory setups. See the IsolatedEnvironment source for more docs.

Matcher: "exit_with"

This matcher can be used to test that a command executed with execute exited with a good exit status. If it didn't, this matcher will produce helpful output containing the stdout/stderr of the command:

result = execute("vagrant", "up")
expect(result).to exit_with(0)

Although, a shortcut for this pattern is assert_execute:

assert_execute("vagrant", "up")

Writing Unit Tests

In addition to having a full acceptance test suite, vagrant-spec contains helpers that can be used to helper unit test plugins for Vagrant. These unit test helpers are test framework agnostic but there are also some helpers for rspec if you use that.

To use the unit tests helpers, just load vagrant-spec/unit somewhere early in your test files:

require "vagrant-spec/unit"