Automated Testing

We strive to practice Test-Driven Development (TDD) -- that means we try to write our tests first, and then write our code. While we don't always live up to this ideal, we do write automated tests for as much of our back-end code as possible, and in most cases, we do not accept pull requests for code changes that aren't covered by automated tests.

We use RSpec unit tests (located in the spec folder) and Cucumber integration tests (in the features folder).

In addition to running tests on our development environments, we also run them using two continuous integration services:

• GitHub Actions run our tests when a pull request is submitted or merged

Table of Contents Unit Tests Our RSpec Standards Integration Tests Our Cucumber Standards Running Tests Prepare the database Run tests Generate coverage Known Test Failures Pending Tests

Unit Tests

Use RSpec for unit testing Ruby code. Unit tests match to code functionality and not to features, and run quickly. They are helpful to ensure that you're not changing behavior accidentally when refactoring or adding new functionality, and that your code is working the way you'd expect it to.

Writing unit tests makes it easier to catch mistakes before committing them and reduces the burden on our human testing volunteers.

Resources on writing good tests with RSpec:

• Everyday Rails Testing with RSpec
• Better Specs
• RSpec Style Guide
• Structure of RSpec Tests Slides

Our RSpec Standards

• Use double quotes for strings
• Avoid "should" when describing behavior, e.g. it "has a valid subject line" rather than it "should have a valid subject line"
• Use create(:comment) rather than FactoryBot.create(:comment) for factories
• Use the Redirect Expectation Helper when testing redirects with flash messages

Integration Tests

We write specification in Cucumber. These specify behavior and are easy to understand if you're not a developer.

They run much more slowly than RSpec; don't introduce too many unnecessary steps, and be mindful of unnecessary database calls!

Resources on writing good tests with Cucumber:

• Write Great Cucumber Tests
• Writing Better Cucumber Scenarios; or, Why We're Deprecating FactoryGirl's Cucumber Steps

Our Cucumber Standards

• Prefer user-facing field labels to input IDs or names, e.g. When I fill in "Additional Tags" with "Tag Name" rather than When I fill in "work_freeforms" with "Tag Name"
  • Input IDs or names are fine in step definitions, e.g. fill_in("work_freeforms", with: "Tag Name")
• Prefer factories to fixtures
• Prefer Cucumber expressions to regular expressions in step definitions
  • For existing step definitions using regular expressions, prefer "(.*?)" to "([^"]*)"

Running Tests

Our test suite can take a very long time to run even on fast computers, so when you need to run the entire suite, we recommend pushing your branch and manually running the Automated Tests workflow using GitHub Actions. GitHub Actions runs multiple test groups concurrently, so it's much faster.

However, when you only need to run a few tests, it can be more convenient to do that on your own computer.

The following instructions will help you get your tests running on a local install. If you are using Docker, you should run the tests inside the test container since it has the code directory mounted already and all the right dependencies installed.

Prepare the database

Before you run the automated tests, you may need to prepare your test database. This is usually only necessary the first time you run tests or after running a migration:

  bundle exec rake db:test:prepare

Failing that, you can recreate the test database altogether:

  mysql -e "drop database otwarchive_test;"
  mysql -e "create database otwarchive_test DEFAULT CHARACTER SET utf8mb4 DEFAULT COLLATE utf8mb4_unicode_ci;"
  RAILS_ENV=test bundle exec rake db:schema:load
  RAILS_ENV=test bundle exec rake db:migrate

Run tests

• Run all RSpec tests: RAILS_ENV=test bundle exec rspec spec
• Run a specific RSpec test: RAILS_ENV=test bundle exec rspec spec/my_model_spec.rb
• Run all Cucumber tests: RAILS_ENV=test bundle exec cucumber features
• Run a specific Cucumber test: RAILS_ENV=test bundle exec cucumber features/my_archive_process.feature
• Run a specific scenario within a Cucumber test: RAILS_ENV=test bundle exec cucumber features/my_archive_process.feature:10, where :10 is the line number of the scenario
• See verbose output: add --format pretty (more info)

Generate coverage

Whenever you run a test, coverage results will be at coverage/index.html, which can be viewed in a browser.

You should also check out Codecov, if you need to generate coverage for the entire test suite.

Known Test Failures

Tests can fail intermittently, particularly when run on CI services. If any tests fail frequently enough, we'll document them here. Feel free to make issues and submit pull requests for these if you have the proper Jira permissions and an issue isn't already listed.

Currently there are no known flaky tests.

Pending Tests

Mark failing tests pending if a test is written but isn't working for known reasons, or if you need to merge something urgently and don't want to break the build.

In RSpec, do this using xit:

xit "should only get works under that tag" do
  get :index, tag_id: @fandom.name
  expect(assigns(:works).items).to include(@work)
  expect(assigns(:works).items).not_to include(@work2)
end 

In Cucumber, use the When /^"([^\"]*)" is fixed$/ step and comment out any failing steps:

Scenario: Post Without Preview
  Given I am logged in as "whoever" with password "whatever"
    And I add the work "public" to series "be_public"
    And I follow "be_public"
    And "Issue 2169" is fixed
  # Then I should not see the "title" text "Restricted" within "h2"