Automated Testing

Why Automated Testing?

Nobody likes when software is broken, but it's a really unpleasant feeling to know you broke the software for others. As a first line of defense against broken software, ChurchCRM uses automated, browser based testing. We've developed a set of scripts that emulates a real user by clicking and typing in strategic locations within the application to ensure that all of the features work as intended.

We've automated testing of some of the critical functionality. This way, components are tested every time any code is changed.

Details

There are many different automated testing strategies (unit testing, integration testing, etc). As ChurchCRM matures, so will our testing strategies; however, we're currently only using browser automation testing. This form of testing gives us the largest amount of code coverage for the smallest amount of testing code.

Toolset

• Behat Behat is a PHP framework for driving the various browser automation test drivers

• MinkExtension MinkExtension provides additional capabilities to Behat

• Gherkin Gherkin is the language used to write the browser automation tests. It is considered an External Domain Specific Language because it allows a QA developer to write a test for ChurchCRM without any PHP knowledge. The tests read as "plain English" Syntax Examples: https://gist.github.com/mnapoli/5848556

• SauceLabs SauceLabs generously provides us with hosting of a Slenium Grid. Think: a server farm running a bunch of web browsers, clicking on things auto-magically.

• Selenium Selenium is an open source project that allows QA developers to interact with browsers in a scripted, automated manner.

https://github.com/minkphp/MinkSelenium2Driver http://docs.behat.org/en/v2.5/guides/7.config.html https://michaelheap.com/behat-selenium2-webdriver-with-minkextension/ http://mink.behat.org/en/latest/drivers/selenium2.html

Tests

Page Tests

ChurchCRM uses Behat for page load testing Behavior Driven Development.

Running Behat Page Tests

1. While SSH'd into the Vagrant development box, run the following commands:

cd /vagrant
npm test

2. You should see the result of all individual tests.

Writing Behat Page Tests

Behat page tests use the gherkin syntax.

Behat uses tests written with this syntax to execute requests and validate responses against the defined pages.

All tests should follow this outline:

Feature: Calendar
  In order to see calendar events
  As a User
  I am able to visit the calendar

  Scenario: Open the calendar
    Given I am authenticated as "admin" using "changeme"
    And  I am on "/calendar.php"
    Then I should see "Church Calendar"

Useful Reference Tests

• Supplying input values, Navigating Pages: FamilyAdd.Feature
• Defining new gherkin verbs: iAmAuthenticatedAs: FeatureContext.php

Unit Tests

Unit testing is not yet fully implemented.

View Test Results

Accessing the TravisCI job

On the bottom of the PR for which you want to view test results, click "Details" next to the appropriate test:

Access to the TravisCI tests for a specific commit is also possible by clicking the "Test Status Indicator" to the right of the commit:

Viewing Job Details

1. Click on "Details"

2. Click one of the builds

3. Review the logs. The bottom of the log will contain Selenium test results:

Restarting a job

From the TravisCI job page, click the "Restart Job" button for either all jobs, or a specific job:

Links

• https://saucelabs.com/blog/adding-sauce-to-behat
• https://github.com/jaffamonkey/behat-3-kickstart
• https://docs.travis-ci.com/user/sauce-connect/
• http://mink.behat.org/en/latest/guides/session.html
• https://gist.github.com/mnapoli/5848556
• http://www.tentacode.net/10-tips-with-behat-and-mink
• http://elnur.pro/use-the-domain-language-in-bdd-features/