Skip to content

Visual regression testing

Jump to bottom
Sean Moore edited this page Mar 24, 2021 · 4 revisions

Overview

All components that are covered by regression testing have visual regression testing in place along with functional tests. Visual regression checks that elements on the page have:

  • correct positioning
  • correct colors theme
  • have correct state applied (hover, focused or selected states)

Approved screenshots which are used as expected component/element states are called baseline screenshot.

Storing the baseline screenshots

All baseline screenshots are stored in the e2e/wdio/baselineScreenshots folder of the repository.

When the test executes for the first time, baseline screenshots are generated automatically and saved to the e2e/wdio/baselineScreenshots/platform or core/<component_name>/<browser_name> folder.

For all the following executions, new baseline screenshots will not be generated if the baseline screenshot with the same name already exists in this folder. Instead, a comparison of the new screenshot vs the baseline will be performed.

In case the tests fail, two screenshots will be generated, for each comparison action:

  • actual screenshot
  • difference screenshot
    These two types of screenshots will be saved in the .tmp/actual/<browser_name> and .tmp/diff/<browser_name> folders respectively.

All baseline screenshots that are generated for new components need to be committed to the repository along with the e2e test case code.

Cross Browser visual regression

Component/element visual regression baseline screenshots are created for each browser + OS combination. For example if one component is under visual regression and tests are executed for Chrome and FF browsers for Win and MacOs, four separate screenshots will be generated and need to be committed along with the test case code.

Visual regression test cases failure debug.

Visual regression test case indicates the mismatch of the baseline screenshot and actual screenshot (in percentage). If the baseline screenshot and actual screenshot doesn’t match the particular test case will fail with an error of mismatch percentage. (Some test cases needs to be refactored to provide a more meaningful error message.)

In case the visual regression test case is failing:

  • The test case needs to be re-executed from the local environment on the Saucelabs side on the env on which tests case failed.
  • The difference screenshots need to be evaluated by PR owner to determine if the difference is expected or not. In the .tmp/diff folder you will be able to find the screenshot with mismatch highlighted in purple color.

In case the PR introduces unexpected changes, it needs to be fixed without merging any unexpected change to the main branch.
In case the change to the component is expected, the baseline screenshot need to be replaced with the new actual screenshot. (You can do this by moving the screenshot from the .tmp/actual folder to the e2e/wdio/baselineScreenshots/platform or core/<component_name>/<browser_name> folder). The newly added (newly approved baseline) screenshots need to be committed to this PR.

All baseline screenshots need to be generated on the Saucelabs side (the same environment on which actual testing form the CI will be performed). Images generated during the local tests execution can't be used as a baseline screenshots, unless all the environment parameters completely match (OS version, Browser version, screen resolution).

Clone this wiki locally