Helper for resemble.js, used for image comparison in tests with Playwright, Webdriver, Puppeteer, Appium, TestCafe!
@currys-co-uk/codeceptjs-resemblehelper is a CodeceptJS helper which can be used to compare screenshots and make the tests fail/pass based on the tolerance allowed.
If two screenshot comparisons have difference greater then the tolerance provided, the test will fail.
NPM package: https://www.npmjs.com/package/@currys-co-uk/codeceptjs-resemblehelper
To install the package, just run npm install @currys-co-uk/codeceptjs-resemblehelper.
This helper should be added in codecept.json/codecept.conf.js
Example:
{
helpers: {
ResembleHelper: {
require: "@currys-co-uk/codeceptjs-resemblehelper",
baseFolder: "./tests/screenshots/base/",
diffFolder: "./tests/screenshots/diff/",
prepareBaseImage: true,
tolerance: 10,
skipFailure: true,
createDiffInToleranceRange: true,
alwaysSaveDiff: true,
createSubFoldersInBaseFolder: true,
updateMismatchedBaseImage: true,
ignoreNothing: true,
scaleToSameSize: true
}
}
}To use the Helper, users may provide the parameters:
baseFolder: Mandatory. This is the folder for base images, which will be used with screenshot for comparison.
diffFolder: Mandatory. This will the folder where resemble would try to store the difference image, which can be viewed later.
prepareBaseImage: Optional. When true then the system replaces all of the baselines related to the test case(s) you ran. This is equivalent of setting the option prepareBaseImage: true in all verifications of the test file.
If this parameter is missing in .conf file, value is undefined.
tolerance: Optional. When value is present, system sets tolerance for all tests, which does not have it set. Set tolerance in tests has always higher priority, than global tolerance in config.
skipFailure: Optional. When true is set, Resemble helper continue in test run even in the case of detected mismatch and returns warning to console.
createDiffInToleranceRange: Optional, when true is set, diff is created only if mismatch is in range of set tolerance (mismatch is not 0 but is less than tolerance).
alwaysSaveDiff: Optional. When set as true diff image is created in every case.
createSubFoldersInBaseFolder: Optional. When true is set, for every base image is created sub folder in /base folder. Sub folder is named as first 50 chars of test Scenario title.
updateMismatchedBaseImage: Optional. When true, existing base images, which meet condition mismatch > tolerance, are updated with actual screenshot of web site, tolerance is then set to 0 and diff with mismatch is not saved to folder.
Use only if base images exists. Parameter is not compatible with prepareBaseImage.
ignoreNothing: Optional. When true, resemblejs covers 100% comparison of compared images and ignores nothing (e.g. similar colors) in found mismatch - every small change, every difficult pixel is resolved as a mismatch.
scaleToSameSize: Optional. When true, resemblejs compares different sizes of the same image.
These are the major functions that help in visual testing:
First one is the seeVisualDiff which basically takes two parameters
baseImageName of the base image, this will be the image used for comparison with the screenshot image. It is mandatory to have the same image file names for base and screenshot image.optionsoptions can be passed which includeprepareBaseImageandtolerance.
/**
* Check Visual Difference for Base and Screenshot Image
* @param baseImage Name of the Base Image (Base Image path is taken from Configuration)
* @param options Options ex {prepareBaseImage: true, tolerance: 5} along with Resemble JS Options, read more here: https://github.com/rsmbl/Resemble.js
* @returns {Promise<void>}
*/
async seeVisualDiff(baseImage, options) {}Second one is the seeVisualDiffForElement which basically compares elements on the screenshot, selector for element must be provided.
It is exactly same as seeVisualDiff function, only an additional selector CSS|XPath|ID locators is provided
/**
* See Visual Diff for an Element on a Page
*
* @param selector Selector which has to be compared, CSS|XPath|ID
* @param baseImage Base Image for comparison
* @param options Options ex {prepareBaseImage: true, tolerance: 5} along with Resemble JS Options, read more here: https://github.com/rsmbl/Resemble.js
* @returns {Promise<void>}
*/
async seeVisualDiffForElement(selector, baseImage, options){}Note:
seeVisualDiffForElementonly works when the page for baseImage is open in the browser, so that WebdriverIO can fetch coordinates of the provided selector.
Third one is the screenshotElement which basically takes screenshot of the element. Selector for the element must be provided. It saves the image in the output directory as mentioned in the config folder.
I.screenshotElement('selectorForElement', 'nameForImage');Finally to use the helper in your test, you can write something like this:
Feature('to verify monitoried Remote Db instances');
Scenario('Open the System Overview Dashboard', async (I, adminPage, loginPage) => {
adminPage.navigateToDashboard('OS', 'System Overview');
I.saveScreenshot('Complete_Dashboard_Image.png');
adminPage.applyTimer('1m');
adminPage.viewMetric('CPU Usage');
I.saveScreenshot('Complete_Metric_Image.png');
});
Scenario('Compare CPU Usage Images', async (I) => {
// setting tolerance and prepareBaseImage in the options object
I.seeVisualDiff('Complete_Metric_Image.png', { prepareBaseImage: false, tolerance: 5 });
// passing a selector, to only compare that element on both the images now
// We need to navigate to that page first, so that webdriver can fetch coordinates for the selector
adminPage.navigateToDashboard('OS', 'System Overview');
I.seeVisualDiffForElement("//div[@class='panel-container']", 'Complete_Dashboard_Image.png', { prepareBaseImage: false, tolerance: 3 });
});Note:
seeVisualDiffandseeVisualDiffElementwork only when the dimensions of the screenshot as well as the base image are same so as to avoid unexpected results.
Is it needed to have prepared base image before and only compare it, or create new base image? It depends how for every test is set:
prepareBaseImageparameter in configoptionsin test (e.g.I.seeVisualDiff("image.png", {prepareBaseImage: true}))- does base image already exists, or is missing? Note: Every state(rule/condition) is valid for every one instance of base image
| state | config | options | base image already exists? | behavior |
| ----- | ---------- | --------- | -------------------------- | ---------------------- |
| 1 | undefined | undefined | yes | compare |
| 2 | undefined | undefined | no | create new base image |
| 3 | undefined | true | yes | create new base image |
| 4 | undefined | true | no | create new base image |
| 5 | undefined | false | yes | compare |
| 6 | undefined | false | no | Error |
| ----- | ---------- | --------- | -------------------------- | ---------------------- |
| 7 | true | undefined | yes | create new base image |
| 8 | true | undefined | no | create new base image |
| 9 | true | true | yes | create new base image |
| 10 | true | true | no | create new base image |
| 11 | true | false | yes | compare |
| 12 | true | false | no | Error |
| ----- | ---------- | --------- | -------------------------- | ---------------------- |
| 13 | false | undefined | yes | compare |
| 14 | false | undefined | no | Error |
| 15 | false | true | yes | create new base image |
| 16 | false | true | no | create new base image |
| 17 | false | false | yes | compare |
| 18 | false | false | no | Error |
| ----- | ---------- | --------- | -------------------------- | ---------------------- |You can also exclude part of the image from comparison, by specifying the excluded area in pixels from the top left.
Just declare an object and pass it in options as ignoredBox:
const box = {
left: 0,
top: 10,
right: 0,
bottom: 10,
};
I.seeVisualDiff('image.png', { prepareBaseImage: true, tolerance: 1, ignoredBox: box });After this, that specific mentioned part will be ignored while comparison.
This works for seeVisualDiff and seeVisualDiffForElement.
Similar as ignored box, when specific element is excluded from image comparison. You need to pass it in options as ignoredElement.
Scenario('Ignore element for screenshot visual diff', async ({ I }) => {
// I.amOnPage('https://the-internet.herokuapp.com/context_menu');
// I.saveScreenshot('IGNORED.png');
await I.seeVisualDiff('IGNORED.png', { ignoredElement: '#hot-spot' });
});After this, specific element will be ignored while comparison.
This works for seeVisualDiff & seeVisualDiffForElement.
Similar as Ignored Box, but with more excluded parts of the image comparison specifying the excluded area in pixels from top left.
const box1 = {
left: 100,
top: 200,
right: 300,
bottom: 600,
};
const box2 = {
left: 400,
top: 100,
right: 500,
bottom: 600,
};
await I.seeVisualDiff('image.png', { ignoredBoxes: [box1, box2] });After this, that specific mentioned parts will be ignored while comparison.
This works for seeVisualDiff.
Similar as Ignored element, but it possible to specific more elements, which should be excluded from image comparison.
Scenario('Ignore 2 elements for screenshot visual diff', async ({ I }) => {
//I.amOnPage('https://the-internet.herokuapp.com/context_menu');
//I.saveScreenshot('image.png');
await I.seeVisualDiff('image.png', { ignoredElements: ['#hot-spot', '#page-footer'] });
});After this, that specific mentioned parts will be ignored while comparison.
This works for seeVisualDiff & seeVisualDiffForElement.
Note: DON'T use combination of ignored box/boxes/element/elements together, ALWAYS use only one of them as options !!!
Functionality element selection is similar as querySelectorAll in DOM, ignoredQueryElementAll will find all identical/query suitable elements on the page and ignore them.
Scenario('Ignore all same elements', async ({ I }) => {
//I.amOnPage('https://the-internet.herokuapp.com');
//await I.saveScreenshot('image.png');
await I.seeVisualDiff('image.png', { ignoredQueryElementAll: '//ul/li/a' });
});This works for seeVisualDiff & seeVisualDiffForElement.
Default logic of creating diff images need to have greater mismatch, than tolerance.
With set createDiffInToleranceRange as true in config you can affect generating diff images, if mismatch is in range of set tolerance (mismatch is not 0 but is less than tolerance).
createDiffInToleranceRange cover only this condition.
E.g.
{
helpers: {
ResembleHelper : {
...
createDiffInToleranceRange: true
...
}
}
}Options tolerance is 5, and test result mismatch was 2,4. -> diff image is created.
With set alwaysSaveDiff as true in config you can affect generating diff images, it covers all conditions and always returns generated diff image, doesn't matter if test passed or failed.
E.g.
{
helpers: {
ResembleHelper : {
...
alwaysSaveDiff: true
...
}
}
}With set createSubFoldersInBaseFolder as true in the config you can affect creating sub folders for base images. This flag is helpful if you have a lot of base images (e.g. more than 200) and /baseFolder is not providing an easy survey for your eyes.
For sub folder name in /baseFolder is used the first 50 chars of test Scenario title. Sub folder name is also correctly parsed for disallowed chars in Windows/Mac.
E.g.
{
helpers: {
ResembleHelper : {
//...
baseFolder: "./tests/screenshots/base/",
//...
createSubFoldersInBaseFolder: true
//...
}
}
}In test should look like this:
Scenario('creates my first base image for first visual testing', async ({ I }) => {
//I.amOnPage('https://the-internet.herokuapp.com');
//await I.saveScreenshot('image.png');
await I.seeVisualDiff('image.png');
});So, what happened? In base folder ("./tests/screenshots/base/") is created sub folder with name /creates_my_first_base_image_for_first_v (with max 50 first chars) and into them is saved file image.png.
With updateMismatchedBaseImage as true in config you can update existing base images with mismatch. It is needed to meet condition mismatch > tolerance, then existing base image is updated with actual screenshot of web site.
Tolerance is then set to 0 and diff with mismatch is not saved to folder. Use only if base images exists.
Note: Parameter is not compatible with
prepareBaseImageand doesn't replace it.
{
helpers: {
ResembleHelper : {
...
updateMismatchedBaseImage: true
...
}
}
}With ignoreNothing as a true, resemblejs covers 100% comparison of compared images and ignores nothing (e.g. similar colors) in found mismatch - every small change, every difficult pixel is resolved as a mismatch.
If flag is not set, or has false value, resemblejs uses ignore less as default, that means red, blue, green, alpha, minBrightness have 16 tolerance intensity of the color and maxBrightness has 240 tolerance intensity.
That covers very similar colors but in specific range.
{
helpers: {
ResembleHelper : {
...
ignoreNothing: true
...
}
}
}With scaleToSameSize as a true resemblejs compares images even in the case of different images size. If you don't use this flag, it compares only images with the same size as a standard way.
{
helpers: {
ResembleHelper : {
...
scaleToSameSize: true
...
}
}
}You can set further output settings used by resemble.js. Declare an object specifying them and pass it in the options as outputSettings:
const outputSettings = {
ignoreAreasColoredWith: { r: 250, g: 250, b: 250, a: 0 },
// read more here: https://github.com/rsmbl/Resemble.js
};
I.seeVisualDiff('image.png', { prepareBaseImage: true, tolerance: 1, outputSettings: outputSettings });Refer to the resemble.js documentation for available output settings.
You can avoid the test fails for a given threshold but yet generates the difference image.
Just declare an object and pass it in options as skipFailure:
I.seeVisualDiff("image.png", {prepareBaseImage: true, tolerance: 1, skipFailure: true});
or as global in config:
{
helpers: {
ResembleHelper : {
...
skipFailure: true
...
}
}
}After this, the system generates the difference image but does not fail the test.
This works for seeVisualDiff and seeVisualDiffForElement.
Allure reports may also be generated directly from the tool. To do so, add
"plugins": {
"allure": {}
}
in the config file.
The attachments will be added to the report only when the calulated mismatch is greater than the given tolerance.
Set output to where the generated report is to be stored. Default is the output directory of the project.
AWS S3 support to upload and download various images is also provided.
It can be used by adding the aws code inside "ResembleHelper" in the "helpers" section in config file. The final result should look like:
{
helpers: {
ResembleHelper: {
require: "@currys-co-uk/codeceptjs-resemblehelper",
baseFolder: "<location of base folder>",
diffFolder: "<location of diff folder>",
aws: {
accessKeyId: "<Your AccessKeyId>",
secretAccessKey: "<Your secretAccessKey>",
region: "<Region of Bucket>",
bucketName: "<Bucket Name>"
}
}
}
}When this option has been provided, the helper will download the base image from the S3 bucket.
This base image has to be located inside a folder named "base".
The resultant output image will be uploaded in a folder named "output" and diff image will be uploaded to a folder named "diff" in the S3 bucket.
If the prepareBaseImage option is marked true, then the generated base image will be uploaded to a folder named "base" in the S3 bucket.
Note: The tests may take a bit longer to run when the AWS configuration is provided as determined by the internet speed to upload/download images.
Issue in Windows where the image comparison is not carried out, and therefore no Mismatch Percentage is shown. See 'loadImageData' function in resemble.js