-
Notifications
You must be signed in to change notification settings - Fork 102
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Sauce Visual Baseline Overrides (#3022)
* update Python examples with new params * baseline overrides docs page * add baseline override docs * update video --------- Co-authored-by: Logan Graham <logan.graham@saucelabs.com>
- Loading branch information
1 parent
66914c0
commit f859284
Showing
10 changed files
with
290 additions
and
9 deletions.
There are no files selected for viewing
27 changes: 27 additions & 0 deletions
27
docs/visual-testing/_partials/_baseline-overrides-csharp.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,27 @@ | ||
| At the global / client level | ||
|
|
||
| ```csharp | ||
| VisualClient = await VisualClient.Create(Driver, Region.UsWest1); | ||
| VisualClient.BaselineOverride = new BaselineOverride() | ||
| { | ||
| Browser = Browser.Chrome, | ||
| Device = "Desktop (1024x627)", | ||
| OperatingSystem = OperatingSystem.Windows, | ||
| OperatingSystemVersion = "10", | ||
| }; | ||
| ``` | ||
|
|
||
| Or at the snapshot level | ||
|
|
||
| ```csharp | ||
| await VisualClient.VisualCheck("Login Page", new VisualCheckOptions() | ||
| { | ||
| BaselineOverride = new BaselineOverride() | ||
| { | ||
| Browser = Browser.Chrome, | ||
| Device = "Desktop (1024x627)", | ||
| OperatingSystem = OperatingSystem.Windows, | ||
| OperatingSystemVersion = "10", | ||
| } | ||
| }); | ||
| ``` |
18 changes: 18 additions & 0 deletions
18
docs/visual-testing/_partials/_baseline-overrides-python-robot.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,18 @@ | ||
| At the global / client level | ||
|
|
||
| ```robot | ||
| *** Keywords *** | ||
| # Typically you'd set this up in your __init__.robot file at the top-level, right after creating your visual build. | ||
| Setup | ||
| ${override} = Visual BaselineOverride browser=CHROME device=Desktop (1024x627) operating_system=WINDOWS operating_system_version=10 | ||
| Visual Set Global BaselineOverride ${override} | ||
| ``` | ||
|
|
||
| Or at the snapshot level | ||
|
|
||
| ```robot | ||
| *** Test Cases *** | ||
| Valid Login | ||
| ${override} = Visual BaselineOverride browser=CHROME device=Desktop (1024x627) operating_system=WINDOWS operating_system_version=10 | ||
| Visual Snapshot Login Page baseline_override=${override} | ||
| ``` |
33 changes: 33 additions & 0 deletions
33
docs/visual-testing/_partials/_baseline-overrides-python.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,33 @@ | ||
| At the global / client level | ||
|
|
||
| ```python | ||
| from saucelabs_visual.client import SauceLabsVisual | ||
| from saucelabs_visual.typing import BaselineOverride, Browser, OperatingSystem | ||
|
|
||
| visual_client = SauceLabsVisual() | ||
| visual_client.baseline_override = BaselineOverride( | ||
| browser=Browser.CHROME, | ||
| device="Desktop (1024x627)", | ||
| operatingSystem=OperatingSystem.WINDOWS, | ||
| operatingSystemVersion="10", | ||
| ) | ||
| ``` | ||
|
|
||
| Or at the snapshot level | ||
|
|
||
| ```python | ||
| from saucelabs_visual.client import SauceLabsVisual | ||
| from saucelabs_visual.typing import BaselineOverride, Browser, OperatingSystem | ||
| # ... | ||
| visual_client = SauceLabsVisual() | ||
| visual_client.create_snapshot_from_webdriver( | ||
| "Login Page", | ||
| driver=driver, | ||
| baseline_override=BaselineOverride( | ||
| browser=Browser.CHROME, | ||
| device="Desktop (1024x627)", | ||
| operatingSystem=OperatingSystem.WINDOWS, | ||
| operatingSystemVersion="10", | ||
| ) | ||
| ) | ||
| ``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,38 @@ | ||
| At the global / service level | ||
|
|
||
| ```js | ||
| import { Browser, OperatingSystem } from '@saucelabs/visual'; | ||
| export const config = { | ||
| // ... | ||
| services: [ | ||
| 'sauce', | ||
| [ | ||
| '@saucelabs/wdio-sauce-visual-service', | ||
| { | ||
| baselineOverride: { | ||
| browser: Browser.Chrome, | ||
| device: "Desktop (1024x627)", | ||
| operatingSystem: OperatingSystem.Windows, | ||
| operatingSystemVersion: '10', | ||
| }, | ||
| }, | ||
| ], | ||
| ], | ||
| } | ||
| ``` | ||
|
|
||
| Or at the snapshot level | ||
|
|
||
| ```js | ||
| import { Browser, OperatingSystem } from '@saucelabs/visual'; | ||
| // ... | ||
| // Passing on a per-snapshot level | ||
| await browser.sauceVisualCheck('Inventory Page', { | ||
| baselineOverride: { | ||
| browser: Browser.Chrome, | ||
| device: "Desktop (1024x627)", | ||
| operatingSystem: OperatingSystem.Windows, | ||
| operatingSystemVersion: '10', | ||
| } | ||
| }); | ||
| ``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,160 @@ | ||
| --- | ||
| sidebar_label: Cross Browser / OS | ||
| --- | ||
|
|
||
| import useBaseUrl from '@docusaurus/useBaseUrl'; | ||
| import Tabs from '@theme/Tabs'; | ||
| import TabItem from '@theme/TabItem'; | ||
| import BaselineOverridesWDIO from '../_partials/_baseline-overrides-wdio.md'; | ||
| import BaselineOverridesCSharp from '../_partials/_baseline-overrides-csharp.md'; | ||
| import BaselineOverridesPython from '../_partials/_baseline-overrides-python.md'; | ||
| import BaselineOverridesPythonRobot from '../_partials/_baseline-overrides-python-robot.md'; | ||
|
|
||
| # Cross Browser / OS Visual Testing | ||
|
|
||
| When creating a new Snapshot in Sauce Visual we look up the previous snapshot for a configuration using our [baseline matching](/visual-testing/#baseline-matching) behavior. This generally means that running the same tests with the same device configuration (OS, browser, window size, etc.) will find the last snapshot taken with that setup to compare against. | ||
|
|
||
| Using the 'Baseline Overrides' feature allows overriding specific, or all, keys during baseline matching. Using these options you can test visual snapshots across devices, operating systems, and browsers. | ||
|
|
||
| <figure> | ||
| <img src={useBaseUrl('/img/sauce-visual/cross-browser.png')} alt="Comparing a Chrome baseline vs a Firefox snapshot"/> | ||
| <figcaption style={{textAlign: 'center'}}>Comparing a Chrome baseline vs a Firefox snapshot</figcaption> | ||
| </figure> | ||
|
|
||
| ## Usage | ||
|
|
||
| Below are some general steps for getting up and running using cross browser / OS comparisons: | ||
|
|
||
| ### 1. Configure your browsers | ||
|
|
||
| To get the most out of cross browser comparison, you'll want your browser viewports to be equal in dimensions. We recommend doing this in a `before` hook provided by your framework right after the driver is initialized. Below are some examples in a few frameworks with a snippet for applying a uniform value across tests. | ||
|
|
||
| :::note NOTE: Setting Browser Dimensions | ||
| A browser window cannot be larger than the screen requested in a session, and will often have to be a few hundred pixels smaller in order to account for OS panels, status bars and docks. We recommend targeting a height of ~300px smaller than the smallest requested resolution capability. | ||
| ::: | ||
|
|
||
| For example with the following capabilities tested against: | ||
|
|
||
| - Chrome on Windows 11 at 2560x1600 | ||
| - Firefox on Windows 11 at 2560x1600 | ||
| - Safari on macOS 13 at 2048x1536 | ||
|
|
||
| You may want to target a browser height of ~1200px. We'll be using 1920x1200 as our target for the examples below to ensure all browsers can reach the requested size. | ||
|
|
||
| <Tabs | ||
| defaultValue="wdio" | ||
| values={[ | ||
| { label: 'WDIO', value: 'wdio' }, | ||
| { label: 'C#', value: 'csharp' }, | ||
| { label: 'Python', value: 'python' }, | ||
| { label: 'Python (Robot)', value: 'robot' }, | ||
| ]} | ||
| > | ||
| <TabItem value="wdio"> | ||
| ```js | ||
| // Adding this to the 'before' hook in your 'wdio.conf.ts' | ||
| { | ||
| // ... The rest if your wdio configuration | ||
| before: async (_capabilities, _specs) => { | ||
| // Resize all windows to small, static size | ||
| await browser.setWindowRect(0, 0, 800, 600); | ||
| // Calculate the difference between the window outer and inner dimensions (to account | ||
| // for browser UI elements, etc. | ||
| const [width, height] = await driver.executeScript(` | ||
| return [window.outerWidth - window.innerWidth + arguments[0], | ||
| window.outerHeight - window.innerHeight + arguments[1]]; | ||
| `, [1920, 1200]); | ||
| await browser.setWindowRect(0, 0, width, height); | ||
| }, | ||
| } | ||
| ``` | ||
| </TabItem> | ||
| <TabItem value="csharp"> | ||
| ```csharp | ||
| // In a before / Setup of a fixture following driver initialization | ||
| Driver.Manage().Window.Size = new Size(800, 600); | ||
| var dims = (ReadOnlyCollection<object>) Driver.ExecuteScript( | ||
| "return [window.outerWidth - window.innerWidth + arguments[0], window.outerHeight - window.innerHeight + arguments[1]];", | ||
| [1920, 1200] | ||
| ); | ||
| Driver.Manage().Window.Size = new Size(Convert.ToInt32(dims[0]), Convert.ToInt32(dims[1])); | ||
| ``` | ||
|
|
||
| [//]: # (</object> commented out, but here to prevent mdx parsing in IDE) | ||
| </TabItem> | ||
| <TabItem value="python"> | ||
| ```python | ||
| # Call after your driver is initialized in your test setup | ||
| driver.set_window_size(800, 600) | ||
| [width, height] = driver.execute_script( | ||
| 'return [window.outerWidth - window.innerWidth + arguments[0], window.outerHeight - ' | ||
| 'window.innerHeight + arguments[1]];', | ||
| 1920, | ||
| 1200, | ||
| ) | ||
| driver.set_window_size(width, height) | ||
| ``` | ||
| </TabItem> | ||
| <TabItem value="robot"> | ||
| ```robot | ||
| # Recommended to set in your global setup after opening your browser | ||
| Setup | ||
| Set Window Size 800 600 | ||
| ${dims} = Execute Javascript return [window.outerWidth - window.innerWidth + 1920, window.outerHeight - window.innerHeight + 1200]; | ||
| Set Window Size ${dims}[0] ${dims}[1] | ||
| ``` | ||
| </TabItem> | ||
| </Tabs> | ||
|
|
||
| ### 2. Set a baseline | ||
|
|
||
| With your new configuration use a binding & test framework of your choice create a build for your target browser. For example, if you want to compare all browsers against Chrome you would run a single build with Chrome as the browser under test. When complete, approve any / all snapshots that are a part of that build. This will give us a full set of baselines to compare other browsers against. | ||
|
|
||
| ### 3. Configure baseline overrides | ||
|
|
||
| With the baselines of your target browser saved gather the keys of the baseline run you wish to compare against. You can do this in the UI by navigating into a build, selecting a diff, viewing the snapshot meta, and clicking 'Copy as JSON.' | ||
|
|
||
| <video style={{width: '100%', height: 'auto'}} autoPlay loop title="Copy baseline hash keys"> | ||
| <source src={useBaseUrl('/img/sauce-visual/snapshot-metadata.webm')} /> | ||
| </video> | ||
|
|
||
| For cross browser / OS testing you will likely want the following: | ||
|
|
||
| - `operatingSystem` | ||
| - `operatingSystemVersion` | ||
| - `browser` | ||
| - `device` | ||
|
|
||
| Use the baseline override feature of your binding (see the documentation for your specific binding on the sidebar). A few examples are included below for reference: | ||
|
|
||
| <Tabs | ||
| defaultValue="wdio" | ||
| values={[ | ||
| { label: 'WDIO', value: 'wdio' }, | ||
| { label: 'C#', value: 'csharp' }, | ||
| { label: 'Python', value: 'python' }, | ||
| { label: 'Python (Robot)', value: 'robot' }, | ||
| ]} | ||
| > | ||
| <TabItem value="wdio"> | ||
| <BaselineOverridesWDIO /> | ||
| </TabItem> | ||
| <TabItem value="csharp"> | ||
| <BaselineOverridesCSharp /> | ||
| </TabItem> | ||
| <TabItem value="python"> | ||
| <BaselineOverridesPython /> | ||
| </TabItem> | ||
| <TabItem value="robot"> | ||
| <BaselineOverridesPythonRobot /> | ||
| </TabItem> | ||
| </Tabs> | ||
| ### 4. Run your build | ||
|
|
||
| Run a new build for each browser you wish to compare with and optionally approve the snapshots for each browser to allow them to take over their own baselines. | ||
|
|
||
| ## Limitations | ||
|
|
||
| - Scrollbars on Windows generally take up space which can cause invalid / incorrect viewport dimensions for snapshots. You can use workarounds such as [setting the scrollbar width](https://developer.mozilla.org/docs/Web/CSS/scrollbar-width) to disable them temporarily. We'll do our best to do this automatically while using a Sauce Visual runner. | ||
| - Browser dimensions need to be the same for the best results. While having a uniform width can cover most cases, having a uniform height will ensure that pages which center content (and do not have enough content to scroll) will match enough to be visually diffed. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file not shown.