feat: replace visual test driver to jest-image-snapshot and Firefox (…

…Playwright) (#1945) * feat: replace jest visual test driver to jest-image-snapshot * chore: add .jest-image-snapshot-touched-files to .gitignore * chore: remove outdated deps update limitation * chore: refactor/align script commands * chore: remove screenshot as a part of the description * chore: remove all old snapshots * feat: ensure page and ScrollView is not scrolling * feat: refactor setupPageScreenshot * chore: rename testPageScreenshot to makeScreenshot * chore: upload diff images to GitHub Action as an artifact * chore: convert from puppeteer to playwright Background: I tried these versions of puppeteer: 8 to 15 + 19 Version 8, 9 and 10 did work fine. While version 10.2.0 has a change in its method of calculating page offsets. Here is it [better explained](puppeteer/puppeteer#7514 (comment)). This change leads to a negative side effect so snapshots differ from when run on CI macOS vs local macOS. A typical comparison looks then as so: ![avatar-src-have-to-match-png-image-of-local-src snap-diff](https://user-images.githubusercontent.com/1501870/213706793-3b41d185-0782-4088-ba72-7d5eb7b54b94.png) I also see that "jest-image-snapshot" has not so many "parameters" to tweak. We basically have `failureThreshold`. Before we had [several more](https://github.com/dnbexperience/eufemia/blob/88279d1e6903497fce40d6c4322dd9b7c7b9ac3b/packages/dnb-eufemia/src/core/jest/jestSetupScreenshots.js#L39). When running our local generated snapshots on the CI, we typical get this: ![button-icon-have-to-match-icon-button snap-diff](https://user-images.githubusercontent.com/1501870/214040509-b0345bed-0d88-4653-93df-e7f650524bd8.png) * chore: align visual test and update snapshots Align Dialog Align Paragraph Align Anchor (new) Align Hr Align Lists Align Typography Align Heading Align Autocomplete Align Avatar Align Badge Align Breadcrumb Align Button Align Checkbox Align Drawer Align Dropdown Align FormLabel Align FormRow and FormSet Align FormStatus Align GlobalError Align GlobalStatus Align HelpButton Align Icon Algin InfoCard Align Input Align Modal Align NumberFormat Align Pagination Align ProgressIndicator Align Radio button Align Seciton Align Skeleton Align Slider Align Space Align StepIndicator Align switch Align Tabs Align Tag Align Textarea Align Timeline Align ToggleButton Align Tooltip Align Upload Align VisuallyHidden Align Image Align Blockquote Align DrawerList Align DatePicker Align Table Align PaymentCard Align HelperClasses Align Logo s
dnbexperience · May 31, 2023 · 4486fb3 · 4486fb3
1 parent aa367db
commit 4486fb3
Show file tree

Hide file tree

Showing 1,052 changed files with 5,610 additions and 6,026 deletions.
diff --git a/.github/workflows/visual-test.yml b/.github/workflows/visual-test.yml
@@ -112,6 +112,17 @@ jobs:
           key: ${{ secrets.CACHE_VERSION }}-${{ runner.os }}-deps-${{ hashFiles('**/yarn.lock') }}
           restore-keys: ${{ secrets.CACHE_VERSION }}-${{ runner.os }}-deps-
 
+      - name: Use Playwright cache
+        uses: actions/cache@v3
+        id: playwright-cache
+        with:
+          path: ~/Library/Caches/ms-playwright
+          key: ${{ secrets.CACHE_VERSION }}-${{ runner.os }}-playwright-${{ hashFiles('**/yarn.lock') }}
+      - run: npx playwright-firefox install --with-deps firefox
+        if: steps.playwright-cache.outputs.cache-hit != 'true'
+      - run: npx playwright-firefox install-deps firefox
+        if: steps.playwright-cache.outputs.cache-hit == 'true'
+
       - name: Install dependencies
         run: yarn install --immutable
 
@@ -133,6 +144,20 @@ jobs:
       - name: Run visual tests
         run: yarn workspace dnb-design-system-portal test:screenshots:ci
 
+      - uses: actions/upload-artifact@v3
+        if: failure()
+        with:
+          name: visual-test-artifact
+          path: ./packages/dnb-eufemia/src/**/*.snap-diff.png
+
+      - name: Run visual tests info
+        if: failure()
+        run: echo '\n\n👉 Download the diff files as a ZIP file. \nIt is called "visual-test-artifact" and you find it in the test "Summary" under "Artifacts".\n\n\n'
+
+      - uses: geekyeggo/delete-artifact@v2
+        with:
+          name: portal-build-artifact
+
       - name: Slack
         uses: 8398a7/action-slack@v3
         with:

diff --git a/packages/dnb-design-system-portal/package.json b/packages/dnb-design-system-portal/package.json
@@ -33,7 +33,7 @@
     "test:ci": "jest --ci --passWithNoTests",
     "test:staged": "jest --bail --findRelatedTests ",
     "test:screenshots": "yarn workspace @dnb/eufemia test:screenshots",
-    "test:screenshots:ci": "start-server-and-test serve http://localhost:8000 'yarn workspace @dnb/eufemia test:screenshots'",
+    "test:screenshots:ci": "start-server-and-test serve http://localhost:8000 'yarn workspace @dnb/eufemia test:screenshots:ci'",
     "test:types": "tsc --noEmit",
     "test:types:watch": "tsc --noEmit --watch",
     "test:update": "jest --updateSnapshot"

diff --git a/packages/dnb-design-system-portal/src/docs/contribute/faq.md b/packages/dnb-design-system-portal/src/docs/contribute/faq.md
@@ -17,10 +17,6 @@ LINT_STAGED=1
 
 ## Dependency issues
 
-### puppeteer
-
-- When upgrading to a newer version than v8, puppeteer behaves inconsistent. Sometimes the content is just tiny bit off. But most importantly, > v10.4 is very inconsistent and off running on the GitHub Actions maxOS.
-
 ### Stylelint
 
 - v14 has changed a good amount of their default styling rules. Updating would require us to refactor a good amount of SCSS code. We are currently on v13.

diff --git a/...b-design-system-portal/src/docs/contribute/first-contribution/before-started.md b/...b-design-system-portal/src/docs/contribute/first-contribution/before-started.md
@@ -261,34 +261,6 @@ The build will be exported to the `/public` directory. You can now also run a lo
 $ yarn serve
 ```
 
-### Testing
-
-Testing is currently handled by [Jest](https://jestjs.io/) and written using [Testing Library](https://testing-library.com/).
-
-Run the integration tests with
-
-```bash
-yarn test <file/folder>
-```
-
-Run the screenshot tests (make sure to have a portal running locally) with
-
-```bash
-yarn test:screenshots <file/folder>
-```
-
-Check out the results of the screenshot test with `open ./packages/dnb-eufemia/jest-screenshot-report/index.html`.
-
-### Run Algolia search queries locally
-
-In order to commit Algolia search queries to the `dev_eufemia_docs` index, you have to:
-
-Create a `.env` file inside `dnb-design-system-portal` with valid:
-
-- `ALGOLIA_INDEX_NAME=dev_eufemia_docs`
-- `ALGOLIA_APP_ID=SLD6KEYMQ9`
-- `ALGOLIA_API_KEY=secret`
-
 ## What happens in the build steps
 
 During the build, a lot of various things will happen. First, a prebuild before the build and afterward a postbuild.

diff --git a/packages/dnb-design-system-portal/src/docs/contribute/getting-started.md b/packages/dnb-design-system-portal/src/docs/contribute/getting-started.md
@@ -393,6 +393,8 @@ yarn test:update breadcrumb avatar
 
 3. Run the visual test against the portal:
 
+**NB:** Make sure you have the portal running locally on port 8000.
+
 ```bash
 # 1. First start the portal
 yarn start
@@ -401,22 +403,26 @@ yarn start
 yarn test:screenshots breadcrumb avatar
 ```
 
-You can also create a screenshot report for all components running `yarn test:screenshots`. Check the result / reports, located in: `open ./packages/dnb-eufemia/jest-screenshot-report/index.html`
-
-4. Update eventually new or valid PNG snapshots:
+4. Update eventually new or valid visual PNG snapshots:
 
 ```bash
 # Update all screenshot tests including 'breadcrumb'
 yarn test:screenshots:update breadcrumb
 ```
 
-5. How to deal with failing tests on the CI server?
+You can also press the `u` during a watch mode to update outdated snapshots.
+
+5. How to deal with failing visual tests?
+
+When a visual test fails, a visual comparison file (diff) will be created. Its location and name will be:
+
+- `**/__tests__/__image_snapshots__/__diff_output__/*.snap-diff.png`
 
-You may check out the logs for detailed reports.
+Please do not commit these files.
 
-**Failing visual screenshot tests**
+You may check out the CI/CLI logs for more details.
 
-If visual screenshot tests are failing on the CI, you can scroll all the way to the end of the logs. There you will find a link **Download the Report** you can use to download a zip file containing the visual test reports and diffs.
+**GitHub Actions:** If visual screenshot test is failing on the CI, you can navigate to the test "Summary" where you find "Artifacts". There you can download the **visual-test-artifact** zip file, containing the visual diff files.
 
 ### Support SCSS snapshot test
 
@@ -475,3 +481,13 @@ From a clone:
 - Make your changes and commit it to the repo in a new branch.
 - Make a _Pull Request_ to `origin/main`.
 - Watch the result of the tests.
+
+### Run Algolia search queries locally
+
+In order to commit Algolia search queries to the `dev_eufemia_docs` index, you have to:
+
+Create a `.env` file inside `dnb-design-system-portal` with valid:
+
+- `ALGOLIA_INDEX_NAME=dev_eufemia_docs`
+- `ALGOLIA_APP_ID=SLD6KEYMQ9`
+- `ALGOLIA_API_KEY=secret`
diff --git a/packages/dnb-design-system-portal/src/docs/uilib/components/drawer/Examples.js b/packages/dnb-design-system-portal/src/docs/uilib/components/drawer/Examples.js
@@ -89,19 +89,6 @@ export const FullDrawerExample = () => (
               phasellus praesent justo mollis montes velit taciti gravida
               lacus commodo senectus feugiat lorem etiam consequat
               penatibus cum hendrerit accumsan orci potenti purus nulla
-              interdum metus sollicitudin magnis libero sapien habitant non
-              class ridiculus consectetur congue nec litora sociosqu
-              aliquet felis in rhoncus nascetur odio ultricies nullam a
-              iaculis massa nisi ante nam cras aenean erat facilisi vivamus
-              ut cursus auctor arcu lobortis himenaeos dictum habitasse
-              tristique mauris at blandit sagittis nibh dignissim
-              condimentum per integer duis lacinia malesuada est adipiscing
-              maecenas donec eleifend turpis dictumst dapibus tempor fusce
-              aliquam torquent hac ac curabitur venenatis et tincidunt
-              augue porta vehicula enim facilisis posuere primis molestie
-              convallis diam vel fringilla dolor leo quis diam cursus massa
-              sapien tristique cum senectus sed tortor natoque amet
-              hendrerit ut fusce ipsum quis
             </P>
           </>
         )

diff --git a/packages/dnb-eufemia/.gitignore b/packages/dnb-eufemia/.gitignore
@@ -3,9 +3,8 @@
 
 # testing
 /coverage
-/jest-screenshot-report
-/reports/*
-!/reports/*.tgz
+**/__diff_output__/
+.jest-image-snapshot-touched-files
 
 # production
 *.min.mjs

diff --git a/packages/dnb-eufemia/.ncurc.json b/packages/dnb-eufemia/.ncurc.json
@@ -4,8 +4,6 @@
     "ora",
     "react",
     "react-dom",
-    "node-sass",
-    "puppeteer",
     "stylelint",
     "stylelint-config-prettier",
     "stylelint-config-recommended",

diff --git a/packages/dnb-eufemia/.npmignore b/packages/dnb-eufemia/.npmignore
@@ -3,7 +3,8 @@
 
 # testing
 /coverage
-/jest-screenshot-report
+/__diff_output__
+.jest-image-snapshot-touched-files
 jest.config.screenshots.js
 /reports
 __tests__

diff --git a/packages/dnb-eufemia/assets/images/__tests__/Logo.screenshot.test.js b/packages/dnb-eufemia/assets/images/__tests__/Logo.screenshot.test.js
diff --git a/packages/dnb-eufemia/jest.config.js b/packages/dnb-eufemia/jest.config.js
@@ -12,7 +12,6 @@ const config = {
     '<rootDir>/build/',
     '<rootDir>/assets/',
     '<rootDir>/stories/',
-    '<rootDir>/jest-screenshot-report/',
   ],
   transformIgnorePatterns: ['/node_modules/(?!(ora|globby)/)'],
   transform: {

diff --git a/packages/dnb-eufemia/jest.config.screenshots.js b/packages/dnb-eufemia/jest.config.screenshots.js
@@ -1,14 +1,26 @@
-// because --testPathIgnorePatterns=[array] has a bug, we do handle the ignore screenshots with this config
 const config = require('./jest.config.js')
 module.exports = {
-  ...config,
   ...{
+    preset: 'jest-playwright-preset',
+    testEnvironmentOptions: {
+      'jest-playwright': {
+        launchOptions: {
+          headless: true,
+        },
+        browsers: [
+          // 'chromium',
+          // 'webkit',
+          'firefox',
+        ],
+      },
+    },
     testRegex: 'screenshot.test.(js|ts|tsx)$',
-    globalSetup: './src/core/jest/jestPuppeteerSetup.js',
-    globalTeardown: './src/core/jest/jestPuppeteerTeardown.js',
     testEnvironment: './src/core/jest/jestPuppeteerEnvironment.js',
     setupFilesAfterEnv: ['./src/core/jest/setupJestScreenshot.js'],
-    reporters: ['default', 'jest-screenshot/reporter'],
+    reporters: [
+      'default',
+      'jest-image-snapshot/src/outdated-snapshot-reporter.js',
+    ],
     testPathIgnorePatterns: config.testPathIgnorePatterns.filter((i) => {
       return i !== 'screenshot'
     }),

diff --git a/packages/dnb-eufemia/package.json b/packages/dnb-eufemia/package.json
@@ -68,12 +68,12 @@
     "test": "jest",
     "test:auto-generated-types": "jest ./postTypeGeneration.test.js --ci --testPathIgnorePatterns=[]",
     "test:ci": "jest --ci",
-    "test:screenshots": "jest --config=./jest.config.screenshots.js --forceExit --detectOpenHandles",
+    "test:screenshots": "jest --config=./jest.config.screenshots.js --detectOpenHandles --testPathPattern ",
     "test:screenshots:ci": "jest --config=./jest.config.screenshots.js --ci --bail",
     "test:screenshots:ci:update": "yarn test:screenshots:ci --updateSnapshot",
-    "test:screenshots:recreate": "find . -name '*.snap.png' -type f|xargs rm -f && yarn test:screenshots",
-    "test:screenshots:update": "jest --config=./jest.config.screenshots.js --updateSnapshot --detectOpenHandles",
-    "test:screenshots:watch": "jest --config=./jest.config.screenshots.js --forceExit --detectOpenHandles --watchAll --testPathPattern ",
+    "test:screenshots:reset": "find . -name '*.snap.png' -type f|xargs rm -f && yarn test:screenshots",
+    "test:screenshots:update": "jest --config=./jest.config.screenshots.js --updateSnapshot --testPathPattern ",
+    "test:screenshots:watch": "JEST_IMAGE_SNAPSHOT_TRACK_OBSOLETE=1 jest --config=./jest.config.screenshots.js --watchAll --detectOpenHandles --testPathPattern ",
     "test:staged": "jest --bail --findRelatedTests ",
     "test:types": "tsc --noEmit",
     "test:types:watch": "tsc --noEmit --watch",
@@ -157,6 +157,7 @@
     "@testing-library/user-event": "13.5.0",
     "@types/jest": "29.2.6",
     "@types/jest-axe": "3.5.5",
+    "@types/jest-image-snapshot": "6.1.0",
     "@typescript-eslint/eslint-plugin": "5.28.0",
     "@typescript-eslint/parser": "5.43.0",
     "@wojtekmaj/enzyme-adapter-react-17": "0.6.5",
@@ -200,10 +201,10 @@
     "jest": "29.3.1",
     "jest-axe": "7.0.0",
     "jest-environment-jsdom": "29.3.1",
-    "jest-environment-node": "29.3.1",
+    "jest-image-snapshot": "6.1.0",
     "jest-matchmedia-mock": "1.1.0",
+    "jest-playwright-preset": "3.0.1",
     "jest-raw-loader": "1.0.1",
-    "jest-screenshot": "0.3.5",
     "jest-tobetype": "1.2.3",
     "lint-staged": "11.2.6",
     "lockfile-lint": "4.6.2",
@@ -216,14 +217,14 @@
     "opentype.js": "1.3.4",
     "ora": "5.4.1",
     "packpath": "0.1.0",
+    "playwright-firefox": "1.30.0",
     "postcss": "8.3.11",
     "postcss-calc": "8.0.0",
     "postcss-preset-env": "6.7.0",
     "prettier": "2.4.1",
     "prettier-package-json": "2.6.0",
     "process": "0.11.10",
     "prop-types": "15.7.2",
-    "puppeteer": "8.0.0",
     "react": "17.0.2",
     "react-dom": "17.0.2",
     "react-fake-props": "1.0.3",