Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce Testing to Stacks #1194

Merged
merged 38 commits into from
Dec 15, 2022
Merged
Show file tree
Hide file tree
Changes from 34 commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
f07da4d
test(setup): add a poc for component testing
giamir Oct 7, 2022
d1e430f
match orginial indentation for diff purposes
dancormier Oct 18, 2022
414fb28
test(tooltip): add wtr visual regression example
giamir Oct 27, 2022
de78eb0
Merge branch 'develop' into gbuoncristiani/introduce-testing-setup
giamir Nov 25, 2022
b3c1ba1
refactor(tooltip): remove old tooltip version
giamir Nov 25, 2022
3df1318
test(backstop): mark backstop tests as legacy
giamir Nov 25, 2022
7f075c3
ci(workflow): add basic pipeline to build and run tests
giamir Nov 28, 2022
664bef5
fix(styles): fix stylelint rules violations
giamir Nov 28, 2022
53937b3
fix(ts): fix eslint rules violations
giamir Nov 28, 2022
b5d1aea
fix(format): fix all prettier violations
giamir Nov 28, 2022
7559641
test(visual): remove baseline images from regular git tracking
giamir Nov 28, 2022
4793b1a
test(visual): track baseline images with git lfs
giamir Nov 28, 2022
688c071
fix(lint-ts): adjust rule exception after formatting
giamir Nov 28, 2022
d03dea1
ci(test): trigger another netlify deployment
giamir Nov 28, 2022
0d0b6dd
Merge branch 'develop' into gbuoncristiani/testing
giamir Nov 28, 2022
55d0c85
ci(test): update github workflow
giamir Nov 29, 2022
440ecc1
test(tooltip): fix width for visual regression test
giamir Nov 29, 2022
b7ab801
test(tooltip): adjust width for visual regression test
giamir Nov 29, 2022
1a850b9
ci(test): upload visual regression diffs on failure
giamir Nov 29, 2022
4e1fecf
ci(test): fix workflow syntax
giamir Nov 29, 2022
5f7a2f1
test(visual): settle on a default font family for tests
giamir Nov 29, 2022
56d88db
ci(test): fix small typo
giamir Nov 29, 2022
3613a6d
test(visual): set a failure threshold
giamir Nov 29, 2022
bf8999a
ci(test): adjust playwright install command
giamir Nov 29, 2022
9cc7f65
ci(test): add firefox launch options
giamir Nov 29, 2022
a4e6d71
ci(test): disable firefox temporarily
giamir Nov 29, 2022
b472f08
test(tooltip): add prefix s- for consistency
giamir Nov 29, 2022
f795b07
ci(test): fix and add back testing against firefox in CI
giamir Nov 29, 2022
690a8fe
Update README.md
giamir Dec 1, 2022
6055c67
Merge branch 'develop' into gbuoncristiani/testing
giamir Dec 1, 2022
c46b5b9
Merge branch 'develop' into gbuoncristiani/testing
giamir Dec 2, 2022
170ba48
fix(dev-server-rollup): apply temp patch to run tests in windows
giamir Dec 4, 2022
b4e1b0b
test(visual): relax failure threshold to 5 percent
giamir Dec 4, 2022
166236f
test(cleanup): remove backstop legacy tests
giamir Dec 4, 2022
91a56eb
test(visual): update default fonts
giamir Dec 6, 2022
b1ebfac
Merge branch 'develop' into gbuoncristiani/testing
giamir Dec 6, 2022
0caa564
Update .gitignore
giamir Dec 15, 2022
483c0cc
Merge branch 'develop' into gbuoncristiani/testing
giamir Dec 15, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
screenshots/** filter=lfs diff=lfs merge=lfs -text
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎉 Nice, this should keep us from accidentally committing images to the git history

46 changes: 46 additions & 0 deletions .github/workflows/main.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
name: CI

on:
push:
branches: [develop]
pull_request:
branches: [develop]

jobs:
build-and-test:
name: Build and Test
runs-on: ubuntu-latest
steps:
- name: ⬇️ Checkout
uses: actions/checkout@v3
with:
lfs: true

- name: ⎔ Setup node
uses: actions/setup-node@v3
with:
node-version: 'lts/*'

- name: 🏗 Install and Build
run: |
npm ci
npx playwright install --with-deps
npm run build

- name: ▶️ Lint and Test
run: |
npm run lint
npm run test -- --config web-test-runner.config.ci.mjs

- name: ⬆️ Upload Visual Regression Test Results
uses: actions/upload-artifact@v3
if: failure()
with:
name: visual-regression-test-results
path: screenshots


# cancel the jobs if another workflow is kicked off for the same branch
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
8 changes: 5 additions & 3 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -78,11 +78,13 @@ dist/

# Testing Assets #
###############
/backstop_data/bitmaps_test/
/backstop_data/bitmaps_reference/
/backstop_data/html_report/
/screenshots/**/failed/**

# GitHub Actions #
###############
/.github/main.workflow
.vscode/

# JetBrains Editors #
###############
/.idea
giamir marked this conversation as resolved.
Show resolved Hide resolved
42 changes: 40 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,9 +55,47 @@ npm run lint:format
```

## Testing Stacks
Stacks has implemented visual regression testing with [Backstop](https://github.com/garris/BackstopJS). To test if your new feature introduces visual regressions, run `npm run test` in a new Terminal window while Stacks is running. After the tests have run, a new browser window with any regressions will show. If the regressions are desired, you can run `npm run approve` to establish the new baseline.

Individual routes to test are found in [backstop.json](/backstop.json)
Run all test suites by running:
```sh
npm test
```
### Unit/Component Tests

Unit/Component tests are written with [DOM Testing Library](https://testing-library.com/docs/dom-testing-library/intro).
Please follow the library's principles and documentation to write tests.

Stacks uses [Web Test Runner](https://modern-web.dev/docs/test-runner/overview/) and [Playwright](https://modern-web.dev/docs/test-runner/browser-launchers/playwright/) to run tests in a real browser context.

Execute the unit/component tests suite by running:
```sh
npm run test:unit
```
or if you prefer watch mode run:
```sh
npm run test:unit:watch
```

### Visual Regression Tests

**Prerequisite:**
In order to pull and upload baseline images you need to have `git lfs` installed in your local machine. Please follow [this guide](https://docs.github.com/en/repositories/working-with-files/managing-large-files/installing-git-large-file-storage) to install that.
b-kelly marked this conversation as resolved.
Show resolved Hide resolved


This [Web Test Runner plugin](https://www.npmjs.com/package/@web/test-runner-visual-regression) is used to run visual regression tests. [DOM Testing Library](https://testing-library.com/docs/dom-testing-library/intro).
Visual regression tests end with this suffix `*.visual.test.ts`.

Execute the visual regression tests suite by running:
```sh
npm run test:visual
```

Update the visual baseline via:
```sh
npm run test:visual:update
```

Failing tests (including diffs) can be found under `screenshots/[browser]/failed/` folders.

## Bugs and feature requests
Have a bug or feature request? First search existing or closed issues to make sure the issue hasn’t been noted yet. If not, review our [issue guidelines](/CONTRIBUTING.md#open-an-issue) for submitting [a bug report](/CONTRIBUTING.md#reporting-bugs) or [feature request](/CONTRIBUTING.md#feature-requests).
Expand Down
Loading