Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Accessibility test Kitchen Sink with Playwright #1260

Merged
merged 56 commits into from
Apr 14, 2023
Merged
Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
56 commits
Select commit Hold shift + click to select a range
e3ed101
Add core-js so dev build works
gabalafou Mar 20, 2023
a16cc88
add more pages, test server
gabalafou Mar 21, 2023
68079b2
add documentation
gabalafou Mar 21, 2023
5030606
add prerequisites to nox test
gabalafou Mar 21, 2023
423ed5d
update package-lock.json via stb
gabalafou Mar 21, 2023
de4e2dd
update package-lock.json via nox -s compile
gabalafou Mar 22, 2023
310f171
streamline, update docs
gabalafou Mar 22, 2023
09906c1
typo
gabalafou Mar 22, 2023
2480b00
sync lockfile (remove core-js)
gabalafou Mar 22, 2023
6f938ba
Update tests/utils.py
gabalafou Mar 27, 2023
b5b57ca
Update tests/README.md
gabalafou Mar 27, 2023
f47a60a
Update docs/community/topics/accessibility.md
gabalafou Mar 27, 2023
62b6ae7
Update tests/README.md
gabalafou Mar 27, 2023
fc04acf
Update tests/utils.py
gabalafou Mar 27, 2023
23be214
Update tests/utils.py
gabalafou Mar 27, 2023
66fac74
- nox session a11y
gabalafou Mar 28, 2023
bc311db
update docs
gabalafou Mar 28, 2023
94fb49f
Merge branch 'main' into playwright-a11y-test-kitchen-sink
gabalafou Mar 28, 2023
9436844
uncomment commented-out tests
gabalafou Mar 28, 2023
e5a8814
regenerate package-lock.json via:
gabalafou Mar 28, 2023
3f86b11
oops finish updating tests
gabalafou Mar 28, 2023
700b64a
do not include a11y tests in main test session
gabalafou Mar 28, 2023
c57011d
remove redundant decorator
gabalafou Mar 29, 2023
ced281d
Update tests/README.md
gabalafou Apr 12, 2023
33b10ac
Update tests/README.md
gabalafou Apr 12, 2023
f8cee2c
Update tests/README.md
gabalafou Apr 12, 2023
9df077a
Remove base-url Pytest plugin
gabalafou Apr 12, 2023
1c2b8d5
Update tests/utils/pretty_axe_results.py
gabalafou Apr 12, 2023
a4325f5
Apply suggestions from code review
gabalafou Apr 12, 2023
72dfedc
Merge branch 'main' into playwright-a11y-test-kitchen-sink
gabalafou Apr 12, 2023
f1c8007
fix pretty_axe_results
gabalafou Apr 12, 2023
0317729
Generate each test run from parameters
gabalafou Apr 12, 2023
744fc94
remove unused marker url_path
gabalafou Apr 12, 2023
5e06478
undo multiline
gabalafou Apr 12, 2023
6820864
update doc string
gabalafou Apr 12, 2023
419d9e8
Add CI check for --with-deps suggested by Tony
gabalafou Apr 12, 2023
c18dbd6
oops, rm extra playwright install line
gabalafou Apr 12, 2023
e63d3e7
update comment
gabalafou Apr 12, 2023
83af9ac
update docstring
gabalafou Apr 12, 2023
178a689
Merge remote-tracking branch 'upstream/main' into playwright-a11y-tes…
drammock Apr 12, 2023
f7b92ea
testing the pytest-axe mark
drammock Apr 12, 2023
926d426
run prettier
drammock Apr 12, 2023
49c15a9
Revert "testing the pytest-axe mark"
drammock Apr 12, 2023
036506b
Fix D104
drammock Apr 12, 2023
0f0d599
skip a11y tests on CI runs
drammock Apr 12, 2023
2176326
run a11y tests on one CI worker only
drammock Apr 12, 2023
ec6a9d7
:white_check_mark: Register a11y mark and separate dependencies
trallard Apr 14, 2023
452ef73
:construction_worker: Update CI - delegate playwright install and cache
trallard Apr 14, 2023
71a86ac
📝 Update test docs
trallard Apr 14, 2023
645202c
:white_check_mark: Ensure a11y tests are skipped unles called
trallard Apr 14, 2023
07b9566
:construction_worker: Ensure CI installs all deps
trallard Apr 14, 2023
19ccd1b
:rewind: Revert to installing playwright
trallard Apr 14, 2023
112a27a
:construction_worker: Forgot to re-add a11y deps
trallard Apr 14, 2023
00005d2
:pencil2: Fix typo
trallard Apr 14, 2023
b9d83f6
:wrench: Fix path - seems to have been messed up when moving things
trallard Apr 14, 2023
22953ed
Update tests/test_a11y.py
drammock Apr 14, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 14 additions & 5 deletions docs/community/setup.md
Original file line number Diff line number Diff line change
Expand Up @@ -161,19 +161,28 @@ will cause the development server to do the following:

## Run the tests

This theme uses `pytest` for its testing, with a lightweight fixture defined
This theme uses `pytest` for its testing. There is a lightweight fixture defined
in the `test_build.py` script that makes it easy to run a Sphinx build using
this theme and inspect the results.
this theme and inspect the results. There are also accessibility checks in
gabalafou marked this conversation as resolved.
Show resolved Hide resolved
`test_accessibilty.py`.

In addition, we use [pytest-regressions](https://pytest-regressions.readthedocs.io/en/latest/)
to ensure that the HTML generated by the theme is what we'd expect. This module
In addition, we use
[pytest-regressions](https://pytest-regressions.readthedocs.io/en/latest/) to
ensure that the HTML generated by the theme is what we'd expect. This module
provides a `file_regression` fixture that will check the contents of an object
against a reference file on disk. If the structure of the two differs, then the
test will fail. If we _expect_ the structure to differ, then delete the file on
disk and run the test. A new file will be created, and subsequent tests will pass.
disk and run the test. A new file will be created, and subsequent tests will
pass.

To run the tests with `nox`, run the following command:

```console
$ nox -s test
```

To run just the accessibility tests:

```console
$ nox -s test -- -k accessibility
gabalafou marked this conversation as resolved.
Show resolved Hide resolved
```
4 changes: 4 additions & 0 deletions docs/community/topics/accessibility.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,9 @@
# Accessibility checks

Note 22-March-2023: we are currently
gabalafou marked this conversation as resolved.
Show resolved Hide resolved
[re-evaluating how we do accessibility checks](https://github.com/pydata/pydata-sphinx-theme/issues/1168)
and reporting, so this may change soon.
gabalafou marked this conversation as resolved.
Show resolved Hide resolved

The accessibility checking tools can find a number of common HTML patterns which
assistive technology can't help users understand.
We run a [Lighthouse](https://developers.google.com/web/tools/lighthouse) job in our CI/CD, which generates a "score" for all pages in our **Kitchen Sink** example documentation.
Expand Down
4 changes: 3 additions & 1 deletion noxfile.py
Original file line number Diff line number Diff line change
Expand Up @@ -71,8 +71,10 @@ def docs_live(session):
def test(session):
"""Run the test suite."""
if _should_install(session):
session.install("-e", ".[test]")
session.install("-e", ".[test,doc]")
gabalafou marked this conversation as resolved.
Show resolved Hide resolved
session.run("playwright", "install")
gabalafou marked this conversation as resolved.
Show resolved Hide resolved
12rambau marked this conversation as resolved.
Show resolved Hide resolved
gabalafou marked this conversation as resolved.
Show resolved Hide resolved
_compile_translations(session)
session.run("sphinx-build", "-b=html", "docs/", "docs/_build/html", "-v")
trallard marked this conversation as resolved.
Show resolved Hide resolved
session.run("pytest", *session.posargs)


Expand Down
Loading