Accessibility test Kitchen Sink with Playwright #1260

gabalafou · 2023-03-22T00:25:34Z

Closes #1168.

This pull request adds accessibility tests for the Kitchen Sink section of the PyData Sphinx Theme docs in order to check the theme for accessibility issues. As such it adds two major dependencies:

pytest-playwright to the Python test dependencies
axe-core to the Node (npm) dev dependencies

It updates the Nox file accordingly, and adds instructions to the docs.

Though mentioned already in the files added and changed, it bears repeating here: doing an accessibility check with axe-core is only a first-pass at testing for accessibility, since automated accessibility checks can only catch a fraction of web accessibility issues.

Install pytest-playwright and (npm) axe-core Write test that scans admonitions with Axe How to run test: nox -s compile nox -s docs python -m http.server -d docs/_build/html/ nox -s test -- -k accessibility

gabalafou

Done self reviewing!

To whoever reviews my PR, please bear in mind that I am a novice Python programmer, so please do not hesitate to call out any anti-patterns or code smells.

We'll also want to think about how we integrate this code into Github Actions, but rather than hold back this PR until everything is fleshed out, I wanted to get up some working code now to help anchor the conversation about what to do next.

For those who are curious, I have pasted the output from accessibility tests in a gist.

Most of the failures are color contrast issues, but there are also: landmarks need to be unique, scrollable regions need to be keyboard focusable, and links need to be clearly distinguishable.

noxfile.py

package-lock.json

pyproject.toml

tests/README.md

tests/test_accessibility.py

tests/utils.py

noxfile.py

docs/community/setup.md

drammock

Thanks for tackling this! here's a first pass with some comments. Please ping when you're ready for a second review.

docs/community/setup.md

noxfile.py

tests/README.md

tests/test_accessibility.py

tests/utils.py

tests/README.md

noxfile.py

trallard

Did a first pass review - lmk if there is anything need clarification

docs/community/topics/accessibility.md

docs/community/setup.md

tests/README.md

tests/utils.py

noxfile.py

tests/utils.py

Co-authored-by: Daniel McCloy <dan@mccloy.info>

Co-authored-by: Tania Allard <taniar.allard@gmail.com>

- add pytest-base-url plugin - push more test setup into page fixture - add license header to pretty_axe_results - move pretty_axe_results and check_warnings to utils/

rm package-lock.json rm -r node_modules stb npm install --include=dev

This reverts commit f7b92ea.

drammock · 2023-04-12T23:12:40Z

I think #1260 (comment) got missed in the last review rounds, so I'm brining it back up and highlighting it here.
One thing I don't like right now is that the following command fails:
nox -s test -- -k a11y

FWIW, locally this command runs for me (there are failures, but they're not due to missing playwright).

I've been pushing commits today to get the CIs to run (and then to get them to succeed) and in the process I noticed that pytest-axe supposedly has a built-in pytest mark that you can use to decorate tests; then running pytest --axe should run only those tests, and the absence of --axe should skip them. That seems to me like a pretty handy way to separate out the a11y tests. However, I tried quickly and couldn't get it to work, probably because I'm doing something wrong. Maybe you can get it to work @gabalafou? But even if so, pytest-axe is a public archive repo; so IDK if we want to use it if it's no longer maintained.

As an aside: I'm having trouble wrapping my head around all of this so it's hard to say what is the best approach. I think what we have here is (simplified):

├── .github/workflows/tests.yml
│   ├── run-pytest        -> runs on various OSes and Python versions; calls `pytest` directly
│   ├── build-site        -> runs on 3 OSes; calls `sphinx-build`;
│   │                        checks for (un)expected build warnings
│   ├── new-a11y-CI-test  -> ***MISSING / NOT WORKING YET***
│   └── audit             -> old lighthouse job; "needs" build-site; runs on ubuntu only;
│                            creates small site (just kitchen sink), runs `sphinx-build`,
│                            and runs the lighthouse action on that static site.
│
├── noxfile.py
│   ├── docs  ->  calls `stb compile` and `sphinx-build`
│   └── a11y  ->  calls `playwright install`, `nox -s docs`, and `pytest -k a11y`
│
└── tests/
    ├── test_build.py  -> calls `sphinx-build` via a factory function
    └── test_a11y.py   -> sets up a webserver and calls `axe.run()` a bunch of times

The mix-and-match of stb versus sphinx-build, calling pytest or sphinx-build or stb directly vs calling the nox job... it all has my head spinning; currently we don't use nox in our CI config (except for the "compile MO files" step of the run-pytest job? why??)

Probably this deserves its own issue for discussion, but in the meantime we need to implement the new a11y CI test somehow so if we can quickly converge on what we should be doing everywhere then we can just do that in one place now and then (I or someone else) can propogate that decision to the other CI jobs.

Thoughts/advice from anyone are very welcome.

gabalafou · 2023-04-12T23:26:57Z

FWIW, locally this command runs for me (there are failures, but they're not due to missing playwright).

That's odd. This is how it looks in my terminal

drammock · 2023-04-13T14:41:13Z

Not sure what's different. I definitely don't have playwright installed globally (which playwright returns nothing), and it is in the relevant nox env:

$ ll .nox/a11y/bin/playwright 
-rwxrwxr-x 1 drmccloy drmccloy 247 Apr 12 15:41 .nox/a11y/bin/playwright*

trallard · 2023-04-13T15:26:44Z

The mix-and-match of stb versus sphinx-build, calling pytest or sphinx-build or stb directly vs calling the nox job... it all has my head spinning; currently we don't use nox in our CI config (except for the "compile MO files" step of the run-pytest job? why??)

Probably this deserves its own issue for discussion

I am really confused about this mix as well. So I'd be +1 on moving this to a separate discussion in the interest of standardisation

we need to implement the new a11y CI test somehow

@gabalafou and I are meeting today so will make sure to discuss this PR and come up with a "good enough" approach to get this moving

drammock · 2023-04-13T15:58:01Z

@gabalafou and I are meeting today so will make sure to discuss this PR and come up with a "good enough" approach to get this moving

excellent. I'll open a new issue about untangling the mix for the whole project.

noxfile.py

trallard · 2023-04-14T11:20:03Z

pyproject.toml

-]
+test = ["pytest", "pytest-cov", "pytest-regressions"]
+dev = ["pyyaml", "pre-commit", "nox", "pydata-sphinx-theme[doc,test]"]
+a11y = ["pytest-playwright"]


Moved this to a separate env - since it is not needed for the general tests

trallard · 2023-04-14T11:20:48Z

tests/test_a11y.py

+        process.wait()
+
+
+@pytest.mark.a11y


I decided register this as a custom mark instead of using the -k pytest flag for string matching

tests/test_a11y.py

trallard · 2023-04-14T11:52:55Z

@drammock and @gabalafou - I just pushed some changes that should fix the issue with missing dependencies.
I tested locally by running.

nox -s test

nox -s a11y

And all runs as expected. I also made changes to address the concerns re having separate sessions and as long as the CI is happy I suppose we can call this our first iteration and improve on top of this

trallard · 2023-04-14T13:54:17Z

noxfile.py

+        if os.environ.get("CI") or os.environ.get("GITPOD_WORKSPACE_ID"):
+            # CI and other cloud environments are potentially missing system
+            # dependencies, so we tell Playwright to also install the system
+            # dependencies
+            session.run("playwright", "install", "--with-deps")


I had initially removed this until I had to update the CI tests and realised I duplicated a lot of code trying to use pytest directly like in other tests.
So, for now, I am leaving this here and calling tests through nox.

Note: I am unsure if we should keep the GITPOD_WORKSPACE_ID bit. I am a Gitpod user, but I wonder if it will confuse the rest of the folks who do not use Gitpod.

I hadn't heard of Gitpod, but I wasn't confused :) I figured "probably some containerized thing" (I've since looked it up).

trallard · 2023-04-14T14:44:20Z

🎉 Tests are running in CI and can be run locally with nox.

I will wait for @drammock to give this a last check but I think this can be merged and we can iterate on reporting and the such from here onwards

drammock

one tiny typo that I'll commit through the web interface, then let's merge. Thanks @gabalafou and @trallard !!

drammock · 2023-04-12T20:35:20Z

tests/test_a11y.py

+@pytest.mark.parametrize(
+    "url_page,selector",
+    [
+        ("admonitions.html", "#admonitions"),
+        ("api.html", "#api-documentation"),
+        ("blocks.html", "#blocks"),
+        ("generic.html", "#generic-items"),
+        ("images.html", "#images-figures"),
+        ("lists.html", "#lists"),
+        ("structure.html", "#structural-elements"),
+        ("structure.html", "#structural-elements-2"),
+        ("tables.html", "#tables"),
+        ("typography.html", "#typography"),
+    ],


drammock · 2023-04-14T19:29:08Z

noxfile.py

+        if os.environ.get("CI") or os.environ.get("GITPOD_WORKSPACE_ID"):
+            # CI and other cloud environments are potentially missing system
+            # dependencies, so we tell Playwright to also install the system
+            # dependencies
+            session.run("playwright", "install", "--with-deps")


I hadn't heard of Gitpod, but I wasn't confused :) I figured "probably some containerized thing" (I've since looked it up).

tests/test_a11y.py

gabalafou added 6 commits March 20, 2023 19:38

Add core-js so dev build works

e3ed101

Install pytest-playwright and (npm) axe-core Write test that scans admonitions with Axe How to run test: nox -s compile nox -s docs python -m http.server -d docs/_build/html/ nox -s test -- -k accessibility

add more pages, test server

a16cc88

add documentation

68079b2

add prerequisites to nox test

5030606

update package-lock.json via stb

423ed5d

update package-lock.json via nox -s compile

de4e2dd

trallard added the tag: accessibility Issues related to accessibility issues or efforts label Mar 22, 2023

trallard self-requested a review March 22, 2023 17:23

gabalafou added 2 commits March 22, 2023 17:48

streamline, update docs

310f171

typo

09906c1

gabalafou commented Mar 22, 2023

View reviewed changes

sync lockfile (remove core-js)

2480b00

12rambau reviewed Mar 23, 2023

View reviewed changes

noxfile.py Outdated Show resolved Hide resolved

noxfile.py Outdated Show resolved Hide resolved

docs/community/setup.md Outdated Show resolved Hide resolved

drammock mentioned this pull request Mar 23, 2023

Update node packages and remove unneeded ones #1258

Merged

drammock reviewed Mar 23, 2023

View reviewed changes

drammock mentioned this pull request Mar 23, 2023

infinite loop in nox -s docs-live #1256

Closed

gabalafou mentioned this pull request Mar 23, 2023

Team update - Mar 22, 2023 Quansight-Labs/jupyter-a11y-mgmt#204

Closed

trallard reviewed Mar 27, 2023

View reviewed changes

gabalafou and others added 12 commits March 27, 2023 09:20

Update tests/utils.py

6f938ba

Co-authored-by: Daniel McCloy <dan@mccloy.info>

Update tests/README.md

b5b57ca

Co-authored-by: Tania Allard <taniar.allard@gmail.com>

Update docs/community/topics/accessibility.md

f47a60a

Co-authored-by: Tania Allard <taniar.allard@gmail.com>

Update tests/README.md

62b6ae7

Co-authored-by: Tania Allard <taniar.allard@gmail.com>

Update tests/utils.py

fc04acf

Co-authored-by: Tania Allard <taniar.allard@gmail.com>

Update tests/utils.py

23be214

Co-authored-by: Tania Allard <taniar.allard@gmail.com>

- nox session a11y

66fac74

- add pytest-base-url plugin - push more test setup into page fixture - add license header to pretty_axe_results - move pretty_axe_results and check_warnings to utils/

update docs

bc311db

Merge branch 'main' into playwright-a11y-test-kitchen-sink

94fb49f

uncomment commented-out tests

9436844

regenerate package-lock.json via:

e5a8814

rm package-lock.json rm -r node_modules stb npm install --include=dev

oops finish updating tests

3f86b11

drammock added 6 commits April 12, 2023 16:02

testing the pytest-axe mark

f7b92ea

run prettier

926d426

Revert "testing the pytest-axe mark"

49c15a9

This reverts commit f7b92ea.

Fix D104

036506b

skip a11y tests on CI runs

0f0d599

run a11y tests on one CI worker only

2176326

drammock mentioned this pull request Apr 13, 2023

can we simplify how our tests are defined and run? #1292

Closed

trallard added 3 commits April 14, 2023 11:26

✅ Register a11y mark and separate dependencies

ec6a9d7

👷 Update CI - delegate playwright install and cache

452ef73

📝 Update test docs

71a86ac

trallard reviewed Apr 14, 2023

View reviewed changes

✅ Ensure a11y tests are skipped unles called

645202c

trallard reviewed Apr 14, 2023

View reviewed changes

tests/test_a11y.py Outdated Show resolved Hide resolved

👷 Ensure CI installs all deps

07b9566

⏪ Revert to installing playwright

19ccd1b

trallard reviewed Apr 14, 2023

View reviewed changes

trallard added 3 commits April 14, 2023 14:55

👷 Forgot to re-add a11y deps

112a27a

✏️ Fix typo

00005d2

🔧 Fix path - seems to have been messed up when moving things

b9d83f6

drammock approved these changes Apr 14, 2023

View reviewed changes

Update tests/test_a11y.py

22953ed

drammock merged commit cbebfc9 into pydata:main Apr 14, 2023

gabalafou mentioned this pull request Aug 14, 2023

MAINT: Broaden coverage of accessibility tests with Playwright #1378

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Accessibility test Kitchen Sink with Playwright #1260

Accessibility test Kitchen Sink with Playwright #1260

gabalafou commented Mar 22, 2023 •

edited

Loading

gabalafou left a comment

drammock left a comment

trallard left a comment

drammock commented Apr 12, 2023

gabalafou commented Apr 12, 2023

drammock commented Apr 13, 2023

trallard commented Apr 13, 2023

drammock commented Apr 13, 2023

trallard Apr 14, 2023

trallard Apr 14, 2023

trallard commented Apr 14, 2023

trallard Apr 14, 2023

drammock Apr 14, 2023

trallard commented Apr 14, 2023

drammock left a comment

drammock Apr 12, 2023

drammock Apr 14, 2023

		process.wait()


		@pytest.mark.a11y

Accessibility test Kitchen Sink with Playwright #1260

Accessibility test Kitchen Sink with Playwright #1260

Conversation

gabalafou commented Mar 22, 2023 • edited Loading

gabalafou left a comment

Choose a reason for hiding this comment

drammock left a comment

Choose a reason for hiding this comment

trallard left a comment

Choose a reason for hiding this comment

drammock commented Apr 12, 2023

gabalafou commented Apr 12, 2023

drammock commented Apr 13, 2023

trallard commented Apr 13, 2023

drammock commented Apr 13, 2023

trallard Apr 14, 2023

Choose a reason for hiding this comment

trallard Apr 14, 2023

Choose a reason for hiding this comment

trallard commented Apr 14, 2023

trallard Apr 14, 2023

Choose a reason for hiding this comment

drammock Apr 14, 2023

Choose a reason for hiding this comment

trallard commented Apr 14, 2023

drammock left a comment

Choose a reason for hiding this comment

drammock Apr 12, 2023

Choose a reason for hiding this comment

drammock Apr 14, 2023

Choose a reason for hiding this comment

gabalafou commented Mar 22, 2023 •

edited

Loading