Accessibility test Kitchen Sink with Playwright

.github/workflows/prerelease.yml

@@ -36,7 +36,7 @@ jobs:
 
       - name: Check that there are no unexpected Sphinx warnings
         if: matrix.python-version == '3.10'
-        run: python tests/check_warnings.py
+        run: python tests/utils/check_warnings.py
 
       - name: Run the tests
         run: |

.github/workflows/tests.yml

@@ -118,7 +118,7 @@ jobs:
       - name: Build docs
         run: sphinx-build -b html docs/ docs/_build/html --keep-going -w warnings.txt
       - name: Check for unexpected Sphinx warnings
-        run: python tests/check_warnings.py
+        run: python tests/utils/check_warnings.py
 
   # Run local Lighthouse audit against built site
   audit:

docs/community/setup.md

@@ -161,19 +161,28 @@ will cause the development server to do the following:
 
 ## Run the tests
 
-This theme uses `pytest` for its testing, with a lightweight fixture defined
+This theme uses `pytest` for its testing. There is a lightweight fixture defined
 in the `test_build.py` script that makes it easy to run a Sphinx build using
-this theme and inspect the results.
+this theme and inspect the results. There are also a number of automated accessibility checks in
+`test_a11y.py`.
 
-In addition, we use [pytest-regressions](https://pytest-regressions.readthedocs.io/en/latest/)
-to ensure that the HTML generated by the theme is what we'd expect. This module
+In addition, we use
+[pytest-regressions](https://pytest-regressions.readthedocs.io/en/latest/) to
+ensure that the HTML generated by the theme is what we'd expect. This module
 provides a `file_regression` fixture that will check the contents of an object
 against a reference file on disk. If the structure of the two differs, then the
 test will fail. If we _expect_ the structure to differ, then delete the file on
-disk and run the test. A new file will be created, and subsequent tests will pass.
+disk and run the test. A new file will be created, and subsequent tests will
+pass.
 
-To run the tests with `nox`, run the following command:
+To run the build tests with `nox`, run the following command:
 
 ```console
 $ nox -s test
 ```
+
+To run the accessibility checks:
+
+```console
+$ nox -s a11y
+```

docs/community/topics/accessibility.md

@@ -1,5 +1,10 @@
 # Accessibility checks
 
+```{note} 
+22-March-2023: we are currently
+[re-evaluating how we do accessibility checks](https://github.com/pydata/pydata-sphinx-theme/issues/1168)
+and reporting, so this may change soon.
+
 The accessibility checking tools can find a number of common HTML patterns which
 assistive technology can't help users understand.
 We run a [Lighthouse](https://developers.google.com/web/tools/lighthouse) job in our CI/CD, which generates a "score" for all pages in our **Kitchen Sink** example documentation.

noxfile.py

@@ -5,6 +5,7 @@
 
     nox -s docs -- -r
 """
+import os
 import shutil as sh
 import tempfile
 from pathlib import Path
@@ -87,7 +88,28 @@ def test(session: nox.Session) -> None:
     if _should_install(session):
         session.install("-e", ".[test]")
     session.run(*split("pybabel compile -d src/pydata_sphinx_theme/locale -D sphinx"))
-    session.run("pytest", *session.posargs)
+    session.run("pytest", "-k", "not a11y", *session.posargs)
+
+
+@nox.session()
+def a11y(session: nox.Session) -> None:
+    """Run the accessibility test suite."""
+    if _should_install(session):
+        session.install("-e", ".[test]")
+        # Install the drivers that Playwright needs to control the browsers.
+        if os.environ.get("CI") or os.environ.get("GITPOD_WORKSPACE_ID"):
+            # CI and other cloud environments are potentially missing system
+            # dependencies, so we tell Playwright to also install the system
+            # dependencies
+            session.run("playwright", "install", "--with-deps")
+        else:
+            # But most dev environments have the needed system dependencies
+            session.run("playwright", "install")
+    # Build the docs so we can run accessibility tests against them.
+    session.run("nox", "-s", "docs")
+    # The next step would be to open a server to the docs for Playwright, but
+    # that is done in the test file, along with the accessibility checks.
+    session.run("pytest", "-k", "a11y", *session.posargs)
 
 
 @nox.session(name="test-sphinx")

package.json

@@ -5,6 +5,7 @@
     "build": "webpack"
   },
   "devDependencies": {
+    "axe-core": "^4.6.3",
     "copy-webpack-plugin": "^11.0.0",
     "css-loader": "^3.4.2",
     "css-minimizer-webpack-plugin": "^4.2.2",

pyproject.toml

@@ -81,6 +81,7 @@ test = [
   "pytest",
   "pytest-cov",
   "pytest-regressions",
+  "pytest-playwright",
   "codecov",
 ]
 dev = [

tests/README.md

@@ -0,0 +1,28 @@
+# PyData Sphinx tests
+
+This directory contains the Python tests for the theme. These tests are built with [pytest](https://docs.pytest.org/en/stable/) and are called through `nox`.
+
+`test_build.py` checks that the static HTML output of the build process conforms
+to various expectations. It builds static HTML pages based on configurations in
+the `sites/` directory. The tests run various assertions on the static HTML
+output, including snapshot comparisons with previously compiled outputs that are
+stored in `test_build/`. In other words, it uses
+[`pytest-regressions`](https://pytest-regressions.readthedocs.io/) to compare
+the output created during the test run with a previous known and verified output
+(stored under `test_build`) to make sure nothing has changed.
+
+`test_a11y.py` checks PyData Sphinx Theme components for accessibility issues.
+It's important to note that [only a fraction of accessibility issues can be
+caught with automated
+testing](https://accessibility.blog.gov.uk/2017/02/24/what-we-found-when-we-tested-tools-on-the-worlds-least-accessible-webpage/).
+In contrast to the build test suite, the accessibility suite checks components as
+they appear in the browser, meaning with any CSS and JavaScript applied. It does
+this by building the PyData Sphinx Theme docs, launching a local server to the
+docs, then checking the "Kitchen Sink" example pages with
+[Playwright](https://playwright.dev), a program for developers that allows
+loading and manipulating pages with various browsers, such as Chrome (chromium),
+Firefox (gecko), Safari (WebKit).
+
+The ["Kitchen Sink" examples](https://pydata-sphinx-theme.readthedocs.io/en/stable/examples/kitchen-sink/index.html)
+are taken from [sphinx-themes.org](https://sphinx-themes.org/) and showcase
+components of the PyData Sphinx Theme, such as admonitions, lists, and headings.

tests/test_a11y.py

@@ -0,0 +1,110 @@
+"""Using Axe-core, scan the Kitchen Sink pages for accessibility violations."""
+
+import time
+from http.client import HTTPConnection
+from pathlib import Path
+from subprocess import PIPE, Popen
+from urllib.parse import urljoin
+
+import pytest
+from playwright.sync_api import Page
+
+from .utils.pretty_axe_results import pretty_axe_results
+
+# Important note: automated accessibility scans can only find a fraction of
+# potential accessibility issues.
+#
+# This test file scans pages from the Kitchen Sink examples with a JavaScript
+# library called Axe-core, which checks the page for accessibility violations,
+# such as places on the page with poor color contrast that would be hard for
+# people with low vision to see.
+#
+# Just because a page passes the scan with no accessibility violations does
+# *not* mean that it will be generally usable by a broad range of disabled
+# people. It just means that page is free of common testable accessibility
+# pitfalls.
+
+path_repo = Path(__file__).parent.parent
+path_docs_build = path_repo / "docs" / "_build" / "html"
+
+
+@pytest.fixture(scope="module")
+def url_base():
+    """Start local server on built docs and return the localhost URL as the base URL."""
+    # Use a port that is not commonly used during development or else you will
+    # force the developer to stop running their dev server in order to run the
+    # tests.
+    port = "8213"
+    host = "localhost"
+    url = f"http://{host}:{port}"
+
+    # Try starting the server
+    process = Popen(
+        ["python", "-m", "http.server", port, "--directory", path_docs_build],
+        stdout=PIPE,
+    )
+
+    # Try connecting to the server
+    retries = 5
+    while retries > 0:
+        conn = HTTPConnection(host, port)
+        try:
+            conn.request("HEAD", "/")
+            response = conn.getresponse()
+            if response is not None:
+                yield url
+                break
+        except ConnectionRefusedError:
+            time.sleep(1)
+            retries -= 1
+
+    # If the code above never yields a URL, then we were never able to connect
+    # to the server and retries == 0.
+    if not retries:
+        raise RuntimeError("Failed to start http server in 5 seconds")
+    else:
+        # Otherwise the server started and this fixture is done now and we clean
+        # up by stopping the server.
+        process.terminate()
+        process.wait()
+
+
+@pytest.mark.parametrize("theme", ["light", "dark"])
+@pytest.mark.parametrize(
+    "url_page,selector",
+    [
+        ("admonitions.html", "#admonitions"),
+        ("api.html", "#api-documentation"),
+        ("blocks.html", "#blocks"),
+        ("generic.html", "#generic-items"),
+        ("images.html", "#images-figures"),
+        ("lists.html", "#lists"),
+        ("structure.html", "#structural-elements"),
+        ("structure.html", "#structural-elements-2"),
+        ("tables.html", "#tables"),
+        ("typography.html", "#typography"),
+    ],
+)
+def test_axe_core_kitchen_sink(
+    page: Page, theme: str, url_base: str, url_page: str, selector: str
+):
+    """Should have no Axe-core violations at the provided theme and page section."""
+    # Load the page at the provided path
+    url_base_kitchen_sink = urljoin(url_base, "/examples/kitchen-sink/")
+    url_full = urljoin(url_base_kitchen_sink, url_page)
+    page.goto(url_full)
+
+    # Run a line of JavaScript that sets the light/dark theme on the page
+    page.evaluate(f"document.documentElement.dataset.theme = '{theme}'")
+
+    # Inject the Axe-core JavaScript library into the page
+    page.add_script_tag(path="node_modules/axe-core/axe.min.js")
+
+    # Run the Axe-core library against a section of the page. (Don't run it
+    # against the whole page because in this test we're not trying to find
+    # accessibility violations in the nav, sidebar, footer, or other parts of
+    # the PyData Sphinx Theme documentation website.)
+    results = page.evaluate(f"axe.run('{selector}')")
+
+    # Expect Axe-core to have found 0 accessibility violations
+    assert len(results["violations"]) == 0, pretty_axe_results(results)