Skip to content

Add pa11y testing and reporting #294

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 31 commits into from
Aug 27, 2021
Merged

Add pa11y testing and reporting #294

Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
b63a39c
add pa11y-ci reporting
bollwyvl Dec 11, 2020
5755c49
update docs
bollwyvl Dec 11, 2020
ab8748e
fix warnings in new docs
bollwyvl Dec 11, 2020
cb3149d
linting, comment out a roadmap item so it will fail
bollwyvl Dec 11, 2020
1c10ad7
add some developer docs, in-browser tools
bollwyvl Dec 11, 2020
2ca1778
try to get audit to show up in annotation
bollwyvl Dec 11, 2020
dfabae5
yep, i sure don't know rst
bollwyvl Dec 11, 2020
e26b2cb
try annotations directly
bollwyvl Dec 11, 2020
ecf6970
clean up more docs
bollwyvl Dec 11, 2020
c8001f2
Merge remote-tracking branch 'upstream/master' into add-pa11y
bollwyvl Dec 18, 2020
8e122bb
further streamline test workflow
bollwyvl Dec 18, 2020
970ddd5
rework language about audits, links to specs
bollwyvl Dec 18, 2020
8db7e24
fix reporter
bollwyvl Dec 18, 2020
0997016
address more comments
bollwyvl Dec 18, 2020
4bb257c
uncomment intentional fail
bollwyvl Dec 18, 2020
125d5fb
restore http streaming log
bollwyvl Dec 18, 2020
eb84c10
more reporting
bollwyvl Dec 18, 2020
fac2b3d
get SEO score back up by adding metatags, normalizing URLs to 127.0.0.1
bollwyvl Dec 18, 2020
c846cae
fix url for accessibility page in lighthouse
bollwyvl Dec 18, 2020
3376f82
tweak some language settings
bollwyvl Dec 18, 2020
7c41ef5
merge upstream
bollwyvl Mar 9, 2021
746c5d0
yarn config updates
bollwyvl Mar 9, 2021
7149244
resolve yarn.lock
bollwyvl Mar 9, 2021
88c60f3
merge upstream
bollwyvl Mar 25, 2021
d6c043d
resolve yarn.lock
bollwyvl Mar 25, 2021
6aef1c6
merge upstream
bollwyvl Aug 25, 2021
c8ecd78
resolve yarn.lock
bollwyvl Aug 25, 2021
df90d50
revert default port to 8000
bollwyvl Aug 25, 2021
1f1ba7a
install nodejs
bollwyvl Aug 25, 2021
11a4650
improve artifact uploading
bollwyvl Aug 25, 2021
809c5ab
Update docs/user_guide/accessibility.rst
bollwyvl Aug 25, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
168 changes: 135 additions & 33 deletions .github/workflows/tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,12 @@
name: continuous-integration

on: [push, pull_request]

on:
push:
branches: [master]
pull_request:
branches: [master]


jobs:

Expand All @@ -9,7 +15,7 @@ jobs:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: [3.8]
python-version: [3.9]

steps:
- uses: actions/checkout@v2
Expand All @@ -24,12 +30,44 @@ jobs:
with:
node-version: '10.x'

- name: Install dependencies
- name: Install python setup dependencies
run: |
pip install --user --upgrade pip setuptools wheel

- name: Get pip cache dir
id: pip-cache
run: |
python -m pip install --upgrade pip
python -m pip install --upgrade pre-commit
pip install -e .
yarn
echo "::set-output name=dir::$(pip cache dir)"

- name: Cache pip
uses: actions/cache@v1
with:
path: ${{ steps.pip-cache.outputs.dir }}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do we now need this much more complex install config? It's fine if it's really needed, but just looking at it seems much less-maintainable to an average person to me 🤔

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure, can drop the cache.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

well if it's really useful then we should keep it, I just think we should be in the habit of justifying added complexity and maintenance burden, so I'm wondering what benefit it brings! (not out of skepticism, just because I do not know)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

simplified on 8e122bb etc

Copy link
Collaborator Author

@bollwyvl bollwyvl Dec 18, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using the CI cache makes lighter use of community resources, e.g. pypi. It's not making a detectable speed difference, however in this case.

To really drive down CLI complexity, I can highly recommend doit... it's nice for multi-language projects like this.

key: ${{ runner.os }}-pip-lint-${{ matrix.python-version }}-${{ hashFiles('setup.py') }}
restore-keys: |
${{ runner.os }}-pip-lint-${{ matrix.python-version }}-
${{ runner.os }}-pip-lint-

- name: Install linting dependencies
run: |
pip install . pre-commit

- name: Check pip environment
run: |
pip freeze
pip check

- name: Cache yarn
uses: actions/cache@v1
with:
path: .yarn-packages
key: ${{ runner.os }}-yarn-${{ hashFiles('yarn.lock') }}
restore-keys: |
${{ runner.os }}-yarn-

- name: Install yarn dependencies
run: |
yarn --prefer-offline --no-optional

- name: Lint
run: |
Expand All @@ -42,69 +80,111 @@ jobs:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: [3.6, 3.7, 3.8]
python-version: [3.6, 3.7, 3.8, 3.9]

steps:
- uses: actions/checkout@v2

- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v1
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies

- name: Install python setup dependencies
run: |
pip install --user --upgrade pip setuptools wheel

- name: Get pip cache dir
id: pip-cache
run: |
echo "::set-output name=dir::$(pip cache dir)"

- name: Cache pip
uses: actions/cache@v1
with:
path: ${{ steps.pip-cache.outputs.dir }}
key: ${{ runner.os }}-pip-test-${{ matrix.python-version }}-${{ hashFiles('setup.py', 'docs/requirements.txt') }}
restore-keys: |
${{ runner.os }}-pip-test-${{ matrix.python-version }}-
${{ runner.os }}-pip-test-

- name: Install python test dependencies
run: |
python -m pip install --upgrade pip
pip install -e .
pip install -r docs/requirements.txt
pip install . -r docs/requirements.txt

- name: Check pip environment
run: |
pip freeze
pip check

# Build the docs
- name: Build docs to store
run: |
export PATH="$HOME/miniconda/bin:$PATH"
sphinx-build -b html docs/ docs/_build/html -W --keep-going

# Run tests
- name: Run the tests
run: pytest
run: |
pytest


# Run local Lighthouse audit against built site
audit:

runs-on: ubuntu-latest
strategy:
matrix:
python-version: [3.8]
python-version: [3.9]
env:
PORT: 8000
PORT: 8080
# the host interface to listen on, might need to be 0.0.0.0
HOST: 127.0.0.1
# the base url
URL: http://127.0.0.1:8000
URL: http://127.0.0.1:8080

steps:
- uses: actions/checkout@v2

- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v1
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies

- name: Install python setup dependencies
run: |
pip install --user --upgrade pip setuptools wheel

- name: Get pip cache dir
id: pip-cache
run: |
echo "::set-output name=dir::$(pip cache dir)"

- name: Cache pip
uses: actions/cache@v1
with:
path: ${{ steps.pip-cache.outputs.dir }}
key: ${{ runner.os }}-pip-test-${{ matrix.python-version }}-${{ hashFiles('setup.py', 'docs/requirements.txt') }}
restore-keys: |
${{ runner.os }}-pip-test-${{ matrix.python-version }}-
${{ runner.os }}-pip-test-

- name: Install python test dependencies
run: |
python -m pip install --upgrade pip
pip install -e .
pip install -r docs/requirements.txt
pip install -vv . -r docs/requirements.txt

- name: Check pip environment
run: |
pip freeze
pip check

# Build the docs
- name: Build docs to store
run: |
export PATH="$HOME/miniconda/bin:$PATH"
sphinx-build -b html docs/ docs/_build/html -W --keep-going

# Serve the docs and wait to be ready
- name: Serve the built site
run: |
nohup python docs/serve.py --port=${PORT} --host=${HOST} &
curl --retry 10 --retry-connrefused --retry-max-time 60 ${URL}/index.html

# Run the audit
# TODO: use the hosted API with a secret? would allow for comparison over time...
- name: Make folder for Lighthouse reports
run: mkdir -p /tmp/lighthouse/lighthouse-${{ github.run_number }}
Expand All @@ -121,15 +201,22 @@ jobs:
outputDirectory: /tmp/lighthouse/lighthouse-${{ github.run_number }}
verbose: true

# Store the audit
- name: Upload Lighthouse Reports
uses: actions/upload-artifact@v2
- name: Cache yarn
uses: actions/cache@v1
with:
name: Lighthouse Report ${{ github.run_number }}
path: /tmp/lighthouse
path: .yarn-packages
key: ${{ runner.os }}-yarn-${{ hashFiles('yarn.lock') }}
restore-keys: |
${{ runner.os }}-yarn-

- name: Install yarn dependencies
run: |
yarn --prefer-offline --no-optional

- name: Run the accessibility audit
run: |
python docs/a11y.py --no-serve

# Check the audit for threshold values
# TODO: write this someplace after a PR is merged, and load?
- name: Assess Lighthouse Check results
uses: foo-software/lighthouse-check-status-action@v1.0.1
with:
Expand All @@ -138,6 +225,21 @@ jobs:
minBestPracticesScore: "85"
minPerformanceScore: "10"
minSeoScore: "80"
if: always()

- name: Publish Lighthouse reports
uses: actions/upload-artifact@v2
with:
name: Lighthouse ${{ github.run_number }}
path: /tmp/lighthouse
if: always()

- name: Publish Pa11y reports
uses: actions/upload-artifact@v2
with:
name: Pa11y ${{ github.run_number }}
path: docs/_build/pa11y
if: always()

publish:

Expand Down
1 change: 1 addition & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,3 +104,4 @@ envs/

node_modules/
.vscode
.yarn-packages
2 changes: 2 additions & 0 deletions .yarnrc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
yarn-offline-mirror "./.yarn-packages"
yarn-offline-mirror-pruning true
11 changes: 11 additions & 0 deletions docs/a11y-roadmap.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# These are the pa11y error codes we'd like to fix. PRs welcome!

# TODO: remove this line, left in for documentation and review purposes
# WCAG2AA.Principle1.Guideline1_3.1_3_1.H39.3.LayoutTable

WCAG2AA.Principle1.Guideline1_3.1_3_1.H43,H63
WCAG2AA.Principle1.Guideline1_3.1_3_1.H43.HeadersRequired
WCAG2AA.Principle1.Guideline1_4.1_4_3.G18.Fail
WCAG2AA.Principle3.Guideline3_2.3_2_2.H32.2
WCAG2AA.Principle4.Guideline4_1.4_1_2.H91.A.EmptyNoId
WCAG2AA.Principle4.Guideline4_1.4_1_2.H91.A.NoContent
Loading