Infrastructure: Add missing test for 6 examples #1424

spectranaut · 2020-06-12T21:32:36Z

#1378

This PR adds tests for:

the button idl example
Fixes some references in the combobox grid text file
Adds some missing tests for the rest of the combobox tests
Adds the missing keyboard interaction tests to the spin button date picker

nschonni · 2020-09-30T22:13:19Z

@spectranaut it would be good to rebase this one to see how the regression test comment job handles this one

github-actions · 2020-10-09T15:49:04Z

Regression test coverage:

Examples without any regression tests:

dialog-modal/alertdialog.html

Examples missing some regression tests:

dialog-modal/datepicker-dialog.html:
- textbox-aria-describedby
menu-button/menu-button-actions-active-descendant.html:
- menu-up-arrow
- menu-down-arrow
- menu-character
toolbar/toolbar.html:
- toolbar-tab
- toolbar-right-arrow
- toolbar-left-arrow
- toolbar-home
- toolbar-end
- toolbar-toggle-esc
- toolbar-toggle-enter-or-space
- toolbar-radio-enter-or-space
- toolbar-radio-down-arrow
- toolbar-radio-up-arrow
- toolbar-button-enter-or-space
- toolbar-menubutton-enter-or-space-or-down-or-up
- toolbar-menu-enter-or-space
- toolbar-menu-down-arrow
- toolbar-menu-up-arrow
- toolbar-menu-escape
- toolbar-spinbutton-down-arrow
- toolbar-spinbutton-up-arrow
- toolbar-spinbutton-page-down
- toolbar-spinbutton-page-up
- toolbar-checkbox-space
- toolbar-link-enter-or-space
- toolbar-aria-controls
- toolbar-button-aria-pressed
- toolbar-button-aria-hidden
- toolbar-radiogroup-role
- toolbar-radiogroup-aria-label
- toolbar-radio-role
- toolbar-radio-aria-checked
- toolbar-radio-aria-hidden
- toolbar-button-aria-disabled
- toolbar-menubutton-aria-label
- toolbar-menubutton-aria-haspopup
- toolbar-menubutton-aria-controls
- toolbar-menubutton-aria-expanded
- toolbar-menu-role
- toolbar-menu-aria-label
- toolbar-menuitemradio-role
- toolbar-menuitemradio-aria-checked
- toolbar-menuitemradio-tabindex
- toolbar-spinbutton-role
- toolbar-spinbutton-aria-label
- toolbar-spinbutton-aria-valuenow
- toolbar-spinbutton-aria-valuetext
- toolbar-spinbutton-aria-valuemin
- toolbar-spinbutton-aria-valuemax

Example pages with Keyboard or Attribute table rows that do not have data-test-ids:

dialog-modal/alertdialog.html
- "Keyboard Support" table(s):
  - Tab
  - Shift + Tab
  - Escape
  - Command + S
  - Control + S
- "Attributes" table(s):
  - alertdialog
  - aria-labelledby=IDREF
  - aria-describedby=IDREF
  - aria-modal=true
  - alert

SUMMARY:

55 example pages found.
1 example pages have no regression tests.
3 example pages are missing approximately 50 out of approximately 781 tests.

ERROR - missing tests:

Please write missing tests for this report to pass.

spectranaut · 2020-10-09T15:49:08Z

@jongund, any chance you could code review this PR? It would be nice to get the missing tests down to zero :)

nschonni · 2020-10-25T06:29:00Z

@spectranaut looks like the prettier stuff caused some conflicts

jongund · 2020-10-29T19:10:14Z

@spectranaut

Valarie, did you add the missing tests or just reformat the existing test files to use "prettier"?

spectranaut · 2020-10-29T20:03:16Z

Valarie, did you add the missing tests or just reformat the existing test files to use "prettier"?

@jongund I wrote some tests! I'm not sure why I have a failure on the PR for the js linter though.

nschonni · 2020-10-29T20:09:24Z

@spectranaut there are some in-line comments on the file tab around 'use strict' is unnecessary inside of modules for the linting failures

spectranaut · 2020-10-29T20:17:12Z

@nschonni yeah I'm a bit baffled by them, there aren't errors for the use of "no strict" in other files, I can't replicate locally...

nschonni · 2020-10-29T20:19:48Z

They got removed with #1566
I'm guessing you might need to rebase and/or maybe re-run npm ci locally

jongund · 2020-10-29T20:36:38Z

@nschonni yeah I'm a bit baffled by them, there aren't errors for the use of "no strict" in other files, I can't replicate locally...

I have had this problem with 'strict' in the past too, I don't remember how I fixed it.

nschonni · 2020-10-29T21:14:30Z

OK, tested it out locally:

checkout locally and run npm ci. npm run fix had no changes or errors
checkout locally, rebase with master, and run npm ci. npm run fix updated the 2 flagged files

I think the checkout for CI is using the merge commit onto main, so that's why it's showing the errors there and not with just the local branch merge

spectranaut · 2020-10-29T21:25:13Z

@nschonni I think we did a little duplicate work, but thanks for look into it :) I rebased and fixed and force pushed and the linter now passes. Looking into the regression test failure now.

spectranaut · 2020-10-29T23:03:59Z

Ok I'm a bit confused by the regression test failures. It's a time out waiting for a link when testing "key enter" on the link apg example -- which is code this PR does not touch. We have seen this error before and guessed it had to do with the travis check environment's resources in some case, or sending to many requests to wikipedia in another case. I tried testing just link over on the bocoup branch (bocoup#43) with some console logs to try to get a better sense of what is happening, and it failed on time and passed the next.

Anyway, on the most recent run off all the checks it passes. I'm going to open a backlog issue to track these link related failures in case they come up again.

nschonni · 2020-10-29T23:07:38Z

Maybe another 429 response code like the wikipedia ones for the tab tests before?

rwaldron

I was asked (by @spectranaut) to provide a peer review to help move this forward. I've read through all of the changes here and nothing stands out. When I tested locally, I experienced one test failure that was distinct from failures I experienced when running the test from the master branch, and which is only reproducible on my local machine. I don't consider this a blocking issue.

I've followed the review guidelines and have determined that this work satisfies all of the criteria laid out in that document.

(I'd like to acknowledge that I do not have the same level of familiarity with the project as the regular contributors)

rwaldron · 2020-11-06T17:50:09Z

test/tests/button_button-idl.js

+const ex = {
+  printSelector: '#action',
+  toggleSelector: '#toggle',
+  svg: '#example svg',


For consistency, I believe this should be named svgSelector?

jongund · 2020-11-06T19:07:58Z

@rwaldron
Thank you for the feedback, but right now I am focused on finishing up some other ARIA example test cases, but when I get to toolbar I will review your comments.

spectranaut · 2020-11-06T21:04:48Z

Thanks @rwaldron ! I adopted your feedback, including simplifying one test that remove the failure you got when you run it locally.

spectranaut · 2020-11-06T22:52:23Z

The failing test is due to the flaky link tests currently being fixed in #1600

spectranaut mentioned this pull request Jun 12, 2020

Missing regression tests for button/button_idl.html and dialog-modal/alertdialog.html #1378

Closed

spectranaut force-pushed the issue1378-add-missing-tests branch 2 times, most recently from c4d9749 to c9d2be1 Compare June 15, 2020 16:27

spectranaut marked this pull request as ready for review June 15, 2020 21:44

This was referenced Jul 13, 2020

combobox examples rows "combobox-id" are missing tests #1347

Closed

Bugs with regression test script: tests running when they should not, and not running when they should #1449

Closed

spectranaut requested review from jongund, mcking65 and a11ydoer and removed request for jongund and mcking65 July 24, 2020 21:50

spectranaut force-pushed the issue1378-add-missing-tests branch from 644eaef to 4be7896 Compare October 9, 2020 15:48

Valerie Young added 7 commits October 29, 2020 13:42

Write tests for button_idl

16b3a11

Add combobox-id tests

7d1a865

Fix tests

66b96ad

Add spinbutton key tests

920c115

Link to issue for failing tests

1ed2d22

Mark Page Down on day as failing

ec9659b

Satisfy linter

2be3fcf

spectranaut force-pushed the issue1378-add-missing-tests branch from 756a9b9 to 2be3fcf Compare October 29, 2020 21:11

carmacleod mentioned this pull request Oct 29, 2020

Pill list grid example: increase border and button color contrast for remove buttons #1592

Merged

spectranaut mentioned this pull request Oct 29, 2020

Occasional regression test timeout when waiting for link to load #1594

Closed

rwaldron approved these changes Nov 6, 2020

View reviewed changes

rwaldron reviewed Nov 6, 2020

View reviewed changes

Valerie Young added 2 commits November 6, 2020 12:03

Normalize variable names

307fde8

Apply suggest simplification of test

cf0a699

Merge branch 'master' into issue1378-add-missing-tests

dae3b96

mcking65 linked an issue Nov 8, 2020 that may be closed by this pull request

Missing regression tests for button/button_idl.html and dialog-modal/alertdialog.html #1378

Closed

mcking65 changed the title ~~Write missing tests~~ Infrastructure: Add missing test for 6 examples Nov 8, 2020

mcking65 merged commit 1b2ae58 into master Nov 8, 2020

mcking65 deleted the issue1378-add-missing-tests branch November 9, 2020 00:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Infrastructure: Add missing test for 6 examples #1424

Infrastructure: Add missing test for 6 examples #1424

spectranaut commented Jun 12, 2020 •

edited

Loading

nschonni commented Sep 30, 2020

github-actions bot commented Oct 9, 2020 •

edited

Loading

spectranaut commented Oct 9, 2020

nschonni commented Oct 25, 2020

jongund commented Oct 29, 2020

spectranaut commented Oct 29, 2020

nschonni commented Oct 29, 2020

spectranaut commented Oct 29, 2020

nschonni commented Oct 29, 2020

jongund commented Oct 29, 2020

nschonni commented Oct 29, 2020

spectranaut commented Oct 29, 2020 •

edited

Loading

spectranaut commented Oct 29, 2020 •

edited

Loading

nschonni commented Oct 29, 2020

rwaldron left a comment

rwaldron Nov 6, 2020

jongund commented Nov 6, 2020

spectranaut commented Nov 6, 2020

spectranaut commented Nov 6, 2020

Infrastructure: Add missing test for 6 examples #1424

Infrastructure: Add missing test for 6 examples #1424

Conversation

spectranaut commented Jun 12, 2020 • edited Loading

nschonni commented Sep 30, 2020

github-actions bot commented Oct 9, 2020 • edited Loading

Regression test coverage:

Examples without any regression tests:

Examples missing some regression tests:

Example pages with Keyboard or Attribute table rows that do not have data-test-ids:

SUMMARY:

spectranaut commented Oct 9, 2020

nschonni commented Oct 25, 2020

jongund commented Oct 29, 2020

spectranaut commented Oct 29, 2020

nschonni commented Oct 29, 2020

spectranaut commented Oct 29, 2020

nschonni commented Oct 29, 2020

jongund commented Oct 29, 2020

nschonni commented Oct 29, 2020

spectranaut commented Oct 29, 2020 • edited Loading

spectranaut commented Oct 29, 2020 • edited Loading

nschonni commented Oct 29, 2020

rwaldron left a comment

Choose a reason for hiding this comment

rwaldron Nov 6, 2020

Choose a reason for hiding this comment

jongund commented Nov 6, 2020

spectranaut commented Nov 6, 2020

spectranaut commented Nov 6, 2020

spectranaut commented Jun 12, 2020 •

edited

Loading

github-actions bot commented Oct 9, 2020 •

edited

Loading

spectranaut commented Oct 29, 2020 •

edited

Loading

spectranaut commented Oct 29, 2020 •

edited

Loading