Axe Randomly Fails to Inject on IE Selenium Grid 3.14.0

[emilyrohrbough]

Bug Report

Description:

This is an issue related to Selenium 3.14.0 IE driver and the axe command.

When injecting axe on the test page, the axe command crashing the current selenium session. This is not an issue for all test urls, but for certain test urls; although these urls seem random, errors can be seen consistently on these urls.

Details:

When calling, Terra.it.isAccessibly() the terra axe command is leveraged. The axe-command

• Checks if axe-core exists on the dom and will synchronous inject it on the page if it does not exist.
  • For some test urls, the axe-core injection is silently failing. (we currently do not check the result)
• Then, it asynchronous runs axe to determine the accessibility results.
  • because the axe-core injection failed, axe is not available to run
  • because this is an asynchronous execution, a callback must be invoked to signal the func is finished
  • currently, the axe-core will call this callback when it finishes
  • however, because the run failed, the entire selenium session bonks and terminates itself which results in selenium unable to take the failure screenshots
  • Then, wdio sees this as a test failure and moves onto the next Mocha test to execute, which then fails with:

What needs to be done:

1. Investigation is needed to understand why injecting axe-core on the test page is failing for various test pages and only for Selenium 3.14.
   1. A potential reason could be selenium issue: [IE-Driver] "Error executing JavaScript" when accessing TextContent attribute on re-inserted element SeleniumHQ/selenium#6538 doing a quick ctrl+f on the axe-core file we inject in the dom, axe does uses the textContent attribute.
2. Add try-catch logic to the runAxeTest method to ensure a callback is always called in browser.executeAsync to prevent crashing/terminating the selenium session.

Environment

• Browser Name and Version: IE 11 Selenium Grid Driver 3.14.0-helium

@ Mentions

@ryanthemanuel @mjhenkes

[JoeJennings]

Any updates?

[nramamurth]

@emilyrohrbough

How to reproduce the issue in-house and then further verify changes?

[emilyrohrbough]

@nramamurth You should sync up with @benbcai. He spent some time working through this, but I am unsure of the full extent to what he discovered.

[benbcai]

@nramamurth @emilyrohrbough When I was investigating this what I discovered was that on IE Selenium Grid 3.14 that axe would fail to inject on any page that has a terra-icon. I used these test examples for troubleshooting and what I noticed was the error would occur on any of these test pages that renders a list-item with a chevron. If I removed the chevron, then it would pass. To rule out that this is not related to terra-list, I created a test page with just a chevron icon (without list-item or list), axe would still fail. I then tried rendering a different icon and it would still fail. From what I can tell up to this point, axe would fail on a page that has any terra-icon..

We lock axe-core to 3.2.2 so I wouldn't be any recent changes in axe-core that is causing this. Since Selenium Grid was bumped from 2.xx to 3.14, perhaps this issue could have existed not only in 3.14 but older versions of 3.xx. I was going to build a box with older versions of the grid to pinpoint which specific 3.x version introduced this to see if that would lead us to new findings but my priorities have shifted and haven't done that. I will continue the investigation when I can get back to this.

[JoeJennings]

Any updates?

[benbcai]

When the error occurs on a page that contains only an icon, the following is the Error object that is returned in the error result from executing axe.run.

TypeError: Can't call method on undefined
{
[functions]: ,
proto: { },
message: "Can't call method on undefined",
name: "TypeError",
stack: "TypeError: Can't call method on undefined
at t.exports (http://10.132.20.102:8080/core-js-2b73a16cefac888ed650.js:1:5108)
at t.exports (http://10.132.20.102:8080/core-js-2b73a16cefac888ed650.js:1:4295)
at t.exports (http://10.132.20.102:8080/core-js-2b73a16cefac888ed650.js:1:34710)
at xe (Unknown script code:12:104282)
at Anonymous function (Unknown script code:12:104600)
at xe (Unknown script code:12:104282)
at Anonymous function (Unknown script code:12:104600)
at xe (Unknown script code:12:104282)
at Anonymous function (Unknown script code:12:104600)
at xe (Unknown script code:12:104282)",
Symbol()_7.mwk153tfynm: undefined,
Symbol(observable)_w.mwk153tfykd: undefined,
Symbol(react.async_mode)_v.mwk153tfykd: undefined,
Symbol(react.concurrent_mode)_q.mwk153tfykd: undefined,
Symbol(react.context)_p.mwk153tfykd: undefined,
Symbol(react.element)_j.mwk153tfykd: undefined,
Symbol(react.forward_ref)_r.mwk153tfykd: undefined,
Symbol(react.fragment)_l.mwk153tfykd: undefined,
Symbol(react.lazy)_u.mwk153tfykd: undefined,
Symbol(react.memo)_t.mwk153tfykd: undefined,
Symbol(react.portal)_k.mwk153tfykd: undefined,
Symbol(react.profiler)_n.mwk153tfykd: undefined,
Symbol(react.provider)_o.mwk153tfykd: undefined,
Symbol(react.strict_mode)_m.mwk153tfykd: undefined,
Symbol(react.suspense)_s.mwk153tfykd: undefined
}

The error doesn't provide any information on which object is undefined. I was able to execute other axe APIs and the expected results are returned but fails when executing the run API regardless of the WCAG standards (wcag2a, wcag2aa, wcag21aa, or section508).

[benbcai]

It turns out that the cause of this issue is dequelabs/axe-core#525. For any page in IE that has an SVG element (i.e. terra-icon), accessing .children on an SVGElement on this line in axe-core would return undefined and ends up throwing the above exception. Other browsers treat SVGs as HTMLCollections but IE treats them as SVGElement. See this post for more details.

This error can be avoided/skipped by changing restoreScroll to false here in terra-toolkit to skip scroll restore but I don't think we can do that since doing so would break tests.

dequelabs/axe-core#525 is scheduled to be fixed either in axe-core 3.4 or 3.5 but I don't think we want to wait that long. We could make a contribution to get this fixed ASAP.

[emilyrohrbough]

@benbcai nice find!!

[benbcai]

Contribution to axe-core to fix this: dequelabs/axe-core#1820

[benbcai]

axe-core-3.4.0 with the fix is expected to be released on or around 10/21. Once 3.4.0 is released, terra-toolkit will be updated to pull in this version.

[emilyrohrbough]

axe-core-3.4.0 was released this morning. This issue should be resolved.

[benbcai]

Although the underlying issue has been resolved in axe-core 3.4.0, terra-toolkit currently is unable to upgrade axe-core from 3.2.2 to 3.4.0 because 3.4.0 introduces various other accessibility violations in Terra that need to be investigate and addressed. For that reason, this issue is closed in favor of #343, which will address the impact of upgrading 3.4.0.