Selenium Automated Testing cannot catch same violation with aXe devTools plugin

[Tim-Cao]

Description
Manual Testing using aXe devTools with WCAG 2.1 AA standard and the latest axe-core to scan this page can catch some accessibility violations

However, when automated testing using the axe deque selenium api cannot get the above violations.
Would you mind explaining what's the difference between them?

Step to reproduce

1. Clone this repo git@github.com:Tim-Cao/seleniumproject.git
2. Change the chromedriver path in https://github.com/Tim-Cao/seleniumproject/blob/master/src/test/java/com/example/seleniumproject/AccessibilityTesting.java
3. Run the above test

Actual Results
No violations thrown

Expected Results

[Zidious]

Hey @Tim-Cao,

I've investigated this and found a few things, within your Gradle file you are using a older version of our API V4.3.1 which utilises axe-core 4.3.1 vs the extension using axe-core 4.4.2.

I'd recommend to update to v4.4.2 of our Selenium API .

Within your test:

withTags(Arrays.asList("wcag21aa"))

One important thing to note here, specifying wcag21aa only runs rules associated to WCAG2.1AA.

Selecting WCAG 2.1AA via the extension includes: WCAG2.0A/AA and WCAG2.1A. To mimic this behaviour with the selenium API you would need to include the following tags:

withTags(Arrays.asList("wcag2a", "wcag2aa", "wcag21a", "wcag21aa"));

Running the below snippet produces four rule violations and one incomplete (needs review within the extension) summing the nodes totals 10 issues (breakdown below):

Snippet:

    WebDriver webDriver = new ChromeDriver();

    webDriver.get("https://www.liferay.com/");

    AxeBuilder axeBuilder = new AxeBuilder();
    axeBuilder.withTags(Arrays.asList("wcag2a", "wcag2aa", "wcag21a", "wcag21aa"));

    /* accept cookies popup */
    webDriver.findElement(By.cssSelector("#onetrust-accept-btn-handler")).click();

    Results results = axeBuilder.analyze(webDriver);

    List<Rule> violations = results.getViolations();

    /* denoted as needs review within the extension */
    List<Rule> incomplete = results.getIncomplete();

    int violationNodeCount = violations.stream().mapToInt(v -> v.getNodes().size()).sum();
    int incompleteNodeCount = incomplete.stream().mapToInt(v -> v.getNodes().size()).sum();

    System.out.println("Sum of violations (nodes): " + violationNodeCount);
    System.out.println("Sum of incomplete (nodes): " + incompleteNodeCount);
    System.out.println("Total: " + (violationNodeCount + incompleteNodeCount));

    AxeReporter.writeResultsToJsonFile("./liferay.json", results);
    webDriver.close();

Outputs:

Sum of violations (nodes): 8
Sum of incomplete (nodes): 2
Total: 10

[Tim-Cao]

Many thanks @Zidious. It works now.