fixrule(meta_refresh_delay | meta_redirect_optional) Reports "cannot tell" and "passed" while ACT bisz58 expects "Failed" for 3 Fail Example test cases

[philljenkins]

Checker rule meta_redirect_optional is also involved with these ACT rule(s) below.

Checker rule meta_refresh_delay reports inconsistent with ACT rule(s) below:

• Failed examples should be failed by at least one of the Checker rules fail reason codes.
• Checker should be mapping (report against) the same success criteria as what is expected by the ACT rule.

4. Partially inconsistent with ACT rule Meta element has no refresh delay (no exception) bisz58 with 3 failed test cases:
   • "cannot tell" Failed Example 1
   • "passed" Failed Example 2
   • "passed" Failed Example 3
   • Mapping: Checker reports against 2.2.1 Timing Adjustable, while ACT expects both 2.2.4 Interruptions AAA and 3.2.5 Change on Request AAA.
   • RuleID: meta_refresh_delay with ReasonID: Potential_1 reports against bisz58:cantTell | bc659a:cantTell
   • RuleID: meta_refresh_delay with ReasonID: Pass_0 also reports against bisz58:pass | bc659a:pass
   • RuleID: meta_redirect_optionalwith ReasonID: fail_longrefresh flags (doesn't report to ACT) this Fail Example 2 test case as Violation, but it's mapped to 2.2.1 Timing Adjustable while ACT would expect it to be mapped to 2.2.4 Interruptions AAA and 3.2.5: Change on Request AAA. Again, this ...optional rule does not currently report against this ACT bisz58. The ...optional rule currently only report against ACT bc659a.
   • Secondary requirement: ACT considers 2.2.1 Timing Adjustable (Level A) a "secondary requirement" because 2.2.1 is less strict than this ACT bisz58 rule. 2.2.1 allows redirects longer than 20 hours. The meta element in Failed Example 2 redirects after 20 hours.

Reference

• Checker's Implementation report of ACT rules https://www.w3.org/WAI/standards-guidelines/act/implementations/equal-access/ (updated 7 November 2024)
• Checker's raw results sent to ACT https://unpkg.com/accessibility-checker-engine@3.1.77/act-report-v2.txt
• Checker's Rule mapping spreadsheet file on box

[philljenkins]

See latest Rule mapping spreadsheet:

[philljenkins]

Relevant Artifacts

• these are auto-built with master 3.x and main-4-x, which includes the ACT reporting failures.

master 3.x artifacts examples:

• ACT Text Results
• Rule Listing
• Rule mapping spreadsheet

main 4.x artifact examples (to be used for ACT analysis/QA):

• ACT Text Results
• Rule listing
• Rule mapping spreadsheet

[philljenkins]

Ignoring these test cases in ACT mapping/reporting. We test/QA to be concerned about fail test cases not failed by Checker and correct WCAG SC reporting in this main 4.x Artifact ACT Text Results 2025-01-14 from PR #2158 :

Ignore

• 6a7281 Passed Example 5: Expected passed, but returned earl:failed

• akn7bn Inapplicable Example 5: Expected inapplicable, but returned earl:failed

• akn7bn Inapplicable Example 6: Expected inapplicable, but returned earl:failed

• 73f2c2 Passed Example 8: Expected passed, but returned earl:failed

• 7d6734 Passed Example 2: Expected passed, but returned earl:failed

• etc.

• the above steps need to be re-done when now PRs are available

[philljenkins]

• the analysis/steps below need to be re-done when now PRs are available:

Expected Failed

These are somewhat of a surprises in 4.x. These ACT "fail" test cases should be reported as failed, but were not in Checker 4.x Artifact ACT Text Results 2025-01-14 from PR #2158

Although this is not completed, if these are not resolved they would results in more inconsistencies than previously reported:

ACT Rule 23a2a8 Image has non-empty accessible name:

• 23a2a8 Failed Example 2: Expected failed, but returned earl:inapplicable. Checker should fail this test case.

ACT Rule 7d6734 SVG element with explicit role has non-empty accessible name:

• 7d6734 Failed Example 4: Expected failed, but returned earl:inapplicable. Checker should fail this test case.

ACT Rule b4f0c3 Meta viewport allows for zoom:

• b4f0c3 Failed Example 1: Expected failed, but returned earl:inapplicable. Checker should fail this test case.
  • matches URL at act/rules/b4f0c3/#test-cases accc6adf094723693593ca3c6308f81945930dae.html

These next two appear to be the same test case "Failed Example 2", but have different URLss.

• b4f0c3 Failed Example 2: Expected failed, but returned earl:inapplicable. Checker should fail this test case. Note URL a1240b31761f65c92a8f6d08ed7105ee822d0ebc.html
  • matches URL at act/rules/b4f0c3/#test-cases a1240b31761f65c92a8f6d08ed7105ee822d0ebc.html
• b4f0c3 Failed Example 2 Expected failed, but returned earl:inapplicable. Checker should fail this test case. Note URL 8735b02e59dd802a5a9e9a7ea1934a106d2963eb.html
  • does not match URL at at act/rules/b4f0c3/#test-cases
• b4f0c3 Failed Example 3 Expected failed, but returned earl:inapplicable. Checker should fail this test case. Note URL c7e4980eb42d0b138684ac1e28318a197d86ccfc.html
• b4f0c3 Failed Example 3 Expected failed, but returned earl:inapplicable. Checker should fail this test case. Note URL c94a59f8c3b17d722781af36da3556ff4b418776.html WAR The value "invalid" for key "user-scalable" is invalid, and has been ignored.
• b4f0c3 Failed Example 4 Expected failed, but returned earl:inapplicable. Checker should fail this test case.

[philljenkins]

Moving back to In Progress until @shunguoy creates new PRs ready for

1. master 3.x
2. main-4-x

ACT testing/QA will be done on main-4-x per procedures in Updating, building, and testing Checker Rules & Help
following the 8 steps in Maintaining both a 3.x and 4.x master

[philljenkins]

Testing with PR #2168 that includes the main-4-x branch and code fixes.

[philljenkins]

@tombrunet @shunguoy while testing PR #2168 I noticed that the Checker still reports against 2.2.1 Timing Adjustable while ACT expect it to be mapped to 2.2.4 Interruptions AAA and 3.2.5: Change on Request AAA.

But the Checker doesn't support AAA, so how do we resolve this inconsistency?

This was noted above in #2091 - Partially inconsistent with ACT rule Meta element has no refresh delay (no exception) bisz58. The ACT rule does list 2.2.1 as a secondary requirement, but that is not figured into the consistency mapping:

Secondary Requirements
This rule is related to the following accessibility requirements, but was not designed to test this requirements directly. These secondary requirements can either be stricter than the rule requires, or may be satisfied in ways not tested by the rule:

• 2.2.1 Timing Adjustable (Level A): This success criterion is less strict than this rule. This is because this criterion allows redirects longer than 20 hours. Some of the failed examples satisfy this success criterion.