Assertions Discussion

[rachaelbradley]

We will be discussing Assertions for the next few weeks in the working group meetings.

Reference: Assertions Slide Deck

[rachaelbradley]

How WCAG 3 Define Procedures?

From WCAG 3: A procedure may be limited to guidance:

• approved by AGWG and listed in WCAG,
• that references publicly published guidance, or
• that meets criteria specified in WCAG
• or it may be any process that an organization uses to improve accessibility.

How should WCAG 3 define the procedures that Assertions reference?

[mbgower]

assertions are not repeatable or verifiable

On 7/23 call, I stated that I believe that while that may be true, the outcome may be able to be assessed. For example, if someone has an assertion for a process their team uses to arrive at clear wording, and their resulting headings are wordy and poor, one cannot verify the team has done their process, but can point to the outcome to show that either they did not carry it out, or their process sucks.

[bruce-usab]

On call 7/23, I shared my opinion that the current definition of assertions could include activities frequently included in U.S. legal settlement agreements such as consultation with advocacy organizations. From that perspective, some assertions have some ability to be verified.

[mbgower]

it may be valid to state that the assertion cannot be verified from the page's outcome but that the team may be able to show evidence that their process was carried out. I think this is supported on slide 7 of the assertion deck.

WCAG recommends maintaining additional information that an organization can use to improve or validate procedures and assertions.

[GreggVan]

Good finesse. It doesnt say anyone would have access to them -- or that they would have to show them-- but that it would be good to keep such records "to improve or validate procedures and assertions."

[shawnthompson]

Wasn't there talk early on about maybe adding links to documentation in metadata about assertions? I vaguely remember the idea being brought up and thought it was an interesting idea.

[GreggVan]

Key to the concept of assertions is that the ONLY thing to test is that an assertion was made. There is no requirement for a testable outcome (or we should be using the outcome as the measure - not the assertion).

Also note that we can give points for an assertion for higher levels -- but unless we can confirm that companies can make assertions (in the 2000-2008 period we were told that many large companies were not allowed to by legal departments) we would not be able to include assertions as requirements for conformance at the basic (or any regulated) level.

A key part of our architecture then has to be structuring the guidelines so that testable / requireable provisions are right along with recomendations for doing non-testable but just as important things.

[bruce-usab]

Maybe an example which might be point for call today:

Assertion: We have taken steps to ensure we are using plain language. We test our assumptions as described by Federal plain language guidelines
.

[rachaelbradley]

Summary of Questions from Assertions Template Exercise

1. What level should assertions be placed?

• For example, assertions may reference guidance such as a style guide or testing such as AT testing or usability testing that covers multiple guidelines/outcomes.
• Should the assertion be mentioned both at the outcome and a higher level
• Should we frame assertions so they refer to a section of a larger document (example: Guidance that addresses alt text instead of a style guide that addresses alt text)

2. How can a person or organisation with a small website attain the highest level of accessibility?

• E.g. simple website done by an expert may not need internal guidance in the same way that teams do.
• Related question: If a small website/organization meets all the criteria at each level, is that comparable to the assertion?

3. How are assertions tested?
   -Validating an assertion only tests that an assertion was made and correctly stated/documented. Sometimes testers will see sites that are so bad it is obvious the assertion is false. How is that handled?
   -If the documentation that supports an assertion is not public, how are assertions enforced or validated?

4. Can WCAG 3 require an accessibility statement?

• If yes:
  -- What should in included in order to make it useful?
  -- Should they only be required if the organization is using assertions?
  -If no:
  --There seems to be some consensus that an assertion must be made in a publicly available location. Would that location then be in an accessibility conformance report?

[rachaelbradley]

Summary of Discussion from 27 August 2024 meeting

From a documentation perspective, the group was generally in favor of integrating assertions at multiple levels, alongside requirements. Exactly how they are handled will be dependent on conformance decisions but the process we will continue to follow for the near future is building them into decision trees and then look later to see if they can be grouped. We then discussed how assertions might fit into conformance.

Please reply to the new threads below for the follow up questions:

• Should assertions be available to meet a conformance level? Opinions varied widely during the discussion. Some members felt they should be visible but never used to meet conformance at any level, some members felt that they the option to use assertions to meet any conformance level should be left open, and some felt that assertions should not be used at the most basic level of conformance but should be available as one way to meet higher levels of conformance.
• Do assertions need to be made in a place that is viewable by people who have access to the system or only in a conformance statement that is provided by the author? The group generally agreed that assertions should be documented as part of conformance statements but supporting documentation should not be required required to make a conformance statement.

[detlevhfischer]

1. My view is that assertions should be kept clearly separate from the result of any outcome testing and not enter any score. So they would not be on the same level but in another compartment.
2. Since the level of outcome would be unrelated to any assertions made, there is no problem for smaller organisations that don't have or need a formal process.
3. An assertion template may at least link to a doc (policy, process description or styleguide) that substantiates the assertion. If no such link exists, readers can draw their own conclusions. No test of content can reliably substantiate the assertion (because the fact that a site's content indeed demonstrates what is being asserted may be due to factors other than the documented process).
4. My feeling is an accessiblity statement is on the level of policy, not standard.

[GreggVan]

Regrets on the call tomorrow -- conflict came up
here are my thoughts quickly

What level should assertions be placed?
For example, assertions may reference guidance such as a style guide or testing such as AT testing or usability testing that covers multiple guidelines/outcomes.

• should not be required at any level we want picked up in regs
• but YES definitely want them viewable and seen right along with our most basic requirements !

Should the assertion be mentioned both at the outcome and a higher level

• yes -- mentioned at basic outcomes level -- as ways to get points for doing better to get higher level rating

Should we frame assertions so they refer to a section of a larger document (example: Guidance that addresses alt text instead of a style guide that addresses alt text)

• not sure I understand but I think I know. I think it should be specific. If in larger doc - refer to section.

How can a person or organisation with a small website attain the highest level of accessibility?
E.g. simple website done by an expert may not need internal guidance in the same way that teams do.

• in some ways it could be much easier to. in some ways harder
• we should include a mix of both so it is level playing field

Related question: If a small website/organization meets all the criteria at each level, is that comparable to the assertion?

• do not know what that means

How are assertions tested?
-Validating an assertion only tests that an assertion was made and correctly stated/documented. Sometimes testers will see sites that are so bad it is obvious the assertion is false. How is that handled?

• well - if the assertion is that you did training -- a bad site does not mean that you didn't. You may just have bad designers or designers that don't care.
• an assertion is just that. If you make it -- you pass - no matter what the site looks like. (Otherwise it is an outcome you are looking for and not an assertion

-If the documentation that supports an assertion is not public, how are assertions enforced or validated?
-require that it is public to be an assertion
-you can't enforce an assertion. it is or it isnt.

Can WCAG 3 require an accessibility statement?

• in WCAG 2.0 we were told that some companies would not do that. which meant they could not ever conform. which meant - why try.
• instead we said what was required if you did.

If yes:
-- What should [we] included in order to make it useful?
-- Should they only be required if the organization is using assertions?

• I think no - that might discourage assertions.

-If no:
--There seems to be some consensus that an assertion must be made in a publicly available location. Would that location then be in an accessibility conformance report?

• it could but many companies have confidential accessibility conformance reports and only show them when required as part of an RFP to avoid critique and criticism.

[detlevhfischer]

but YES definitely want them viewable and seen right along with our most basic requirements!

My assumption was that what organisations will want or be able to assert will depend on their own processes (human remedying, automatic checks, AI, whatever) and will come in various kinds of granularity (from addressing one outome, to several, to pledging continued conformance the the whole lot) so I fail to see the point of any kind of prefab assertion baked into the basic requirements.

[rachaelbradley]

Should assertions be available to meet a conformance level?

Opinions varied widely during the discussion. Some members felt they should be visible but never used to meet conformance at any level, some members felt that they the option to use assertions to meet any conformance level should be left open, and some felt that assertions should not be used at the most basic level of conformance but should be available as one way to meet higher levels of conformance.

Please reply to this thread with your position and reasoning or react to responses with thumbs up or thumbs down.

[gundulaniemann]

Assertions should not be part of the decision tree, as a strict separation between testable and non-testable outcomes is needed.
The example assertions we have seen so far ask for guidance/guidelines. These are usually internal to a company, thus they cannot be retrieved by 3rd party testers. As a consequence, they are not suitable for any conformance claim.

[GreggVan]

Yes but not the base level -- or the level that we want to have required in regulations UNLESS we determine that all companies can make assertions. Assertions are different than just doing something. It is like obeying a law (stopping at stop signs) and legally asserting that you stop at stop signs. Everyone can do #1 but a lawyer would not let you assert that you do.

we don't want the basic level of conformance include something that a company can't/won't do - or else they can never even try to conform at that level.

[alastc]

Chair hat off - I'm concerned that we might run into equity problems if we cannot use an assertion at the baseline level.

For example, if there is a language based requirement for people with cognitive impairments that can only be addressed by following a process. It could be as important (to end-users) as things which are more objectively tested.

Why shouldn't that be at the baseline level?

Chair hat on - I also appreciate the concern that we don't want to include things at a baseline level that can't be used by some organisations. I think we need to get an answer to that concern sooner than later.

[GreggVan]

Agree with this

But there is a corollary. If some orgs can't do it -- it can't be in a regulation. Or won't be adopted in regulation. And we don't want out base level to not be adaptable......

So again - a key question we need to determine -- "Are assertions something that can be required? That all companies could reasonably do? So far I know know if we have any reliable conclusion. Just old information (my information is old - but seems like it MIGHT (and might NOT) still hold and wishes and hopes and fears. Would be good to get some input from companies and lawyers at regulators? Tough question. We would need a bunch of legal opinions that agree? unghh

[ChrisLoiselle]

Hi @rachaelbradley using https://www.w3.org/WAI/planning/statements/generator/#create as an example, the "statement" provides what standards were applied, conformance status, "your efforts" which then has a subset of organizational measures, which could be interpreted as "assertions" by a company or org of "what they've done". I understand @nitedog and others were on the project team on the WAI statement generator, so perhaps pulling those individuals in on how we can reformat or utilize what WAI has already done to insert assertions in to that project would be beneficial so we don't lose all that previous work that seems to resemble these efforts. In the use case of using https://www.w3.org/WAI/planning/statements/generator/#create the statement wouldn't be used to meet a conformance level but provide a statement on what you've done to get to the level you are stating you've achieved per your assertions and efforts , methods, outcomes of testing, applicable outcomes analyzed, etc.

[rachaelbradley]

Do assertions need to be made in a place that is viewable by people who have access to the system or only in a conformance statement that is provided by the author?

The group generally agreed that assertions should be documented as part of conformance statements but supporting documentation should not be required required to make a conformance statement.

Please reply to this thread with your position and reasoning or react to responses with thumbs up or thumbs down.

[GreggVan]

Not sure what your first part means. You mean that they would have to put the assertion on every page that a person might visit so that they would be viewable by people who access the site on any page? Or do you mean that there is an accessibility statement someplace on the website (example.com/accessibility) where they can view it? (versus only having it available as a legal document if you request it?)

Can you provide a little bit more information about what the two choices are? Or if you have three or more?

[alastc]

(Chair hat off) I'd like to re-phrase the question a bit in order to answer: Should the documentation behind an assertion be publicly available, available to trusted 3rd parties, or always private?

I think from a WCAG point of view, we should only ask for the assertion, yes/no. That someone is making an assertion should be part of a conformance / accessibility statement.

However, all the documentation behind that should not be required by us.

We can provide informative information on how to assess that you are meeting an assertion, and what documentation you should gather as evidence.

The use of that documentation is then up to organizations and their regulators. It could be required to be public from the regulator, or it might be required if there is some legal action. However, I think it would undermine the use of assertions to require the documentation be public, as it is likely to include non-public information such as internal policies and processes.

[ChrisLoiselle]

@alastc

_Re: We can provide informative information on how to assess that you are meeting an assertion, and what documentation you should gather as evidence.

The use of that documentation is then up to organizations and their regulators. It could be required to be public from the regulator, or it might be required if there is some legal action. However, I think it would undermine the use of assertions to require the documentation be public, as it is likely to include non-public information such as internal policies and processes._

1. I believe what we'd need to determine, if we haven't already, is whether or not assertions are in fact testable. @GreggVan has mentioned before, I believe, that they aren't. @GreggVan correct me if I mis-read that in another discussion/issue.
2. If assertions are not , then how would one assess they are meeting an assertion?
3. Is that the company self validating and verifying internally that they are about to make an assertion publicly and then :
4. Making sure they've gathered and have internally (but won't ever share that documentation publicly), the information needed to make that assertion?
5. Or is what you are stating that the we, AG, would provide information on how to assess the assertion itself that the company made, from a customer or potential customer perspective?
   I've heard both sides of assertions on W3C calls, as to whether or not these would be testable, and I feel group consensus would be beneficial to understand how we'd proceed with these moving forward.

[alastc]

Hi @ChrisLoiselle,

Gregg has stated that assertions are testable, and he means that in quite a particular way. I.e. You can determine ("test") that the company has made a positive assertion. Yes/no, have they made an assertion? That is "testable".

He was not saying that the assertion could be validated.

Essentially, validation of assertions would be out of the scope of WCAG3. We can provide advice on how to document assertions, but then it is up to the organizations / regulators about how that documentation is used.

In practice, I would foresee that an organization making an assertion about alt-text, then having obvious 'fails' on the website, is not going to do well with regulators or in legal cases.

[bruce-usab]

Building on @mbgower comment above, that assertions might be validated (but not by source code inspection), an example an example is that if an org asserts they follow a style guide — an audit could/should include the style guide as an attachment.

[ChrisLoiselle]

@alastc thanks for the comment. If you can't validate something that is part of conformance then I would say the weight or applicability of assertions compared to what you did test and validate in actual testing would be worthwhile to include in a conformance statement. If you can't validate an assertion but you can make one, that does not seem fair to end user or a firm trying to make a procurement decision. For me that gets in to the gatekeeper and the gate analogy and if we are excluding from WCAG 3 or including in a phase for baseline or requirements, that is something to be considered for sure.

[GreggVan]

Assertions by their definition cannot be "evaluated" or "validated",. The assertion is just that. An assertion. the only thing to test is whether it was asserted. And if you have it in your hand - you have proof of it. Anything beyond that is not testable.

You can only rely on ISO 9000 and companies' integrity.

The only test of an assertion is that it was made.

it is not otherwise objective or testable

[ChrisLoiselle]

Hi @GreggVan I'm following your comment, but have a question on "And if you have it in your hand - you have proof of it."

Use case: A company asserts that their methods/best practices include using proper labeling for all interactive controls on their web documentation that is public facing.

The evaluator or end user of documentation finds that there are 3 unlabeled form elements for interactions such as submit this post, cancel and reply within the company's documentation forums. Not necessarily testing but just using the product with a screen reader they come up with unusable elements on a page that stop progress.

Is that proof in hand? To me, a company is asserting confidently they are doing something and the user is proving that they have not done so in entirety. If not a test, are you stating that a person can't test or validate an assertion at all? Would this be more of a "dispute" of fact to counter the assertion? How would someone accept this as true without validating ? If not able to protest or dispute, taking something as fact is nice in an Utopian world, but could get messy legally per what was stated, not stated or stated slightly incorrectly.

You prove a hypothesis by testing/validating it, thus if someone is asserting something there has to be a counter to that so I'm just wondering what that would be in terms of the actual "term" you'd use within this thread.

I appreciate your comments, time and thoughts on this topic and all the work we've done together, thank you!

[mbgower]

Use case: A company asserts that their methods/best practices include using proper labeling for all interactive controls on their web documentation that is public facing.

@ChrisLoiselle, please refer back to the Background slide at the front to the Assertions deck Rachael cites at the start of the discussion. Assertions are intended for considerations that cannot be judged by repeatable and verifiable qualitative or quantitative tests. The use case you cite does not fit, since it is clearly testable.

So I think we need to consider use cases that fit the context of the assertion discussion.

Having said that, some of us have already covered scenarios where an outcome is demonstrably poor, thus offering evidence that an assertion by a company is failing to achieve its goal. That is also called out in the second bullet of the Background slide "(results can be assessed)".

So, in such a case, an assertion could be made about steps a company takes to lead to a desired outcome, but if the outcome isn't achieved, the assertion is obviously challengable, either in the process effectiveness or adoption.

[ChrisLoiselle]

@mbgower I agree with what you are stating. I'm stating if we are using this word , the term and all the variations of what does not fit and what fits needs to be explicitly stated. The move to WCAG3 is to make WCAG more plain text driven / understandable. For each term used in WCAG3 , the defined term should offer a "if/when/then" type of scenario of applicability. Someone who doesn't have an enterprise type company behind them to research all things related to W3 and WCAG may get lost in all the terms we are introducing. Someone nonnative to accessibility may get lost on what these term means. The goal is make the technical not overly technical in that someone should not need to outsource to an accessibility auditor / company to verify what is meant by "X", related to WCAG or WCAG3.

For me, the goal is to make the use of terms and the explanations blatantly obvious so we understand this word means this and is defined as such. That would avoid pointing people to links / references to explain what we mean by things.

I totally agree with what you stated and how you stated it. Thank you for your comment! It takes a village to do what we do and I appreciate your insight.

[mbgower]

I agree that a list of assertion examples could help crystalize the discussion. There are a few in the deck, but it would be helpful to flesh a few out to further the discussion.