aria-post to append additional, contextual information after semantics

[frankelavsky]

Description of bug or feature request

(May or may not intersect with #991.)

I work in accessible data representations and sometimes I'd like to post-pend descriptions after the label and semantics are read out for an element. Currently, most screen readers roughly follow the pattern: "[Label], [semantics], [SR specific stuff]"

I'd like to be able to use aria to do the following between semantics and screen-reader specific announcements:
"[Label], [semantics], [post-label, post-semantics stuff], [SR specific stuff]"

Perhaps a new aria attribute such as aria-post, aria-contextual, or aria-additional could help with this. (Oh, and please let me know if something like this already exists! I'm obviously not aware, if so.)

To me, the use cases largely involve instructions for using a custom interactive element, contextual information (such as how elements might relate to others or data-related context), or for notes/non-essential/potentially redundant label information.

In data visualization/navigable data experiences it is common to have a lot of label bloat at the element level, for example marks in a scatterplot might all have data-related descriptions (aka the label), standard semantics (img or button or link or something), but then also potentially have special information that might be nice to move after the semantics. Examples would be "part of the North America Group," "has 6 siblings," or "press F1 for keyboard controls," etc. These can sometimes feel redundant when many elements are the same and can be painful to wait for, especially when elements have similar keyboard controls but the semantics might be different within a chart (such as img versus link). For now, I've just been adding all of this info into the label. But sometimes I think having semantics first would be helpful!

As some context, there is some research by the folks at MIT (by @jonathanzong and @shulijones) on customization of text at a low level, so there is some precedence here for the bigger picture idea that text label and ordering could even be adjustable beyond this proposal's scope.

Looking forward to discussion on this!

Will this require a change to CORE-AAM?

Unsure.

Will this require a change to the ARIA authoring guide?

Possibly.

[jnurthen]

@cookiecrook please look for any status related issues

[cookiecrook]

Hi @frankelavsky! 👋

Unfortunately I did not find the other "status" related ARIA issue I recalled, but conceptually, this is similar to "Status" in most screen readers... For example the "checked" state of a checkbox with a particular label. Each control type comes with different expectations about when (or even if) role, name/label, or other features are spoken. Low verbosity usually drops roles on most items, and status on some.

Here's an example of the customizability of verbosity announcements... Users can remove or reorder what they want to hear, and even store difference verbosity settings on a per-app or per-site basis.

VoiceOver (and presumably other SRs) allow users to replace a role description for some types (e.g. "link") with a unique sound icon. So I'm a little worried that a general "append this string to the end" attribute won't be well-received by users that like fine control over their screen reader interface. (Though now that we have aria-description, I acknowledge I didn't succeed in eradicating the world of longdesc)

We do have more specific concepts, like, aria-describedby (IDREF), aria-description (string), aria-errormessage so I perhaps one of those may work for the more specific case you're trying to achieve?

[jonathanzong]

Thanks for tagging me in the OP @frankelavsky, and thanks for the helpful response @cookiecrook on how this might overlap with existing concepts. Since Frank mentioned some of the research we've been doing, I thought I'd provide some additional context and point out where it might be interesting to consider something new.

Descriptions as sets of affordances

Like Frank says, different parts of a description address different user needs, and not every part is relevant in every situation. In our work, we've been thinking about breaking down descriptions into tokens of text, where each token has an affordance that addresses a different user need. For example: what is the element I'm on, where in the structure am I, and what is around me that I can navigate to — these are all affordances that would need their own token.

Figure 1 from the customization paper, which is accessible as a table in the HTML paper, showing the different kinds of user affordances a textual token might address (in visualization, but potentially also more generally):

In our paper, we use the token model as a basis for customization: tokens are the unit of text you can re-order, turn on and off, and choose between longer/shorter variants of.

Affordances are domain-specific

Currently, like cookiecrook points out, there is some implicit notion of tokens in ARIA. Each control type has a different set of valid tokens (e.g. name, status, type), which you can turn on and off and reorder. However, currently aria-description as a token is kind of a catchall for everything an author would provide.

What I think Frank is pointing out is that for domains like visualization, there are domain specific tokens that a user would want more granular control over. For example, maybe we want range, index, average, etc. tokens. So for instance, if we take an example description from our research prototype:

Figure 3, see HTML paper for SR accessible version:

We want to be able to treat Values from Jan 1, 2006 to Jan 1, 2008., 5 of 7., 24 values., and The average price ... as distinct tokens that a user can adjust independently. Currently, if we stuff all of this into a single aria-description because it is all author-provided information, we can't offer a user that level of granular adjustment. (In our paper, we prototyped a customization interface and tested it with BLV users, for some initial evidence that this would be good to do.)

tl;dr: especially for domains like visualization, which have domain-specific concepts that aren't in general-purpose docments/forms, it would help to be able to offer granular control over different tokens of text that address different situational user needs

[cookiecrook]

tl;dr: especially for domains like visualization, which have domain-specific concepts that aren't in general-purpose docments/forms, it would help to be able to offer granular control over different tokens of text that address different situational user needs

I would agree with that, but this issue was a general aria-post proposal, which i interpret as "tack this on to the end of the speech string." I don't think that would resolve the need for more granular control for an author, and especially not to an end user.

[frankelavsky]

So I'm a little worried that a general "append this string to the end" attribute won't be well-received by users that like fine control over their screen reader interface. (Though now that we have aria-description, I acknowledge I didn't succeed in eradicating the world of longdesc)

Heh, yeah that's a fair worry for sure. Perhaps an aria-post approach isn't great because it is too simplistic. And I think that the longdesc approach perhaps rightfully should be used with caution. Hmmm I'm not sure there is a good approach here. Capturing the spirit of the more specific problem I run into is really (probably) just about wanting a standard way to communicate additional, functional information about the use of elements in an interactive context (especially when using role=application). Maybe what I want is actually a standard role for input/command instructions that can accompany a role=application element (such as an immediate child of that element or something that has role=instructions etc).

There are also contextual pieces of information that can feel redundant when repeated a lot, which I also wish there was a smart way to provide more control over. And I think that is getting more towards @jonathanzong's part of the discussion. A current approach in some prototypes I've built has been to allow for customizing verbosity and information order in alt text by providing some interface to the user within our environment before interacting with elements. "What information do you want to hear first? Numbers, categories, or groups?" "Do you want to hear data key names? Yes/no?" etc. One of the major downsides to this is obviously that this is non-standard and the work of customization has to be repeated in new environments. Perhaps asking for ARIA to handle a granular type of semantics standardization for my own domain is a bit much. I'll stew on this particular use case more, I think.

But in role=application in particular (going back to my first paragraph response here)... I really do think that it should be an expected standard to provide interaction instructions (for example, keyboard shortcuts at the very least). And I think that having multiple ways for users to customize how they experience this information would be helpful, while only expecting authors to create a single "area" for the instructions.

Element-level, repeated instructions can be painful for an experienced user but are really helpful for novices of a particular interface. Maybe the solution really is just to give users a toggle or way to dismiss instructional information at the application-level and not leave it up to ARIA for this? Hmm...