Notify Screenreader Users When Inline Suggestions Or Decorations Available #130565
Comments
isidorn
added
the
accessibility
|
In general we need a more powerful hidden I think this is something we will need in general, however I am not sure if it would be possible to have this with the current DOM apis. In case that is not possible than we would need a more specific solution for this particular case (like the status bar updating, or a quick announcement) fyi @zersiax |
The status bar updating doesn't ping the screenreader afaik. So that wouldn't really help much. What do you mean by a quick announcement exactly? |
|
@zersiax that we aria live announce whenever there is an inline suggestion. Though this might be too noisy... |
|
Hmm ... mostly out of curiosity, would the same not work for other ornaments? We'd probably need a preference section to dial in what a person wants aria-live-announced at any given point, but it feels to me that if we can detect suggestions and aria announce them, could we not do similar things for the others, granting that this would be quite noisy if you have it on all at once? |
|
What would the aria-live-announcement look like? Would it say something like "(Inline) Suggestions available"? Or would it also include the suggestion text? Maybe we could also explore playing a super short non-speech sound that could possibly be played in parallel to any text speech output? |
|
Since this would be noisy I think it should be as short as possible and not include suggestion text. So that the user is notified but if the user wants to know more I like the idea of super short non-speech sound, even though we try not to do this in general and we leave this to screen readers. We try to provide the context, and we leave the announcing and producing sounds to the screen reader. However I am open for exploring this option. |
|
@joanmarie @LeonarddeR is there some way to make audio cues to the screen reader through aria attributes? In this particular case we would like a short "notification sound" to be sent to the user. And there are some other use cases where this could be helpful. Thanks |
|
@isidorn Not in Orca. But could VSCode itself play sounds? (i.e. why does this need to be done through the screen readers?) |
I think it would be nice if various standardized notification sounds were a feature of screen readers:
|
|
Agreed with @joanmarie. Probably best to do this in Code itself. I believe they planned something like that for full Visual Studio as well. |
|
Agreed that sounds would be the best way to fix this, although I recognize that's more effort than adding an aria alert. This is for instance, how NVDA notifies users that there are search results available when typing in the start menu. Super short sound that can be easily understood parallel to speech. |
|
@hediet nicely outlined what are the benefits that this is handled on the screen reader side. Apart from that we would need a different dependency based on platform, and for me it feels like a natural separation between screen reader and app: screen reader is in charge of producing audio (whether it is speech or sounds). Though this might be just my interpretation. I have also emailed Dante from Visual Studio, so we hear what Visual Studio is doing. |
|
If we were to do it through an aria |
|
@isidorn Regarding "need a different dependency based on platform", found this from a quick search: https://code.tutsplus.com/tutorials/the-web-audio-api-adding-sound-to-your-web-app--cms-23790. |
|
@joanmarie I first wanted to check if there is already some aria attribute that could be helpful in this case? But you are right, doing this using an alert text would be hacky, since live regions typically contain text. Thanks for the sound api. We could investigate to use it in general, there are definitely other use cases on the VS Code side. Though I would love to hear from Dante what Visual Studio is doing. |
@isidorn I both agree and disagree with you in this aspect. Screenreaders read the screen, that is their function. And you'd be absolutely correct in saying that that should include things like earcons; some screenreaders already have this built in up to a point. On the other hand, these particular earcons (lightbulb, breakpoint, error, suggestion etc) are entirely unique to VS Code and wouldn't necessarily be used anywhere else. I don't think there's a single answer to this question currently; but what we do have is knowledge of screenreader capabilities. I can tell you there is no screenreader-agnostic way to have a screenreader make a particular noise at a particular event. This would need to essentially be cusstom-built for VS Code in all screenreaders separately. |
|
Thinking more about this, while I personally enjoy sounds very much, relying on sounds only excludes deafblind people. Another approach could be using contentEditable (#97154) and then provide this info as aria annotations in the editable content. |
|
In the scenario of implementing sounds in the screenreader (not in VS Code), I think something like this could work really well:
...
<div aria-live="polite" class="hidden-status">
</div>
...
...
<div aria-live="assertive" class="hidden-status">
<div aria-notification-type="1">
Inline Suggestions are available
</div>
</div>
...Instead of reading "Inline Suggestions are available" immediately, the screenreader would recognize the attribute A single application should not reuse notification types for different contexts. Screenreader users could then change settings in the screenreader such as:
There could also be other interesting configurable options, where a default could be indicated through some aria attribute:
I think this is an argument for having such an aria-notification-type attribute that is processed by some external applications. A screenreader with support for braille could present this notification haptically rather than acoustically. Haptic notifications seem to be very much out of scope for VS Code.
I would say there could be like 10 different system-wide well-known notification sounds.
They would immediately apply to all IDEs, not just VS Code. I could imagine that such a status notification could also be useful for chat applications, indicating if a user got online or offline. Also, text editors/forms could use error-notifications to point out typos/malformed data. |
|
Good points raised by @hediet |
|
@LeonarddeR makes a good point re: deafblind users, sounds wouldn't work for them. Is the aria annotations idea feasible at all? |
But for deafblind users VS Code (afaik) cannot do anything without a screenreader, which makes an even stronger point for having an aria-attribute that is handled by the screen-reader (that can potentially interact with braille devices etc.).
I agree. |
|
Awareness by screen reader developers of this non-existent ARIA feature will require exposure via chrome/chromium. Most screen readers are getting information about accessible objects via the accessibility tree; not directly accessing elements and their attributes from the DOM. In other words just because you put |
|
Thanks @isidorn for looping me in. I'm the editor owner for Visual Studio (i.e., not Visual Studio Code), and I wanted to chime in on some of the thoughts we're executing from. Visual Studio uses the Windows system for our audio cues (which can be accessed as the "Sound Control Panel" from the Sound section of Windows Settings). This has a couple advantages and a couple disadvantages. Going through the Windows system gives us a single interface for configuring all of the sounds, let's the user use the built in Sound Schemes that Windows provides, etc.... It also allows the system to be used by folks who don't use screen readers. In my mind, dudio cues fall into that same category as subtitles where it's designed for one community, but you don't have to be a member of the community to use it. The most obvious downside is that it's a platform specific system. A cross-platform application like VSCode simply wouldn't be able to use the same system without providing mechanisms in every platform that's supported, which is obviously unreasonable. The other downside is that the audio cues are coming from a different system than the screen reader. As I mentioned, this makes them usable to folks who don't use a screen reader, but now that the two systems are disjoint and not aware of each other, it's entirely possible for the sound to be played over part of what's being read. For instance, if the system is set to make a short beep (say 50ms) when the caret arrives on a line with a breakpoint, when the caret arrives on that line, the screen reader is going to start reading the line and the beep are going to play. We don't really have a mechanism to control when either event occurs and there's no guarantee that the beep isn't going to speak over some part of the text that's important. The solution to the second point is to use extremely short audio cues or reserve audio cues for operations that are independent of user interaction. In the example I gave, the audio cue for arriving on a line with a breakpoint will ALWAYS occur immediately after the user took some action (navigation to the line in question). The screen reader would also be kicking in immediately after the user action, so these two events are almost always going to be in conflict. By contrast, an audio cue that indicates when a test pass has completed or a build operation has completed are going to happen at some arbitrary point in time that has a much lower chance of happening concurrently with a user action. We've experimented with Live Regions in Visual Studio for audio cues, but at least in the very brief explorations we made, it wasn't consistent when the Live Region and the screen reader hit each other. Sometimes the cue would appear at the beginning of the spoken text and other times after (at least, that's what I remember running into). I do think audio cues are extremely important and I want to figure out how to get this right. I sincerely think a beep or something to indicate when the caret has arrived on a piece of code that has an adornment on it (e.g. A compilation error or some Quick Action) as opposed to the screen reader having to explicitly say "Code has an error"... but we haven't quite figured out a general enough solution to it yet. |
Is this that bad? I can imagine that for some subtle audio cues it should be possible to clearly identify both the audio cue and the spoken text at the same time, even if they are played in parallel (I haven't tested this though). |
I'm actually not sure how bad it really is. We don't have a lot of users using the audio cues yet, and for a non-screen reader use who has Narrator speaking at 1x speed, it doesn't feel like it's that bad. But, how will it work in an actual "I'm doing work so my CPU is doing 20 different things and my speech is at 6x or higher". We want to do a prototype and release it in an upcoming version of Visual Studio to get an idea of just how bad it is. But to be clear, this isn't preventing us from moving forward on the prototype. It's just one of the concerns/risks that we've considered while implementing it, and I felt like I needed to at least mention it here. I'd love if the problem turns out to not be a problem... but in this context of VS Code, I think the platform specific issue is the higher order bit. |
|
There are 3 different sounds since yesterday: For lines with foldable areas, with errors and with breakpoints. I think you might have heard the sound for foldable areas. |
|
We are in the process of increasing the sound volume and we are considering if we should do the folded sound only when the region is folded, not foldable. More updates in the next couple of builds :) |
|
Agree on having this folded sound only play when it is collapsed, otherwise it fires too often. Keep up the great work! |
|
Agreed, as well. Folded means there's something to be gained from the sound being there, foldable not so much :) |
|
But then, how could I tell when code can be folded?
Devin Prater
***@***.***
…On Wed, Jan 19, 2022 at 6:41 AM Florian Beijers ***@***.***> wrote:
Agreed, as well. Folded means there's something to be gained from the
sound being there, foldable not so much :)
—
Reply to this email directly, view it on GitHub
<#130565 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ADUMTTVAYUTIT2LNDGDGENTUW2WQRANCNFSM5B6HUZVQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
@devinprater smart question. I think that code in most cases can be folded. It is hard to be in some code area which is not foldable and I would argue that in those cases the user would know that she or he is in the top level scope. Though I do not actually use folding so I would like to hear more feedback. |
|
@devinprater like @isidorn says, code can be foldeed pretty much anywhere and, to my knowledge at least, adheres to the block structure of whatever language you're working with. I don't know if you can fold in, say, the middle of a function, but you can fold at the initial line of one, or the initial line of a loop, conditional, class, that kind of thing. I rarely use it, but mostly because it wasn't all that accessible to do so. This might very well change that, as folding can also be really useful to see things like brace mismatches a lot clearer. |
|
Oh, thanks. Then yeah, it would be test to just have it play a sound when
code is folded. I really hope we can get a sound for when one moves to the
next logical line, so that then word wrap can work. But for now, these
three sounds are a great start! Thanks so much Isidorn for working on this.
Devin Prater
***@***.***
…On Wed, Jan 19, 2022 at 9:52 AM Florian Beijers ***@***.***> wrote:
@devinprater <https://github.com/devinprater> like @isidorn
<https://github.com/isidorn> says, code can be foldeed pretty much
anywhere and, to my knowledge at least, adheres to the block structure of
whatever language you're working with. I don't know if you can fold in,
say, the middle of a function, but you can fold at the initial line of one,
or the initial line of a loop, conditional, class, that kind of thing. I
rarely use it, but mostly because it wasn't all that accessible to do so.
This might very well change that, as folding can also be really useful to
see things like brace mismatches a lot clearer.
—
Reply to this email directly, view it on GitHub
<#130565 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ADUMTTXJFGSBUJOAAKIY2KTUW3M3ZANCNFSM5B6HUZVQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
@devinprater cool. Then let's start with that - only play sound when the region is actually folded. And we can fine tune it in the future. Once we are happy with what we have for these we can think about word wrap - I agree with you that is a problem we should solve. Actually @hediet and @gino-scarpino are doing most of the work, so you should thank them instead :) |
|
This implementation has greatly increased my productivity. I didn't know the utility of folding part of the code. It helps me a lot in navigating the code using a screen reader. Thanks @isidorn, @hediet and @gino-scarpino. |
|
@isidorn @hediet @gino-scarpino this is a great first effort and helps a bunch, thanks a lot for the work so far.
So yeah, I'd say bump up the volume of those two cues and split up "foldable"into "folded" and "foldable", or get rid of "foldable" entirely and ply for "folded" instead :) |
|
Following are my thoughts after using for a day.
|
|
I wonder if NVDA ducking can remedy some of the sound being drown out?
Devin Prater
***@***.***
…On Wed, Jan 19, 2022 at 2:59 PM Florian Beijers ***@***.***> wrote:
@isidorn <https://github.com/isidorn> @hediet <https://github.com/hediet>
@gino-scarpino <https://github.com/gino-scarpino> this is a great first
effort and helps a bunch, thanks a lot for the work so far.
My feedback , also shared on Twitter is the following:
- As was discussed, the "foldable" cue should probably rather be a
"folded" cue, or they should be two different cues, but at that point we're
getting into making config options for the various sounds, as some may
prefer to hear when code is foldable while others would rather depend on
their knowledge of the structure of the code they're working with and not
have it trigger too often. The cue would be more useful if you could
distinguish between foldable and folded, which you currently cannot unless
I missed something :)
- I love the brevity of these sounds, pretty much exactly what I hoped
for, but the hcoice of frequency and amplitude has made them easy to drown
out by text-to-speech voices, particularly ones that speak within a
masculine vocal range; this is not that apparent for the foldable cue
because it uses a lot of highs, but all the more so for breakpoint and
error which are particularly difficult to hear with a screen reader talking
over them
So yeah, I'd say bump up the volume of those two cues and split up
"foldable"into "folded" and "foldable", or get rid of "foldable" entirely
and ply for "folded" instead :)
—
Reply to this email directly, view it on GitHub
<#130565 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ADUMTTV2G6TUNKLH7Q3OG5LUW4Q3JANCNFSM5B6HUZVQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
@florian thank you for the great amplitude and frequency comments. Because Frequency response can vary greatly between humans, I've wondered if assistive sounds could benefit from a tone control. Like on a guitar amp. Your note is absolutely heard. Thanks again!
|
|
Great feedback all, thanks. @jareds I like your point to keep the volume control to the OS application volume level. This is built in on Win, but there are apps on MacOs that control this. Anyways we can add volume control if needed only later. Setting for each sound might be a bit too fine grained for me, let's revisit this once we tune the folded sound which is too triggered too often now. @devinprater I am not sure. Let's ping some nvda people so they are in the loop that we are adding sounds @LeonarddeR |
|
@devinprater the sounds are extremely brief at present so I'm not sure what you mean by "drawn out". If you mean "drowned out" the suggestion makes no sense to me given NVDA's audioducking would make the sounds quieter, not louder :) |
|
Wow you're right. I wasn't thinking at all when I wrote that. Yeah, audio
ducking wouldn't help there. If only keyboards came with vibration motors.
Ah well, yeah, the sounds will have to be made louder or more distinct.
Devin Prater
***@***.***
…On Thu, Jan 20, 2022 at 6:07 AM Florian Beijers ***@***.***> wrote:
@devinprater <https://github.com/devinprater> the sounds are extremely
brief at present so I'm not sure what you mean by "drawn out". If you mean
"drowned out" the suggestion makes no sense to me given NVDA's audioducking
would make the sounds quieter, not louder :)
—
Reply to this email directly, view it on GitHub
<#130565 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ADUMTTRXCJ25JCUL6RFS5ALUW73H5ANCNFSM5B6HUZVQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
@devinprater Haptic Keyboards tied to these sounds would be the whole solution. Agree!
|
|
@isidorn I know I'm not the primary user for these sounds, but I do love enabling theses. Are there any plans to customize what sounds a user wants/does not want? For example, I've love to turn off the "folding" noises for myself but keep the rest. |
|
@misolori we can introduce more granular control later if we feel like there are user looking for this. This can just be a new setting value for |
|
@devinprater wrote:
Actually when NVDA ducking is enabled, the volume of sounds coming from VS Code is lowered. This also applies to JAWS and Narrator ducking I suppose. |
|
I wish desktop screen readers were smarter about what they should duck and what not. For example, even with audio ducking enabled, it doesn't make sense to duck system notification sounds, or even the sounds the scren reader itself plays occasionally. On iOS, this works great, but on both Windows and Mac, either everything is ducked, or nothing. So if screen readers were smarter about that, that setting would not even matter for kind of system notification sounds like the one VS Code is producing now. |
|
@MarcoZehe not ...entirely accurate, I believe. Pretty sure VS Code uses its own thing to play sounds, not so much the same process that , say, the Windows Sound Scheme sounds use. VS IDE does try to hook into that same system for build complete and such, I think. |
|
Since we have introduced sounds for breakpoint, errors and folded region I will close this issue. We are really looking forward to feedback and for ideas for other existing features that might benefit from the use of sound. Thanks all! |
and others
This is a follow up to the feedback given in #129677:
Screenreader users are currently not being informed if there are inline suggestions available.
They have to check explicitly if there are inline suggestions available.
We should explore a push-based approach.
However, if there is almost always an inline suggestion (as it is the case with Copilot), too many notifications might get spammy very quickly.
A solution here might also be applied to diagnostics and quick fixes.
The text was updated successfully, but these errors were encountered: