VS Code native notebook accessibility improvement

[rebornix]

Over the last couple of months, we have been receiving feedbacks about the accessibility of the native notebook support in VS Code. Here are some of the major issues we've heard:

• Navigation can be confusing
  • There isn't any easy/intuitive way to navigate from cell input to output
  • Arrow down jumps to next cell if the cursor is at the last line of the current cell
• Monaco editor inside the notebook is not accessible even in accessible mode (document mode)
• Screenreader
  • No aria label for cells
  • No indication of another cell is being activated/focused
  • No indication of how you can move focus into the editor, move to the container, extra shortcuts to move to cells
  • Outputs are not accessible

We are going to explore the proper approaches to address above issues, the bottom line is we align our experience with Settings Editor which is also a list view with rich list items rendered inside.

Current navigation model

The notebook structure in VS Code is basically a list. So keybindings for list view will work in the notebook editor as well, like arrow up and down move focus between cells.

The major difference between notebook and other list views is notebook cell can contain a nested monaco editor. For example, a code cell consists of a monaco editor for input, an output container for output rendering and also some toolbars. Users can press enter to focus the nested monaco editor, or press escape to move the focus back to the list view (still focused on the cell container). Markdown cell is slightly different but the concepts are almost identical, markdown cells are rendered as HTML preview by default, when users focus on the cell, they can double click, or press enter to turn the cell into editing mode, the HTML preview will be replaced by a nested monaco editor. Once finished editing, users press escape to switch the markdown cell to preview state again.

As described above, there are two focus modes: one is focus on the cell list item, the other is focus on the nested monaco editor. The behavior of arrow keys depend on which focus mode users are on. If the focus is on the cell list item, arrow keys will move focus between cells, just like other list views. While if the focus is on the nested monaco editor in a cell, arrow keys on first line or last line of the cell will jump to the previous/next cell's nested monaco editor. This design allows users to quickly navigate across cells and edit the cell inputs without pressing enter and escape multiple times.

Improvements collected from discussions

• Do not switch to editing mode on enter when the DOM focus is on a link in a markdown cell preview
• Aria alert cell index 0 to cell 1 (1 based, and omit index)

[rebornix]

Some of issues mentioned above were already addressed:

• There isn't any easy/intuitive way to navigate from cell input to output
  • We introduced three commands
    • notebook.cell.focusInOutput to focus into the output of the current cell
    • notebook.cell.focusOutOutput to move focus out of the output
    • notebook.focusNextEditor to move focus to the editor in next cell when the focus is inside an output
  • You can always use the same keybinding command+ctrl+arrowdown
• Outputs are not accessible
  • When the workspace is in accessibility mode, we will try to render accessible output. For example, if there is a multi-mimetype output which has both text/markdown and text/html mime type, we will render text/markdown with builtin markdown renderer.

Our plans for polishing for some of the issues:

• No aria label for cells
  • We will announce "code cell"/"markdown cell" when they are focused.
• No indication of another cell is being activated/focused (especially when moving from one nested editor to another)
  • We will announce "move to next cell editor".
• No indication of how you can move focus into the editor, move to the container, extra shortcuts to move to cells
  • We will announce "cell container focused", "cell editor focused", "cell output focused" with shortcuts to focus other components.

[zersiax]

@rebornix Looks like we're off to a good start here. I'd say the biggest currently unsolved issue that remains is that the nested monaco editors are currently inaccessible irrespective of the screenreader mode setting. Even outside of VS Code, the editor that is used on , say, the try Jupyter pages currently isn't as accessible as it could be, so I'd say having Code work in this instance would be a major win

[webczat]

actually what? I mean... in my experience nested editors are accessible here on linux, at least when I use dotnet interactive (I don't use python notebooks). I never saw any accessibility problems with the editor itself

[jooyoungseo]

When navigating Notebook cells with either JAWS or NVDA, I hear redundant affix, "Has Flows From", per every single field.

Please kindly address this issue as well. :)

[venkateshpotluri]

thank you @rebornix and everybody on the team for taking this up. I wanted to ask a few questions, which I hope could help have a learnable, complete notebook experience.

• the original comment says "markdown cells are rendered as HTML preview by default, when users focus on the cell, they can double click, or press enter to turn the cell into editing mode, the HTML preview will be replaced by a nested monaco editor.". Is this behavior going to be the same for screen reader users as well? If so, how would a screen reader user navigate to the link in the markdown part of a notebook?
• The original comment says "if the focus is on the nested monaco editor in a cell, arrow keys on first line or last line of the cell will jump to the previous/next cell's nested monaco editor. " I wonder if there is value in letting users trap focus in a cell i.e. pressing down in the last line of the cell does not move users to the next cell. I think this would be useful to prevent accidental jumps from cells that produce large amounts of output text. If I understand this behavior correctly, it may be hard to get back to the original cell.
• It would be nice to have indications that cells are running. This could come in handy with cells that have code that take time to run, training a deep learning model for example.

[webczat]

okay, what I've just noticed is that the focus thing doesn't seem to work correctly. In dotnet interactive/orca when I press ctrl+down arrow I land in browse mode and orca says "virtual document window", however, I don't seem to be on the output area. In actual jupyter I had a similar problem but there after pressing ctl+down I just hear "notebook" and I am still in focus mode.

[zersiax]

Curious where we are at with this one? Any accessibility questions I can answer?
This is a big one, and it would enable a lot of people to get their work done way better and quicker :)

[MarcoZehe]

Adding my voice here in support of better accessibility for notebooks.

[rebornix]

I created a separate branch https://github.com/microsoft/vscode/tree/rebornix/nb-accessibility to track the improvement suggestions mentioned above. The first set of improvements (hopefully) include:

• Aria message when focus on a cell container, inner editor or output. It will tell you the cell index (0 based, I'm wondering if it should be 1 based), cell type (markdown or code), and where the focus is, container/editor/output. Also it says how you can move focus to the container or editor
• Aria message when a cell executes or finishes execution

You can play with above changes in a custom build (Mac, Linux x64, and Window) and try with GitHub Issue Notebook. I didn't push the changes directly to master as I'm not sure whether it's too verbose or not.

---

Answers to a few questions above (I'll try to answer all of them if I have a good understanding of the problem)

@venkateshpotluri

the original comment says "markdown cells are rendered as HTML preview by default, when users focus on the cell, they can double click, or press enter to turn the cell into editing mode, the HTML preview will be replaced by a nested monaco editor.". Is this behavior going to be the same for screen reader users as well? If so, how would a screen reader user navigate to the link in the markdown part of a notebook?

When the markdown cell is in preview mode, you can Tab to the links.

The original comment says "if the focus is on the nested monaco editor in a cell, arrow keys on first line or last line of the cell will jump to the previous/next cell's nested monaco editor. " I wonder if there is value in letting users trap focus in a cell i.e. pressing down in the last line of the cell does not move users to the next cell. I think this would be useful to prevent accidental jumps from cells that produce large amounts of output text. If I understand this behavior correctly, it may be hard to get back to the original cell.

Good idea, we can add a setting for this and use different default values for accessibility mode.

It would be nice to have indications that cells are running. This could come in handy with cells that have code that take time to run, training a deep learning model for example.

Added aria alert for cell running status in above custom build.

---

@zersiax

I'd say the biggest currently unsolved issue that remains is that the nested monaco editors are currently inaccessible irrespective of the screenreader mode setting

Can you help confirm you are using the native notebook editor? You can try GitHub Issue Notebook in VS Code Insiders and check if you can get proper screen reader support?

[zersiax]

@rebornix I will have a play with the try build later today after I've gotten some sleep to confirm what I said in the original comment is still accurate.
Also, what @venkateshpotluri meant is that right now, a pretty common pattern is to activate links by hitting enter on them. Right now, that would seem to conflict with the fact that enter opens a cell in the editor, making enter no longer work to click on links, unless that behavior supercedes the editor opening behavior when focus is on an actual link. I'm not sure how hard that would be to implement.
Pressing space on a link should still work as an alternative pattern, but may throw people off slightly if this isn't clearly noted somewhere in the docs.

[zersiax]

@rebornix ok, I had a quick look around.

The inner edditors work fine with the test build and the github notebooks, nothing breaky there.
I'm not entirely sure how I am meant to get to a cell's output. I can jump into browse mode and get there but I'm not sure if that is the intended behavior. It would make sense given output can be a large variety of different things, but you currently can't seem to tab there. Is there another way of movement I am meant to be using?

Hitting enter on a link within a cell's output frame does not bring up the editor but activates the link, which is good.
I'd say the length of the aria message is fine, perhaps the word " index" doesn't need to be in there, and it can just be " cell 0" rather than " cell index 0". The hint about escape and enter doesn't bother me at all, but others may feel differently so perhaps that bit could be a setting?

[MarcoZehe]

Agree "index" should be omitted. What about the index being 0 based? In tables, I would find this unnatural, rows and columns are always 1-based in screen readers. Same is with tree levels, which are even 1-based in the ARIA spec. Would it make sense to have them 1-based, too?

Disclaimer: I don't know what notebooks are or how one is supposed to use them at all, or what for, so there may be a perfectly good reason for these to be 0-based. If that's the case, feel free to ignore the comment.

[rebornix]

Also, what @venkateshpotluri meant is that right now, a pretty common pattern is to activate links by hitting enter on them. Right now, that would seem to conflict with the fact that enter opens a cell in the editor, making enter no longer work to click on links, unless that behavior supercedes the editor opening behavior when focus is on an actual link. I'm not sure how hard that would be to implement.

@zersiax Thanks for the explanation! It makes sense to open the link when the focus is on the link element, other than switching the cell to editing mode.

[rebornix]

Agree "index" should be omitted. What about the index being 0 based? In tables, I would find this unnatural, rows and columns are always 1-based in screen readers. Same is with tree levels, which are even 1-based in the ARIA spec. Would it make sense to have them 1-based, too?

@MarcoZehe agree that index is verbose and should be omitted. The editor selection is also 1 based and it makes sense to do that in notebook as well. Thanks!

[isidorn]

I did not yet have time to try this out, but I do plan to test it out end of this week.

[zersiax]

@rebornix I'd say that these changes, perhaps with the small bits of feedback can be merged into main. I have a person who actively uses notebooks with a screenreader who might be able to give more useful input, but it sounds like your initial changes, knowing where you are and when a cell is done running, are exactly the two things this person is running into. I gave them the try build to try out and will report back when I know more. Were you planning any other things for this branch?

[isidorn]

@rebornix I agree with @zersiax, feel free to push the changes to insiders to get more feedback. Since a lot of our screen reader community is on insiders.

[rebornix]

@zersiax @isidorn it makes perfect sense to me to merge it into main. I'll do it.

[rebornix]

I have merged the initial changes with tweaks suggested above (omitting index) to main. Tomorrow's Insiders will have it.

[zersiax]

Ok. So, the person I was referring to says that these changes are already a big improvement.
One question they raised is that where before output of all cells appeared below each cell, the output currently seems to be limited to the first few cells. Other cells need to be interacd with first before their output becomes visible. Is this intended behavior?

[rebornix]

@zersiax may I ask how they interact with cell inputs and outputs? The old notebook editor is implemented in a webview and it renders all cell input and output all in the view. While building the new editor, we want to ensure it a good file opening performance so we introduced virtualization, so outputs are only rendered when they are scrolled into the viewport the first time.

[rebornix]

@mohammad-suliman @NiSc91 thanks for your feedback and detailed instructions, it helped me understand how you interact with outputs. Let me try to summarize them in my own words and you can correct me if there is anything misinterpreted:

When working with notebooks which are longer than one viewport, or notebooks whose outputs can sometimes be large data frames, you might use Browse Mode to quickly navigate between headers, tables, etc, without losing the cell focus. Right now the rendering of notebook and its outputs are virtualized and also separated in two different frames are breaking this workflow as not all outputs are rendered, and outputs and inputs are in different frames.

I'm thinking about how to improve this and make it closer to the previous webview editor, some ideas are:

• Render all outputs in the notebook when the notebook is opened. So you can always read all outputs once the focus is moved into the output layer. And you can always use keyboard shortcuts to move focus back into cell inputs.
• Set proper aria labels on output containers to provide accurate info about the cells, for example, on an output container, set its aria label to "cell 4, first output".

Let me know if they make sense, thanks in advance!

[zersiax]

@rebornix I think perhaps @NiSc91 might have some more specific feedback on this, but I would argue that the best thing would be to do a combination of both. e.g., make all outputs navigable at all times, and either make their headings indicate what they are outputting, or wrap them in a named region indicating what cell the output is from. @mohammad-suliman does that sound ok to you, as well?

[RichCaloggero]

I'm totally new to vsCode, so forgive me.

I'm using NVDA and tried to create a blank notebook with the "create blank notebook" command. I seem to land on cell 0, I can press enter to get an editor where I can type code, but cannot find any way of seeing output.

I did manage to find a "run cell" button somewhere, but when I click it via enter key, I still cannot find the output. I'm typing really simple python code like "print 'this is a test'".

• I think there needs to be clear navigation keys / commands to focus the cell list, and once on a cell, focus it's output area. Focusing it's input area seems to be attached to the enter key.
• I find that there is always lots of extraneous stuff in workspace at any given time; would be nice to have a simple way to remove it all and focus only on the view at hand, opening other things as needed
• arrow keys in the cell input editor should never move across cell boundaries; you should press escape to move back to cell list and either focus another existing cell, or add one if needed
• what is the difference between native notebook support and jupyter notebook support? does native mode support jupyter by default, or is this built on top of native support?
  Sorry for all the newbie questions.

[RichCaloggero]

I need to also say that vsCode is a huge step forward for screen reader users. It seems rough still in terms of accessibility, but this is the only environment I've yet seen that has the potential for acting as a real screen reader accessible cross platform IDE.

Congrats to all involved!

[zersiax]

@RichCaloggero at present, you would drop out of focus mode (nvda+space) in order to see the cell's output.
Also I'm not sure what you mean by "extraneous stuff" , not sure if that is part of this particular issue though? :)

[isidorn]

@RichCaloggero thanks a lot for the feedback. For non notebook feedback and general vscode accessibility issues feel free to open new issue here on GitHub and ping me @isidorn on the issue and then we can continue the discussion in those issues. Thanks

@rebornix let me know if you would like me to give this another pass, maybe I might have useful feedback as well :)

[RichCaloggero]

Perhaps a setting to toggle moving to next cell with arrows during editing?

[RichCaloggero]

@rebornix wrote:

Right now the rendering of notebook and its outputs are virtualized and also separated in two different frames are breaking this workflow as not all outputs are rendered, and outputs and inputs are in different frames.

Maybe add a setting which changes rendering to the previous style where all notebook content is always rendered. Scrolling issues are very confusing for screen reader users. Having content depend on scrolling breaks separation of semantics and presentation.

[RichCaloggero]

If I make an error in python code, how can I find the error output? When I render the notebook, there is a live region which is triggered telling me there are errors, but when I examine the problems area, it says there are 0 problems in the workspace.

If I switch to the cell editor, and press alt+f8 (move to next problem / error / warning), nothing happens. If I switch to the cell list and press alt+f8, nothing happens.

[NiSc91]

@rebornix I think perhaps @NiSc91 might have some more specific feedback on this, but I would argue that the best thing would be to do a combination of both. e.g., make all outputs navigable at all times, and either make their headings indicate what they are outputting, or wrap them in a named region indicating what cell the output is from. @mohammad-suliman does that sound ok to you, as well?

I agree with @zersiax that a combination of both would be preferable; i.e. render all the output and make some indication of which cell the output is from. Just to clarify though, I still like the layout with the output being rendered underneath each cell so that you can easily press down-arrow in browse mode and get the output of the focused cell, but I'm not sure this was up for being changed in the first place with those new improvements :)

[rebornix]

Thanks for the suggestions!

render all the output and make some indication of which cell the output is from

we can experiment this first. I don't have enough capacity to make it happen this week so will explore some time next week.

Perhaps a setting to toggle moving to next cell with arrows during editing?

From tomorrow's Insiders you can set notebook.navigation.allowNavigateToSurroundingCells to false to disable this feature. The reason we added this setting other than changing the behavior is allowing easy cell navigation through arrow keys still have its value.

[venkateshpotluri]

Thank you @rebornix for continuing this conversation, taking feedback and improving the notebook experience. I just gave the notebook experience a try by creating a Jupyter notebook and playing around with it in the insider build. I am adding my thoughts to earlier comments, and am reporting a few new observations.

@rebornix I think perhaps @NiSc91 might have some more specific feedback on this, but I would argue that the best thing would be to do a combination of both. e.g., make all outputs navigable at all times, and either make their headings indicate what they are outputting, or wrap them in a named region indicating what cell the output is from. @mohammad-suliman does that sound ok to you, as well?

I agree with @zersiax that a combination of both would be preferable; i.e. render all the output and make some indication of which cell the output is from.

Just want to chime in and say that this behavior would be quite useful. However, the indications for output of a particular cell should be designed carefully to not interfere with things that notebook authors (e.g. for tutorials) might add. I am thinking, a heading with the text "sell" cell number followed by the text "output" may make this clear. please feel free to suggest if this would be confusing. Also, scrolling-based access to information for screen reader users may be confusing so it would be great to have access to all the output at all times.

Just to clarify though, I still like the layout with the output being rendered underneath each cell so that you can easily press down-arrow in browse mode and get the output of the focused cell, but I'm not sure this was up for being changed in the first place with those new improvements :)

I echo this preference. Not only would it make screen reader navigation easier, but from what I understand will also keep the order of code and output consistent to the visual display.

I noticed that NVDA announces "Editor content;Press Alt+F1 for Accessibility Options. edit has auto complete multi line blank" when I navigate to a different cell using the down arrow key. This feels very verbose and it would be great if this announcement could be shorter and be more informative. I assume that this may be coming from the Monaco editor (I hear the same verbose announcement when using Google Colab) so am not sure if this can be changed easily. Further, I do not hear cell numbers when I am moving across cells. Is there a setting I need to change to hear this information? I am excited to see notebooks become more accessible.

[travisroth]

Hi, I'm also new to VSCode's Notebook support. I want to put in my observations in case they're useful nonetheless. I am using VSCode 1.59 today.
In the discussion the command notebook.cell.focusInOutput is mentioned. Using JAWS this command has no useful effect. JAWS does detect a frame is attempted to be focused but since JAWS is in application mode it cannot browse until virtual cursor is enabled. Turning on virtual cursor (JAWSKey+z) resets JAWS's focus and if focus was ever on the output or not cannot be determined. While perhaps not perfect I would suggest a workaround of sorts may be to put the actively focused cell's output in a region that is labelled. JAWS has commands to move to regions in it's browse modes, so this may be an opportunity to take advantage of the ability until another idea is figured out as I think managing focus between application mode and browse modes of screen readers is going to be quite difficult. (I think NVDA has this same functionality but I cannot recall if the quicknav key is defined by default.)

Output formatting: reading previous comments it appears at one time output was more formatted such as including table markup when Pandas prints a dataframe. This does not seem to be the case now. I am only getting a plain text view of a table when I do find it in the web view version of the output with JAWS's virtual cursor. As I'm sure you can imagine this is hard to read and use if the table gets more than a handful of columns. In the Settings I notice a setting for controlling how many lines of output is displayed but not one for getting a rich HTML view. Perhaps I'm missing it or perhaps this is just an accessibility issue.

Keyboard navigation between cells: thanks for notebook.navigation.allowNavigateToSurroundingCells setting. I understand the keyboard efficiency of moving between cells with only arrows, but right now switching cells is easily lost with a screen reader and you can easily end up in another cell without realizing it.

[zersiax]

Curious what is happening with this one? It's been quiet for a rather long time?

[webczat]

ping
was wondering how to use the interactive window efficiently. I mean what I am typing in it is to be put into a code cell and executed? but I don't have any efficient way to go to latest output?

[rohanshardha]

Not sure if this thread is active but VS Code has made a lot of improvements to make the output of the notebooks read aloud with screen readers starting with version 1.81. Now you can use control+down arrow to read the output of a cell when the cell is focused( not in edit mode). However, they broke it in v 1.82. Does anyone else had an opportunity to use jupyter notebooks in version 1.82 of the vs code and try reading the output? I am not able to read the output by pressing control+downarrow. It doesn't read anymore. I did verify that the editor is set to accessibility setting. Using JAWS 2023.23-07 with VS Code 1.81 on Windows.

@rebornix

[isidorn]

@rohanshardha thanks for the feedback. @meganrogge @rebornix we should double check the state in latest Insiders.

[meganrogge]

@amunger on insider's, this is working for me on mac, can you pls try?

[amunger]

that command seems to no longer visually indicate that the output is focused, but NVDA still seems to read the content, and opening the accessibility view for the output with alt+F2 works as expected for me. I'll add an issue for the lack of visual focus

[amunger]

closing this issue as a lot of changes/improvements have been made making a lot of the discussion here out of date. Please open new issues if there is anything left incomplete from this thread.