JupyterLab user testing script and results #10
Conversation
results/user-testing-results.md
Outdated
| - What are the full range of notebook cell output types and what feedback to disabled users have on each type? | ||
| - What is the real time collaboration experience in JupyterLab like for assistive tech users? This question requires JupyterLab collaboration modes to be more stable before it can be tested. | ||
|
|
||
| As mentioned in the Limits section, the sample we were able to test with for these sessions was not as comprehensive of varied disabilities and assistive tech as we would like. Recruitment for these sessions was done within existing JupyterLab community spaces and was mostly self-selected; future tests would likely benefit from a more selective sample and outreach to disability communities where appropriate. This should be done with careful consideration as the history of people who make products interrupting disability communities and asking for labor is a fraught one. |
There was a problem hiding this comment.
I wonder if moving this paragraph to the top and mentioning that certain features with huge accessibility implications (like RTC) were not tested. I think that would make it easier for the reader to understand why some of the things are in the list. Does that make sense?
There was a problem hiding this comment.
I can see an argument for it. Let me think on that one.
There was a problem hiding this comment.
Right now my brain outlines this section as so:
- section heading: For future tests
- list of questions to explore further
- paragraph about the need to improve the selection sample
- paragraph about the need to use other user research methodologies
Looking at it this way, I take back my suggestion about moving the paragraph in the third bullet point (selection sample) to the top of the section.
However, because there are some things in the list that the preceding parts of the document did not touch upon, or perhaps only vaguely touched upon, it might make sense to insert some more wording between the section heading and the list, so as not to interrupt the reader's flow.
Let me go through the list of questions, one by one, trying to give you a window into my inner reader voice, showing which questions felt like they were in conversation with previous parts of the document versus ones which felt disconnected:
- Screen readers. My inner voice: this seems in conversation with the "Limits" section, specifically the part about not having any screen reader participants
- Authoring. Me: Unclear. disconnected? but maybe in conversation with the "notebook" section, since it seems that much of the findings were about users consuming a notebook rather than producing one
- Navigating. Me: this is definitely in conversation with multiple parts of the document, including the navigation findings section, as well as the suggestions for improvement
- Cell output types and feedback. Me: vague. in conversation with... the part about cell status and the drastic experience difference between non-sighted versus sighted users?
- RTC. Me: disconnected.
Hope that helps!
There was a problem hiding this comment.
I appreciate the explanation. If you are looking for relation between this section and the rest of the document, I would agree it's mostly not there.
I'll add a suggestion with some changes to the intro to see if that clarifies. I have to add it in another comment though to capture the right lines to change. Thanks!
results/user-testing-results.md
Outdated
|
|
||
| #### Different options for feedback on cell status | ||
|
|
||
| **Current default JupyterLab feedback for notebook cell status is exclusively visual**; the cell prompt and favicon changes rely on users being able to see the icons, especially since these changes are not surfaced to the user unless their focus happens to already be on those icons when they change. This makes knowing the status of a cell difficult to impossible for users who cannot look at the screen in full. Quick to run code cells with a cell output—like the majority of cells run during these tests— tend to make this a trivial issue, it marks a drastic experience difference for sighted and non-sighted users. Fully sighted users can quickly verify the status of a cell in two different places, go about other work while waiting, and more easily verify when a cell has completed running. |
There was a problem hiding this comment.
Sorry. My question wasn't clear. I know what a favicon is generally. My confusion was that until just now (I went and tested), I had no idea that the JupyterLab favicon changes to an hourglass when a cell is running. I also just checked Colab and realized that its favicon grays out while a cell is running. Is it a common web app pattern to reflect app status in the favicon? I guess I've seen this pattern used in chat apps, but I guess I've rarely seen the favicon change anywhere else.
Instead of linking to the Favicon Wikipedia page, I would link to the JupyterLab docs, but I don't see this behavior documented there (except in the changelog), so maybe linking to one of the issues or PRs would be good—maybe this PR comment about the JupyterLab favicon that has pictures/videos.
Or maybe the kernel status reflected in the favicon is already common knowledge, and I'm just late to the party. Maybe a second opinion here? or just your best judgment :)
results/user-testing-results.md
Outdated
|
|
||
| #### Different options for feedback on cell status | ||
|
|
||
| **Current default JupyterLab feedback for notebook cell status is exclusively visual**; the cell prompt and favicon changes rely on users being able to see the icons, especially since these changes are not surfaced to the user unless their focus happens to already be on those icons when they change. This makes knowing the status of a cell difficult to impossible for users who cannot look at the screen in full. Quick to run code cells with a cell output—like the majority of cells run during these tests— tend to make this a trivial issue, it marks a drastic experience difference for sighted and non-sighted users. Fully sighted users can quickly verify the status of a cell in two different places, go about other work while waiting, and more easily verify when a cell has completed running. |
There was a problem hiding this comment.
Much clearer 😀
Just wondering, is "experience difference" a compound noun that's used commonly in user research? It slowed the sentence down for me, but I'm wondering if that's just because I'm not immersed in the common terms. My inclination would otherwise be to omit "experience": "it marks a drastic difference for..."
results/user-testing-results.md
Outdated
| - What are the full range of notebook cell output types and what feedback to disabled users have on each type? | ||
| - What is the real time collaboration experience in JupyterLab like for assistive tech users? This question requires JupyterLab collaboration modes to be more stable before it can be tested. | ||
|
|
||
| As mentioned in the Limits section, the sample we were able to test with for these sessions was not as comprehensive of varied disabilities and assistive tech as we would like. Recruitment for these sessions was done within existing JupyterLab community spaces and was mostly self-selected; future tests would likely benefit from a more selective sample and outreach to disability communities where appropriate. This should be done with careful consideration as the history of people who make products interrupting disability communities and asking for labor is a fraught one. |
There was a problem hiding this comment.
Right now my brain outlines this section as so:
- section heading: For future tests
- list of questions to explore further
- paragraph about the need to improve the selection sample
- paragraph about the need to use other user research methodologies
Looking at it this way, I take back my suggestion about moving the paragraph in the third bullet point (selection sample) to the top of the section.
However, because there are some things in the list that the preceding parts of the document did not touch upon, or perhaps only vaguely touched upon, it might make sense to insert some more wording between the section heading and the list, so as not to interrupt the reader's flow.
Let me go through the list of questions, one by one, trying to give you a window into my inner reader voice, showing which questions felt like they were in conversation with previous parts of the document versus ones which felt disconnected:
- Screen readers. My inner voice: this seems in conversation with the "Limits" section, specifically the part about not having any screen reader participants
- Authoring. Me: Unclear. disconnected? but maybe in conversation with the "notebook" section, since it seems that much of the findings were about users consuming a notebook rather than producing one
- Navigating. Me: this is definitely in conversation with multiple parts of the document, including the navigation findings section, as well as the suggestions for improvement
- Cell output types and feedback. Me: vague. in conversation with... the part about cell status and the drastic experience difference between non-sighted versus sighted users?
- RTC. Me: disconnected.
Hope that helps!
Co-authored-by: Isabela Presedo-Floyd <50221806+isabela-pf@users.noreply.github.com> Co-authored-by: gabalafou <gabriel@fouasnon.com>
There was a problem hiding this comment.
Thanks for this write-up @isabela-pf. It is a really well-structured and easy-to-follow resource.
As always I am impressed at your ability to keep your content clear while highlighting key points in a well-articulated way.
I merged some suggestions that @gabalafou made and that you'd 👍🏽
results/user-testing-results.md
Outdated
|
|
||
| #### Different options for feedback on cell status | ||
|
|
||
| **Current default JupyterLab feedback for notebook cell status is exclusively visual**; the cell prompt and favicon changes rely on users being able to see the icons, especially since these changes are not surfaced to the user unless their focus happens to already be on those icons when they change. This makes knowing the status of a cell difficult to impossible for users who cannot look at the screen in full. Quick to run code cells with a cell output—like the majority of cells run during these tests— tend to make this a trivial issue, it marks a drastic experience difference for sighted and non-sighted users. Fully sighted users can quickly verify the status of a cell in two different places, go about other work while waiting, and more easily verify when a cell has completed running. |
There was a problem hiding this comment.
ha! I have been using notebooks/lab for years and I never noticed the change to an hourglass 🫠 I suppose its placement is not an obvious place to search for "status" cues
I remember us discussing this a while back. I would be +1 on merging and keeping both the environment + testing scripts in a single repository. I have held up making a tag/release for the repo until we resolve this, but we can make a release anytime. If any of @gabalafou or @isabela-pf have particular hesitations about having all the testing resources together (env + scripts) let me know, and we can find a better place. But otherwise, I think having all these together would help with guidance on structuring these testing sessions. |
|
I see no issue with putting the results in a new |
|
Responding to the above comment about merging.
I do remember this discussion, and I had thought the decision was to, at minimum, create a release for this repo to clearly preserve the testing state as it was during these tests. And then merge post release so that we have a clearly denoted point for where we tested from. Maybe I'm being too stubborn about this, but I am interested in to obscuring what was used to host JupyterLab so that
I won't push on this again, but I hope having it written here clarifies why I wrote the initial comment. |
Co-authored-by: Tania Allard <taniar.allard@gmail.com> Co-authored-by: gabalafou <gabriel@fouasnon.com>
|
There is a release for the repository now, but that means that the |
Please do not merge this PR. It exists for easy and public review, but I do not want to interfere with the testing environment. I am open to where this might better live while still being related to this repository.
Finally, here are the full results from the JupyterLab accessibility user testing done with the testing environment in this repository. It's time for review.
This PR adds:
resultsdirectory to keep the files groupeduser-testing-results.md, the full length document discussing testing resultsuser-testing-script.md, an empty version of the testing script used throughout these sessionstesting-environment-info.txtwith the list of packages in the testing environment.Things that might be helpful for review include:
Please let me know if you have any further questions, and thanks for the review! 🌻 cc: @trallard @steff456 and @gabalafou since your names are on the results!