Add accessibility/screen reader integration to the base UI components

[Billaferd]

Describe the project you are working on

Point and Click adventure game that makes heavy use of Text and other GUI Elements.

Describe the problem or limitation you are having in your project

The game is completely unusable to standard Accessibility tools

Describe the feature/enhancement and how it helps to overcome the problem or limitation

By integrating the base UI controls, many games can become more accessible with little to no thought from the individual developers. By integrating the common accessibility frameworks of each platform Godot games can be played by more people by default.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

I would imagine the easiest way would be to create an accessibility server that could be used to hide implementation specifics and also create a general API internally for components and scenes. It could also be done by creating a standard library that can be extended for each platform/accessibility tool. Tools to consider would be the Accessibility Toolkit (ATK) on Linux and the UI Automation Framework on Windows.

If this enhancement will not be used often, can it be worked around with a few lines of script?

A new server could be implemented and all state transitions are telegraphed by signals, but that seems like it would be quite slow.

Is there a reason why this should be core and not an add-on in the asset library?

The speed considerations and usefulness of an inclusive design would benefit many with very little overhead and would not require individual developers to become accessibility experts or even have to remember to add the add-on.

[Zireael07]

@ndarilek has been working on accessibility (making Godot screen reader friendly) and has made an add-on to that effect.

What other accessibility tools do you mean?

[ndarilek]

To be pedantic, and only because this is an important distinction for anyone doing serious accessibility work, I'm not making Godot screen reader friendly because that would be impossible. Screen reader friendliness would involve implementing bindings to accessibility APIs on every platform, which would be impossible for, say, the web. I'm implementing a screen reader in GDScript, which covers a very narrow use case. A truly robust solution would probably involve folding something like Flutter's Semantics class (https://api.flutter.dev/flutter/widgets/Semantics-class.html) into Control, then using that to build out other accessibility services like mine, or working with me to integrate that support with godot-accessibility. Something like a high contrast mode or large text might be useful, but I can't do that so have been focusing on the work I can do.

[Calinou]

Related to #983. I'll leave this proposal open as it's better to have small, focused proposals rather than cramming many ideas in a single proposal.

[Billaferd]

So the main difference between what this proposal is asking for and what @ndarilek is doing is that the godot-accessibility add-on is focusing on adding in accessibility features that are rolled into the game and are important for platforms that don't have accessibility features natively. It also adds accessibility features into the editor itself, which is something that is also extremely important.

What I am proposing is to expose the built-in components like "BaseButton" to the platform's native accessibility controls. The reason is that there are far more accessibility issues than just screen readers, although they are a large part. There are also Braille Displays, TTY Displays, Magnifiers, Eye Trackers, and even Sip N' Puff controls. Hooking and exposing to native accessibility libraries gives two distinct advantages:

1. Accessible by all assistive hardware/technologies in a standard way (by platform).
2. Could be used for UI Automation in most, if not all cases. (Automated UI/Game Testing like Puppeteer).

This is a very large undertaking but would be worthwhile for both reasons above.

Some Reading:

Microsoft UI Automation: https://docs.microsoft.com/en-us/windows/win32/winauto/entry-uiauto-win32
Gnome Accessibility Toolkit: https://en.wikipedia.org/wiki/Accessibility_Toolkit
Android Accessibility: https://developer.android.com/reference/android/view/accessibility/package-summary

Other Accessibility Considerations: https://accessible.games/accessible-player-experiences/

[Billaferd]

Screen reader friendliness would involve implementing bindings to accessibility APIs on every platform, which would be impossible for, say, the web.

Although there are no built-in ways to expose this information to the web browser we can do some tricks like adding hidden elements dynamically as needed to the page to represent the inputs, text, and other things that we would like to make accessible.

Not an ideal solution, but definitely something achievable in time.

[Calinou]

@harrmitch Please don't bump issues without contributing significant new information. Use the 👍 reaction button on the first post instead.