Improve accessibility features for keyboard navigation, UI scaling, TTS, voice recognition, colorblindness and subtitles

[vnen]

Describe the project you are working on:
The Godot Engine.

Describe the problem or limitation you are having in your project:
Accessibility features might be required by law in some game projects. It's also nice to allow more people to play the games. Some accessibility features are not obvious or not possible to do only with the base engine and this needs to be improved.

For instance, the CVAA legislation covers anything with communication. So if your game features text or voice chat, you have to comply with the rules (assuming you are selling the game in the US, which most people are). Here's a blog post that covers how it is applicable to games: https://www.gamasutra.com/blogs/IanHamilton/20190123/334910/Demystifying_CVAA.php

Describe the feature / enhancement and how it helps to overcome the problem or limitation:
Improving documentation for existing features and implement missing features needed for providing greater accessibility in game projects. More details on what's needed are presented in the next section.

Describe how your proposal will work, with code, pseudocode, mockups, and/or diagrams:

There are a few important topics related to improving accessibility in general. I might even be missing something, as I'm not an expert in the area. If that's the case, let me know and I'll edit it in.

Input mapping

Games should provide a way for the user to change the buttons needed to perform actions. Godot's InputMap feature can be used for this, but I'm not sure how difficult it is to make the menu in-game. At least we need a documentation page to describe the process with examples.

Keyboard navigation

Godot can use keyboard to navigate between controls but there are still some issues. For instance, not all controls can receive focus which means they aren't target for keyboard navigation. Also, if you don't set the neighbors properly, navigation can reach a control.

What could be done is improve default control focus and use either the tree or control position to determine neighbors for keyboard navigation.

This is important because some people can't operate a mouse and relies completely on keyboard for getting around.

Touch screen navigation

Mobile OSes provide gestures for blind users to navigate around the UI. Native apps follow this by default without effort when using the native SDKs. But when making a game, devs are using Godot's UI system which don't have the same capabilities AFAIK.

There should be away, preferably baked into the input system, to use the OS gestures to navigate around the game UI.

Font scaling

There must be a way to easily increase the size of all the text displayed. I do believe that if you are properly using themes you can do this by changing font only in a handful of places, but I haven't tested this.

At the very least, we need to improve documentation to show how you can use the available systems to make a global font size slider.

UI scaling

In a similar fashion to the previous point, the whole UI should be scalable. Using the scale property of controls is meant mostly for animations, and using that makes it difficult to correctly position everything.

This also includes making controls bigger so they can be an easy mouse target and also visible for people with low sight.

We should have a way to easily make everything bigger while maintaining defined positioning margins. Maybe there's no easy way to do this, but we need at least to try and make a documentation page about control scaling with correct positioning.

Here's an example of this applied to a game: https://twitter.com/NateElCabro/status/1235290044866473984

Text-to-speech and speech-to-text

This is one of the most problematic topics. Adding a library to manage this in the base engine might be too much, especially since many games don't really need this.

The better possibility is to allow OS-level applications to properly read the screen. People who need a screen reader already have one installed and configured, so letting them use that is a good thing.

Some systems might not have such OS-level applications, and need a custom solution (e.g. consoles) but in this case I believe it's okay to leave the burden to a third-party library. At least we need to make sure that GDNative is capable of binding such libraries, so plugins can be provided without the need of recompiling the engine.

We probably also need some description field for controls that don't have associated text, which would be used by screen readers. There's a long discussion (initiated by a blind developer) about this here: godotengine/godot#14011

Color blind filters

This is more a "nice-to-have" than anything. The idea is to provide filters that simulate common types of color blindness, so developers can adapt their games to make elements distinct in all cases.

This is covered by #473 but I felt the need to include here. There's also a nice plugin that allows you to this already: https://github.com/paulloz/godot-colorblindness

Maybe we should integrate or endorse such plugin.

Subtitling

This is also a "nice-to-have" which is meant to facilitate the addition of subtitles to games. Ideally anything that is said should also have a subtitle, including audio-description (especially when relevant to gameplay).

This one might be more suitable for a plugin, but I added here for completeness. For reference, see this subtitling plugin for the Unreal Engine: https://www.unrealengine.com/marketplace/en-US/product/yellow-subs-machine

If this enhancement will not be used often, can it be worked around with a few lines of script?:
As it stands now it cannot be easily added with a few scripts (and should be used often).

Is there a reason why this should be core and not an add-on in the asset library?:
Many points here are related on how the engine behave which should be standard, not modified by a plugin. Some of the points might be okay as plugins (we do need to make sure such plugins can be made).

[TheDuriel]

Personal thoughts and viewpoints on the matter. This should probably be split into separate proposals as well.

Remapping:
https://github.com/godotengine/godot-demo-projects/tree/3.2-57baf0a/gui/input_mapping

Focus:
All controls should be able to receive focus right now. Some special controls will hand focus to relevant subcontrols as well. Maybe Tree has some issues relinquishing focus?

Fonts:
Load the dynamic font resource. Change its size. Call https://docs.godotengine.org/en/3.0/classes/class_font.html#class-font-update-changes on the resource. A tutorial could be added to the Docs, but it is truly that simple.

UIScape:
It would be nice to have control over DPI scale of the UI. Viewports/CanvasLayers can be used for this, kinda.

TtS StT:
Imho is a gdnative plugin concern rather than an engine concern.

Color blind filters:
Screen space shaders to do this can already be implemented. And it will be easier in vulkan.

Subtitles:
A fairly straightforward ui component. I don't see why the engine should concern itself with something that can be implemented with 6~ lines of code.

[nobuyukinyuu]

Input Mapping

I've worked on input mapping for my project as it is an unusual control scheme that would benefit from users being able to configure input in a way that's comfortable for them. Remapping binds could benefit from a few convenience functions, such as ones to map devices in order of first input (a Microsoft requirement, dunno if it applies to others), a more unified way to detect input axes (usually you have to think of all the different ways this can happen and poll for them manually, see further below), and a built in table/database of easy names for all inputs that's accessible to the coder and not just the editor (edit: can be accessed from InputEvent.as_text(), but doesn't seem to account for naming differences between controllers).

Supporting multiple mice as well as multiple joysticks would also be helpful as some devices map to these.

Allow more coder access to Godot's controller layout guesser (if one exists), seems to use a database like SDL.

Being able to save and load multiple input map layouts. Layouts could be specified in a list or a dictionary (a collection of layouts), allowing some keys to remain universal (UI) while allowing quick rebinding presets for multiple actions to account for different users sharing a game or different classes of users to select from coder-specified presets. (In my game specifically, this feature could've helped a lot making it easier to account for handedness.)

Unifying the reads of joy axis, mouse deltas, and other linear sensors (accelerometers, gyro, and other ones currently exposed to Godot) under some function or in the current Action system (eg: Input.is_axis_pressed, Input.axis_strength) would make it a lot easier for alternative inputs to be dropped into existing code without creating special cases for them all. In addition, being able to remap these disparate sensors individually so that they read transparently in the range your game expects would be helpful. For example, pressure sensitivity might be better read in logarithmic scale and mouse deltas may need their scale adjusted based on the hardware device being mapped to mouse. The deadzone can be changed for some input values but floor and ceiling remapping can't.

Font Scaling

Easier font scaling can be achieved in theory by setting the dynamic font's resource as local to scene, at the expense of losing some theming ability. Being able to specify default font size separate from instance size would allow bitmap fonts to scale just like a dynamic font, but also step around the "local to scene" issue. Having kerning, word spacing, and tracking offsets available for all font types would go a long way to helping improve readability and make it easier to iterate over variations on a font.

Color blind filters

Screen space shaders require backbuffer copies if they're to work with existing games. Multipass shaders currently don't work for canvas items, making it frustrating to implement a simple, custom solution without having to get involved with screen space or multiple viewports.

[vnen]

Supporting multiple mice [...]

This point in particular I don't think it's relevant, especially on the topic of accessibility (is there any person who needs to use two mice?). AFAIK, the API from the OSes will only provide one mouse pointer. You can technically do it by polling the devices directly but I think it's a bit too much for a niche usage, as I don't think it fits the accessibility topic.

[vnen]

Subtitles:
A fairly straightforward ui component. I don't see why the engine should concern itself with something that can be implemented with 6~ lines of code.

Honestly, I don't know how you do that with only 6 lines of code. You need to parse a SRT file or similar to follow voice-over, you have to trigger the text in audio events (like an explosion), and I might be missing something else.

I'm not saying that it's the most difficult thing to do and I'm saying that this (or any other topic for that matter) should be added to the engine. But they should be available in some fashion.

This proposal is to make sure that all accessibility topics are either covered by the engine or facilitated in form of a plugin (that is, such plugin should be possible to make). Ideally we should even maintain those officially.

[Zireael07]

@vnen: The 6 lines of code comment probably referred to the most naive solution where there's a text file for every audio clip, no SRT, no voice-over, just showing the text (which is fairly easy to do given Godot's Control nodes and signals)

EDIT: And mad props for tackling the issue at all, too many games completely forget about people who need subtitles. Being hearing impaired, I tend to prefer old-school games like Baldur's Gate series that just showed all the text on screen, no subtitles needed :P to new-fangled 3D RPGs that don't have subtitles at all!

[nobuyukinyuu]

Supporting multiple mice [...]

This point in particular I don't think it's relevant, especially on the topic of accessibility (is there any person who needs to use two mice?). AFAIK, the API from the OSes will only provide one mouse pointer. You can technically do it by polling the devices directly but I think it's a bit too much for a niche usage, as I don't think it fits the accessibility topic.

Maybe. I built my own device to map to mouse but i don't work in the accessibility field so I can't say how many devices are out there which map to mouse specifically. Usually you can get away with one cursor to do whatever you want if you capture it, so it's probably fine either way.

[StraToN]

May I also suggest W3C's guidleines in the matter of accessibility :

• https://www.w3.org/WAI/standards-guidelines/uaag/
• https://www.w3.org/WAI/standards-guidelines/atag/
• https://www.w3.org/WAI/standards-guidelines/wcag/

ETSI:
https://www.etsi.org/deliver/etsi_en/301500_301599/301549/03.01.01_20/en_301549v030101a.pdf

[kujiu]

Hi here, I'm concerned with accessibility and I've some issues too. I speak about myself (with low vision) here, but I can help you to understand some problems if needed. StraToN posts three links of the W3C, and it's the three guidelines for web. I think all the stuff can be adpated for Godot, and there are three standards for three differents axes.

• One is for the content (video, text, sound, etc.).
• One is for the client software (the browser for web).
• And one is for the authoring tool.

Even if I'm legally blind, I use Godot Engine and its IDE. But, accessibility is not here so I do the maximum with a simple text editor. There are a lot of problems with zoom,
size of lines, etc. So, accessibility of Godot will be a very complex and hard process. I recommand you to read all the work of W3C and the EN301549 (European Union standard about ICT accessibility). These are great, intense and really big documentations. If you want to improve accessibility, I think you need to review it and completing tasks by levels. As you can see, the W3C defines three levels : A, AA and AAA. A is for a non-blocking accessibility part, AA for a good accessibility but not perfect and AAA is the maximum. This nomenclature will help you to find your priorities, because you have a lot of work to do here. It will not be easy, but you'll learn a lot.

I don't have the time to help you, because I have other projects. But I can test when you need, and I'm an user of Godot. I'll keep an eye on this issue, so if you have any question about what is accessibility, I'll try to answer.

[vnen]

Note that this is mostly about providing accessibility for games. But I do believe that it's a big step into making the Godot editor itself more accessible itself.

[Calinou]

We should have a way to easily make everything bigger while maintaining defined positioning margins. Maybe there's no easy way to do this, but we need at least to try and make a documentation page about control scaling with correct positioning.

The 2D stretch mode should handle this well enough out of the box. Still, godotengine/godot#21446 will help make this even better by letting the user increase or decrease the UI scale relative to the automatically determined scale. That PR also provides scaling abilities when using the disabled stretch mode, which is typically the stretch mode you want for non-game applications.

As for custom HUD margins, this is already possible using MarginContainers (once you expose a way for players to change those margins).

[ndarilek]

Hey folks,

FOr a while now, I've been working on a screen reader written in GDScript and a text-to-speech addon. Known to support Linux, Windows, Android, and the web. May also work on UWP as well, though I'd have to jump through a few more hoops to test that. Could probably be ported to MacOS/iOS easily enough if someone could port tts-rs to those platforms. It took a day or two to knock out a WinRT port, so I don't think it'd be hard. I just don't have the hardware.

I'm maybe a month or two out from releasing a game using it, so I'd say it's ready for others to bang on. Note that I myself am blind, so I'm a bit limited on what I can do. In particular, I'm happy to accommodate low-vision users somehow, I just have no way of, well, doing anything meaningful. :) But this at least helps with screen reader and keyboard access.

I'd very much appreciate help with this. I'm trying to figure out how to use the editor by writing the screen reader, which works to a point, but I've definitely hit some barriers, and it'd be useful to pop onto a Zoom call and screenshare an editor session with someone who knows Godot well so I can pepper you with questions. :) Also, if someone wants to see an actual engine bug that might be easy to fix and would be a massive QOL improvement for me and others, check out godotengine/godot#36291. Essentially, keyboard focus doesn't seem to work in some editor trees. I have no idea whether this is specific to the editor, happens in all trees, etc. but it means I'm hand-editing TSCN files in cases my plugins could handle, if only focus worked. I'm happy to try fixing it myself if someone with engine experience could help, but I've tried some things on my own and can't seem to make a difference.

Also would be nice to see my plugins used in other games, both so they're improved, and so I myself have more games to play. :) If you have a game that might be made screen-reader-accessible, reach out and let's collaborate.

[Zireael07]

@ndarilek: You are doing awesome work, man!

[bruvzg]

Could probably be ported to MacOS/iOS easily

Here's macOS and iOS code (it's almost identical) from my old TTS plugin, it's in Objective-C, and I'm not sure how to interact with Obj-C stuff from rust, but it shouldn't be too hard to port.

https://github.com/bruvzg/godot_tts/blob/godot_cpp/src/tts_nsspeech.mm
https://github.com/bruvzg/godot_tts/blob/godot_cpp/src/tts_avspeech.mm

[ndarilek]

Thanks. I'll probably leave the porting effort to someone else for now, though if I do eventually get a Mac then I'll circle back here and check that out. Does that work on iOS too?

There's a Unity accessibility plugin that supports iOS, which I definitely want to support and would probably prioritize over macOS at this point.

[bruvzg]

AVSpeech is for iOS and macOS 10.14+, NSSpeech for older macOS versions.

[ndarilek]

I'm sorry, you *did* say iOS right there in the comment. :/ I blame lack of coffee. Fixing that now...

[Gamemap]

I found this Godot Speech-To-Text module which may fit this topic and may be useful for you.
Unfortunately I don't know any programming language (like C++) to help here.

https://github.com/menip/godot_speech_to_text

[DaGamingWolf]

i'm a developer who is working to provide accessibility to the game i'm working on for the blind, so all of this is very welcome. It is sad to see such features get put off. it's harder to add them in later than earlier, but unfortunately people don't think about it intuitively until it starts becoming an issue. i just want to contribute to the record that yes, there are people who need and want these features using Godot.

[blakeearth]

I'm still watching this issue, but for me, this is the reason I stopped developing with Godot in favor of Kontra + React. I need to be able to support screen readers and accessibility features more broadly.

[btzr-io]

Additionally it will be useful to have something like the accessibility inspector of modern web browsers, either as a core feature or external plugin:

https://firefox-source-docs.mozilla.org/devtools-user/accessibility_inspector/index.html
https://developer.chrome.com/docs/devtools/accessibility/reference/#overview