Basic accessibility support for the terminal view #344
Conversation
services can get the text currently being shown.
|
Thanks a lot! Is it necessary to call |
|
Hello, |
|
Changed, tested and working fine. |
|
@fornwall Please can you take a look one more time? |
|
Did this ever get added to Termux? I installed it today and wasn't able to read the terminal's output. Is there something I need to do to get it working with Talkback? |
|
This PR is still open thus not merged to the code base. |
|
I'm having lots of issues with the jre and android-studio playing nicely together. Is there an apk somewhere that I can just install until this is merged? |
|
@stormdragon2976 you can try this one: https://grimler.se/apps/termux-app-access.apk I haven't tested it at all so it's great if you are willing to! |
|
Thanks so much for this. It is very helpful. The accessibility is very basic indeed, but it does work. Definitely a good start. |
|
@pvagner I'm really sorry for forgetting about this one, and I would really like to see it included! Could you (or anyone else interested) have a look at enabling this only if accessibility services are enabled (or something similar), as this may slow down the app needlessly if not? Perhaps with something like this in the TerminalView constructor private boolean mAccessibilityEnabled;
[...]
public TerminalView(Context context, AttributeSet attributes) {
[...]
AccessibilityManager am = (AccessibilityManager) context.getSystemService(ACCESSIBILITY_SERVICE);
mAccessibilityEnabled = am.isEnabled();and then guarding the call with |
|
I'll be able to use and integrate this code snipped in a week or so. I'm sure something like this is doable. |
|
Thanks a lot! It's beyond the scope of this immediate change, but could you give some pointers about possible next steps to improve accessibility support? |
enabled when starting the view
|
@fornwall I apologize for another delay. I have tested your suggestion on my devices and it's working fine over here. Hopefully I've got it right. I can imagine a few more improvements to the TerminalView accessibility support however I am not sure I am able to implement them.
Perhaps there is more however these are things I can imagine being either critical for the comfortable usage of the app or easy or moderate to implement. |
|
Thanks a lot, both for this change and the suggestions for further improvements! |
|
any idea about what would be required to make new text be read automatically in the terminal? I guess using an accessibility live region for the whole terminal view won't work, as the whole terminal would be read out, not only the new text. A hidden accessibility region containing only the new text would work, but sounds like an ugly hack. Other ideas? Maybe something linked to the accessibility node info @pvagner talked about? |
**tl;dr** it's a lot easier to read the output of the last command and to get to the copy/paste/more... actions It's a bit yiffy to navigate the wall of text in the terminal view. As things stand right now with content description, accessibility services see the `TerminalView` as an opaque blob with a wall-of-text description. Since it's a description, there is nothing locking you in the control when using reading controls (like you would be in a `TextView`). Additionally, if I read the documentation of `setContentDescription` right, it is really meant to offer a description of non-textual elements, or of irritatingly opaque content; *not* to be the content itself. This is probably irrelevant, as the goal was to make it easier to get to the output of the last command. It is also nigh impossible to notice the popup menu (copy/paste/more...) if you don't... *see* it. It ends up at the end of the control stack and it is not announced (if I understand correctly, you have to issue an accessibility focus event, but the accessibility of the selection handle thing is a whole different problem to the one I'm trying to solve here). The goal was to make the `TerminalView` behave more like a text view: - have reading controls (by line, word or line) available regardless of what the system guesses, within the terminal view itself - don't accidently leave the terminal view when using reading controls too much (e.g. by word) - ability to easily jump to end, by swiping up/back with reading controls after activating the view (since it no longer "escapes") - note, reading controls are different to basic navigation; basic navigation let's you flick between the view, soft keys (CTRL, ALT etc), copy/paste/menu popup, your soft keyboard, etc; reading controls allow you to navigate a wall of text within a control, i.e. the terminal view - added accessibility actions to open the More... menu and copy and paste actions; note, these are apparent if TalkBack is running Actual changes, on `TerminalView`: - explicitly mark the view as important to accessibility, because it does things which are important to accessibility, and it is the main focus of the whole app. The documentation of that method asks to set the flag to `YES` if a view does any of the things below; - call `sendAccessibilityEvent(TYPE_VIEW_TEXT_CHANGED)` when text changes and implement `onPopulateAccessibilityEvent` to let accessibility services know the text has changed (and what it changed to) - implement `onInitializeAccessibilityNodeInfo` which is called by accessibility services to detect what is on screen; this gives us a chance to explain that this `TerminalView` is a multiline text container and not some random eye candy that may be skipped over - no longer set content description, since we now set actual content by way of `AccessibilityNodeInfo` and `AccessibilityEvent`; replaced with all of the above - added custom accessibility actions for copy (screen), paste, and open termux menu (the *More...* menu). Accessibility Actions are something an Accessibility Service may query and present in a way that makes sense to the user. In the case of *TalkBack*, it has an *Actions* "reading mode" - added new strings to `strings.xml` and added 3 new IDs to `ids.xml` to support the custom accessibility actions. - refactor method out from `onTouchEvent`, branch `BUTTON_TERTIARY`, to use it for the accessibility paste action The *Copy* action could be better. I had 6 goes at integrating with the custom selection widget handle thing, I failed. So the *Copy*-accessibility action will copy the screen as a compromise. Something ought to be done about the side panel, since you rather have to know it's there. But since the More... menu is now easily discoverable, someone with TalkBack might stumble onto the Help panel and read that there's a secret side panel on the left. That wouldn't be much different to the way I myself discovered that sessions side panel on the left :-) It would be worth adding landmarks for the terminalview, button bar and side panel, but that's a different PR, since it probably needs to be done in the Activity, not here. There's also something iffy about the fact that `AccessibilityManager.isEnabled()` is only checked at start up. Maybe it should be documented somewhere that accessibility is checked only at startup for performance reasons. For people who turn TalkBack on after launching the app, it might appear broken, even though it really isn't. I didn't want to touch that behaviour since it was something requested in the review of !344[1]. I am merely stating my dissent. [1]: termux#344 (comment) Someone also mused about having the terminal speak out text as it comes in, but that's also a different PR, since it rather requires interplay between the TerminalEmulator code and TerminalView (providing accessibility), plus setting up a virtual view that is live; and I assume people might want to turn that off / control verbosity, which is a lot of work, so... not in this PR. It is theoretically possible to do partial text updates in `onPopulateAccessibilityEvent`, (instead of blasting away the whole screen each time you type a character) but I'm not convinced it actually works the way I hope, and it requires a lot of work to get there (see previous paragraph). So... not in this PR. But this PR makes interacting with the TerminalView itself much nicer if you can't see what you're doing, and there's no reason to require people to see what they're doing in this case.
Terminal view as implemented in the termux app is a view resembling terminal windows known from computers. It's designed to be efficient however for me and other people who are using accessibility services such as Google Talkback to access their android devices it's not very usable. By adding contentDescription property to the view dynamically as it repaints, we can get at least very basic accessibility support.
When using termux on Android with Talkback running it is possible to review currently visible text on the screen by characters, by words and by lines.
It is possible to use two finger scrolling (android built-in feature) to scroll the view and access other parts of the text.
I'm using this for a few months already and I have got some friends interested in this functionality so I'm submitting this as a pull request.
I know it's very minimalistic, but even in its current limited form it's a life changing feature for tech savvy blind people so hopefully you can accept it and / or point out how I might be able to enhance it so everyone can benefit.