Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature Idea: Find/Extract Text Attributes for Accessibility #2161

Closed
carlos-zamora opened this issue Jul 30, 2019 · 8 comments · Fixed by #10336
Closed

Feature Idea: Find/Extract Text Attributes for Accessibility #2161

carlos-zamora opened this issue Jul 30, 2019 · 8 comments · Fixed by #10336
Assignees
Labels
Area-Accessibility Issues related to accessibility Issue-Feature Complex enough to require an in depth planning process and actual budgeted, scheduled work. Product-Conhost For issues in the Console codebase Product-Terminal The new Windows Terminal. Resolution-Fix-Committed Fix is checked in, but it might be 3-4 weeks until a release.

Comments

@carlos-zamora
Copy link
Member

Description of the new feature/enhancement

ITextRangeProviders can have support for a FindAttribute() function (link). They support a ton of different text attributes.

We definitely can support background/foreground color. This might be a good opportunity to enhance our accessibility as a terminal. But we need to do some research as to how automation clients (i.e.: narrator) make use of this.

This would also expand as we support more text attributes in the future.

Proposed technical implementation details (optional)

This would go in the shared code for UIATextRange and would require some changes to propagate to XamlUiaTextRange in Windows Terminal.

@carlos-zamora carlos-zamora added Issue-Feature Complex enough to require an in depth planning process and actual budgeted, scheduled work. Product-Conhost For issues in the Console codebase Area-Accessibility Issues related to accessibility Product-Terminal The new Windows Terminal. labels Jul 30, 2019
@ghost ghost added the Needs-Triage It's a new issue that the core contributor team needs to triage at the next triage meeting label Jul 30, 2019
@DHowett-MSFT DHowett-MSFT removed the Needs-Triage It's a new issue that the core contributor team needs to triage at the next triage meeting label Jul 30, 2019
@DHowett-MSFT DHowett-MSFT added this to the Terminal Backlog milestone Jul 30, 2019
@codeofdusk
Copy link
Contributor

Could this be prioritized for 21H1? NVDA definitely can get colours from UIA.

@DHowett-MSFT
Copy link
Contributor

DHowett-MSFT commented Apr 5, 2020

I'm averse to scoping additional things into 21H1 while the team is heads-down on Terminal v1. 😄 Sorry.

@codeofdusk
Copy link
Contributor

With the additional three months, could this issue be reconsidered for 21H1 (just exposing colours, not necessarily #3567)? Thanks!

@codeofdusk
Copy link
Contributor

The discussion on #7691 reminded me of this issue. I think it might be worth reconsidering as well to enable support for more text attributes besides links.

It also seems that the console supports bold/italic/underlined text, could these also be mapped onto UIA formatting attributes? (colours would definitely be a good start though!)

If I have spare cycles and want to look into this, where might I start? Where are colours/formatting attributes exposed? Is it just a matter of creating a mapping from conhost attributes to UIA ones?

@carlos-zamora
Copy link
Member Author

The discussion on #7691 reminded me of this issue. I think it might be worth reconsidering as well to enable support for more text attributes besides links.

It also seems that the console supports bold/italic/underlined text, could these also be mapped onto UIA formatting attributes? (colours would definitely be a good start though!)

If I have spare cycles and want to look into this, where might I start? Where are colours/formatting attributes exposed? Is it just a matter of creating a mapping from conhost attributes to UIA ones?

It seems like most of the work would be mapping the TextBuffer attributes to UIA ones.

I took a quick look at #7251 since that adds a new text attribute and uses it. Here's some useful files too:

Most of the work of finding and exposing TextAttributes already seems to be done in TextBuffer.

@carlos-zamora
Copy link
Member Author

Some text attributes that we can expose:

  • *UIA_FontNameAttributeId 40005: non-localized font name string
  • *UIA_FontSizeAttributeId 40006: point size of the font
  • UIA_FontWeightAttributeId 40007: thickness/boldness of the font
  • UIA_ForegroundColorAttributeId 40008: foreground color of text as COLORREF
  • *UIA_IsActiveAttributeId: indicates whether the control that contains the text range has the keyboard focus or not
  • UIA_IsItalicAttributeId 40014: indicates whether the text is italic or not
  • **UIA_IsReadOnlyAttributeId 40015: indicates whether the text is read-only or can be modified
  • UIA_LinkAttributeId 40035: represents a hyperlink
  • UIA_SelectionActiveEndAttributeId 40037: indicates if we're at the start or end (or neither) of the selected range
  • *UIA_TextFlowDirectionsAttributeId 4028: the direction of text flow
  • *UIA_UnderlineColorAttributeId 40029: the color of the underline text decoration as a COLORREF
  • UIA_UnderlineStyleAttributeId 40030: style of the underline text decoration

*not sure if these are worth adding
**is this even possible for us? Might just be a future consideration.

@codeofdusk
Copy link
Contributor

codeofdusk commented May 26, 2021

*UIA_FontNameAttributeId 40005: non-localized font name string
*UIA_FontSizeAttributeId 40006: point size of the font

Are these customizable by the user or can they otherwise differ? If so, please add them. If the terminal always uses the same font and size, no need.

*UIA_IsActiveAttributeId: indicates whether the control that contains the text range has the keyboard focus or not

Probably not super necessary as the console always has keyboard focus?

*UIA_TextFlowDirectionsAttributeId 4028: the direction of text flow

Probably good for i18n.

*UIA_UnderlineColorAttributeId 40029: the color of the underline text decoration as a COLORREF

Please add.

@ghost ghost added the In-PR This issue has a related PR label Jun 4, 2021
@ghost ghost closed this as completed in #10336 Jul 9, 2021
@ghost ghost added Resolution-Fix-Committed Fix is checked in, but it might be 3-4 weeks until a release. and removed In-PR This issue has a related PR labels Jul 9, 2021
ghost pushed a commit that referenced this issue Jul 9, 2021
## Summary of the Pull Request
This implements `GetAttributeValue` and `FindAttribute` for `UiaTextRangeBase` (the shared `ITextRangeProvider` for Conhost and Windows Terminal). This also updates `UiaTracing` to collect more useful information on these function calls. 

## References
#7000 - Epic
[Text Attribute Identifiers](https://docs.microsoft.com/en-us/windows/win32/winauto/uiauto-textattribute-ids)
[ITextRangeProvider::GetAttributeValue](https://docs.microsoft.com/en-us/windows/win32/api/uiautomationcore/nf-uiautomationcore-itextrangeprovider-getattributevalue)
[ITextRangeProvider::FindAttribute](https://docs.microsoft.com/en-us/windows/win32/api/uiautomationcore/nf-uiautomationcore-itextrangeprovider-findattribute)

## PR Checklist
* [X] Closes #2161 
* [X] Tests added/passed

## Detailed Description of the Pull Request / Additional comments
- `TextBuffer`:
   - Exposes a new `TextBufferCellIterator` that takes in an end position. This simplifies the logic drastically as we can now use this iterator to navigate through the text buffer. The iterator can also expose the position in the buffer.
- `UiaTextRangeBase`:
   - Shared logic & helper functions:
      - Most of the text attributes are stored as `TextAttribute`s in the text buffer. To extract them, we generate an attribute verification function via `_getAttrVerificationFn()`, then use that to verify if a given cell has the desired attribute.
      - A few attributes are special (i.e. font name, font size, and "is read only"), in that they are (1) acquired differently and (2) consistent across the entire text buffer. These are handled separate from the attribute verification function.
   - `GetAttributeValue`: Retrieve the attribute verification of the first cell in the range. Then, verify that the entire range has that attribute by iterating through the text range. If a cell does not have that attribute, return the "reserved mixed attribute value".
   - `FindAttribute`: Iterate through the text range and leverage the attribute verification function to find the first contiguous range with that attribute. Then, make the end exclusive and output a `UiaTextRangeBase`. This function must be able to perform a search backwards, so we abstract the "start" and "end" into `resultFirstAnchor` and `resultSecondAnchor`, then perform post processing to output a valid `UiaTextRangeBase`.
- `UiaTracing`:
   - `GetAttributeValue`: Log uia text range, desired attribute, resulting attribute metadata, and the type of the result.
   - `FindAttribute`: Log uia text range, desired attribute and attribute metadata, if we were searching backwards, the type of the result, and the resulting text range.
   - `AttributeType` is a nice way to understand/record if the result was either of the reserved UIA values, a normal result, or an error.
- `UiaTextRangeTests`:
   - `GetAttributeValue`:
      - verify that we know which attributes we support
      - test each of the known text attributes (expecting 100% code coverage for `_getAttrVerificationFn()`)
   - `FindAttribute`: 
      - test each of the known _special_ text attributes
      - test `IsItalic`. NOTE: I'm explicitly only testing one of the standard text attributes because the logic is largely the same between all of them and they leverage `_getAttrVerificationFn()`.

## Validation Steps Performed
- @codeofdusk has been testing this Conhost build
- Tests added for Conhost and shared implementation
- Windows Terminal changes were manually verified using accessibility insights and NVDA
@ghost
Copy link

ghost commented Jul 14, 2021

🎉This issue was addressed in #10336, which has now been successfully released as Windows Terminal Preview v1.10.1933.0.:tada:

Handy links:

This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Area-Accessibility Issues related to accessibility Issue-Feature Complex enough to require an in depth planning process and actual budgeted, scheduled work. Product-Conhost For issues in the Console codebase Product-Terminal The new Windows Terminal. Resolution-Fix-Committed Fix is checked in, but it might be 3-4 weeks until a release.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants