Implement support for writing UI Automation Remote Operations natively in NVDA using Python #16214
Conversation
… we now use the lowlevel Remote Ops functionality in Windows directly.
…ext manager like IfBlock. EloseBlock must only appear directly after the closing of IfBlock.
…g an element / textRange, and may do it sown separate cross-process call. Not worth it.
…, multiply and devide.
…a function to execute, plus objects to import, and returns the objects included in results. The function takes a RemoteFuncAPI instance, and any remoted objects as arguments, and is epected to return any remote objects that should be included in the results.
…moteFuncAPI nito highLevel.
…a 'set' method to all remote objects. Add ability for methods to cache remoted local constants to reduce amount of instructions needed.
…types to make instruction dumps a little more readable.
* Instruction commenting * build caching
* If remoteLogging argument is different to the build cache, then the build cache is not used. * dumpInstructions is now its own keyword argument. If true the build cache is not used, and an instruction dump is logged. and the build cache is still updated.
…ogic in highLevel.execute. Also move all the exception classes into highLevel.
…__ so it can be captured like: with rfa.catchBlock() as e: and the operation status is cleared before executing the catch body, thus now catchBlock correctly stops the error from causing the entire operation from failing.
…re). Fix remote logging.
…d importing of elements / text ranges.
…adding comments and meta commands. Importing of elements and textRanges are now printed as a meta command.
See test results for failed build of commit 8e47706e76 |
…d of winVersion checks where remote is used.
…ded tests, and make the long_iterable one a little simpler.
… no dll to cleanup anymore.
Co-authored-by: Luke Davis <8139760+XLTechie@users.noreply.github.com>
tests/unit/test_UIARemoteOps/test_highLevel/test_instructionLimit.py
Outdated
Show resolved
Hide resolved
| class _TypedInstruction(InstructionBase): | ||
|
|
||
| @property | ||
| def params(self): |
There was a problem hiding this comment.
can you add a return type here
See test results for failed build of commit 04d14a7bc7 |
|
cc @mltony |
|
It's great to see this coming to master, thank you very much! |
|
@cary-rowen - that's actually not up to me - the problem is that textInfo implementation is too slow in MSWord in current implementation. If this PR would allow faster textInfo then yes. But I don't understand low level UIA enough to tell whetehr this will allow to speed up TextInfo enough. |
|
this pr does not speedup any textInfos yet. However it does lay down
some of the groundwork to do this in the future.
Message ID: ***@***.***>
|
|
I think this needs a changelog entry for developers. right? cc: @jcsteh |
|
The docs here are pretty thorough, do you think anything is missing? That wiki page is pretty out of date, we have goals of moving anything relevant into projectDocs/dev/ like the current up-to-date technical design overview |
|
I think it is not obvious for a new dev where to look for the thorough documentation for it, so I think it would need at least some introductory notes with relevant links in the internal design overview or in the developers guide or so. Von meinem iPhone gesendetAm 26.06.2024 um 02:32 schrieb Sean Budd ***@***.***>:
The docs here are pretty thorough, do you think anything is missing?
https://github.com/nvaccess/nvda/blob/master/source/UIAHandler/_remoteOps/readme.md
That wiki page is pretty out of date, we have goals of moving anything relevant into projectDocs/dev/ like the current up-to-date technical design overview
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: ***@***.***>
|
|
Would you open an issue to track any remaining work you think there is? |
Link to issue number:
None
Background
Several years ago Microsoft added a low-level API in Windows to allow the caller to execute multiple actions in a UI Automation remote provider via only one cross-process call.
Limiting the amount of cross-process calls is necessary for providing a responsive experience, especially for interacting with applications hosted in Windows Defender Application Guard (WDAG) or as a Remote Application Integrated Locally (RAIL). There are also scenarios such as searching for / listing headings in large documents, which can bennifit from limiting cross-process calls, even when the application is running on the local machine.
the Windows.UI.UIAutomation.Core.CoreAutomationRemoteOperation is a winRT interface that allows the caller to specify the actions as a string of byte code representing a specific set of instructions for a conceptual virtual machine which is executed within the remote provider. The instructions allow you to manipulate UI Automation elements, patterns and text ranges, as well as perform basic logic and collect information using ints, floats, strings, GUIDs and arrays created within the conceptual virtual machine.
Although these instructions are well-defined, public documentation is limited and programming with these raw instructions can be slow and complicated to read.
Microsoft released the Microsoft-UI-UIAutomation Remote Operations Library which provides some higher level ways of building and executing these instructions, including intermediate winRT interfaces, and high-level pure c++/winRT interfaces.
NV Access has made many prototypes with this library, and has made significant contributions to that project over the years.
Today, NVDA includes this library as a depedency, and currently uses it to access the new UI Automation Custom Extensions feature, which is required to access certain information in Microsoft Word. In theory, NVDA could make a much greater uuse of UI automation Remote Operations to improve performance across many parts of Windows and other applications.
However, there are several issues with the Microsoft UI Automation Remote Operations Library which slow down or limit greater usage in NVDA, including:
Solution
This PR introduces a new remote operations library written in pure Python, which wraps the low-level CoreAutomationRemoteOperation winRT interfaces in Windows. This new library replaces the need for the older Microsoft Remote Operations Library, and therefore is no longer a dependency.
I had several initial goals in mind with this new library:
I was able to achieve both of these goals.
Description of user facing changes
Description of development approach
UIAHandler._remoteOpspackage has been added, which interfaces with the low-level Windows Remote Operations API, providing higher-level APIs and types to aide in building and executing easy to read remote algorithms. See The NVDA remote operations readme for explanations and example code for all features.Testing strategy:
Known issues with pull request:
Code Review Checklist:
Summary by CodeRabbit
Chores
microsoft-ui-uiautomationsubmodule and its references from the project.New Features
Documentation
microsoft-ui-uiautomationand added guidelines for new remote operations features.