Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement ComWrappers for Windows Automation #13924

Closed
wants to merge 7 commits into from

Conversation

hez2010
Copy link
Contributor

@hez2010 hez2010 commented Dec 12, 2023

What does the pull request do?

Implements ComWrappers for win32 automation to enable automation support under trimming and nativeaot scenarios.

Note that it's not working correctly now and I'm still investigating it. Open as a draft PR.

BTW, do we have a way to debug a11y on Windows?

What is the current behavior?

Automation is not available for NativeAOT and trimming.

Checklist

Breaking changes

I changed some methods in interface to properties. If this is not desired, I can revert such changes.

Fixed issues

Fixes #8006

void Expand();
void Collapse();
ExpandCollapseState ExpandCollapseState { get; }
}

#if NET6_0_OR_GREATER
internal static unsafe class IExpandCollapseProviderManagedWrapper
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need code-generated marshallers checked into the repository?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not generated code, those wrappers are written by hand :)
Currently the com generator cannot handle properties in interfaces and variant marshalling.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should be using microcom codegen then. The main reason why it wasn't done already is that VARIANT/SAFEARRAY need to be marshalled, but since you are planning to handle them by hand it would be better to just declare those structs in microcom IDL.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds fair. Maybe we can just extend the microcom to support VARIANT/SAFEARRAY marshalling?
I'm wondering are there any other pieces of microcom missing now.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would be mostly runtime library support to convert between .NET and native representations, not many changes on codegen side are required

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we have unwrapped SAFEARRAY usages anywhere?

We only use it for marshaling/unmarshaling when crossing the native-managed boundary.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean, is it used directly anywhere or is it always wrapped into a VARIANT?

Copy link
Contributor Author

@hez2010 hez2010 Dec 19, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ahh yeah it's being used directly. for example, ISelectionProvider.GetSelection, IRawElementProviderFragment.GetEmbeddedFragmentRoots and IRawElementProviderFragment.GetRuntimeId which all return a SAFEARRAY to native side directly.
Here is an example of how I handle SAFEARRAY: https://github.com/AvaloniaUI/Avalonia/pull/13924/files#diff-d989ae9f3597b9182989e4060d4360100abb4f2c0bd51d4fd290c8b05b6698e7R57
I think microcom can be updated to support marshaling SAFEARRAY directly?

Copy link
Contributor Author

@hez2010 hez2010 Dec 20, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems that Marshal.GetNativeVariantForObject and Marshal.GetObjectForNativeVariant do work with NativeAOT.

Marshal.GetNativeVariantForObject and Marshal.GetObjectForNativeVariant require built-in COM which is disabled after NativeAOT and trimming. While I think we can take some code from dotnet/runtime#93635

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This snippet does run when compiled with NativeAOT.

using System.Runtime.InteropServices;

public static unsafe class Program
{
    public static void Main()
    {

        byte* mem = stackalloc byte[256];
        
        Marshal.GetNativeVariantForObject((int)5, (IntPtr)mem);

        var obj = Marshal.GetObjectForNativeVariant((IntPtr)mem);
        
        Console.WriteLine(obj);

    }
}

SAFEARRAY would require manual handling, I guess.

@hez2010
Copy link
Contributor Author

hez2010 commented Dec 26, 2023

I currently have no idea about which part is going wrong. From the automation verify tools seems that it's unable to get the content of the window...

image

@@ -82,6 +82,10 @@ public static void Initialize(Win32PlatformOptions options)
{
s_options = options;

#if NET6_0_OR_GREATER
ComWrappers.RegisterForMarshalling(Automation.AutomationNodeComWrappers.Instance);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that we should never call this from framework code, since there is only one instance per app.

@maxkatz6
Copy link
Member

maxkatz6 commented Jul 29, 2024

Out of interest, I checked how CsWin32 would generate these interfaces.
Apparently, you surely can make it generate COM APIs, and you have two options how:

  1. As structs, where you need to manually keep track of references (which is fine), but it doesn't seem possible to implement interfaces to be passed to the native world.
  2. As interfaces, but this approach will use legacy COM interop, and ComWrappers are required.

Potentially, this PR could be slightly reduced in size with CsWin32, while still requiring Com Wrappers. Which isn't ideal, but still.
SafeArray/Variant would still need to be manually marshalled, I assume, which is a smaller problem.

Personally, I am even considering whether we should drop accessibility support for anything older than .NET 8 and use new ComInterop source generators instead. Upd: nah, it doesn't support SafeArray and Variant (Variant is supported in .NET 9 though).

MicroCom is still a possible option here.

@maxkatz6
Copy link
Member

I was able to use ComInterfaceGenerator with .NET 8 while still keeping legacy Com interop on older platforms, with minimally implemented SafeArray/ComVariant. Will create a PR tomorrow so it can be tested.

@maxkatz6
Copy link
Member

Closing this PR, as we are going to go with #16543 approach

@maxkatz6 maxkatz6 closed this Sep 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Avalonia.Win32.Interop.Automation is not compatible with trimming and NativeAOT
4 participants