Using ObjCRuntime from custom .NET runtime host (or outside a Xamarin app bundle in general)

[marzent]

I am attempting to use the ObjCRuntime from a custom .NET runtime host on macOS, specifically targeting net8.0-macos (same issue for net7 as well). However, I encounter the following exception when executing a DLL directly:

Unhandled exception. System.DllNotFoundException: Unable to load shared library '__Internal' or one of its dependencies. In order to help diagnose loading problems, consider setting the DYLD_PRINT_LIBRARIES environment variable: 
dlopen(/Users/marc-aurel/Documents/repos/Aetherium/Aetherium/bin/Debug/net8.0-macos/osx-arm64/__Internal.dylib, 0x0001): tried: '/Users/marc-aurel/Documents/repos/Aetherium/Aetherium/bin/Debug/net8.0-macos/osx-arm64/__Internal.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/marc-aurel/Documents/repos/Aetherium/Aetherium/bin/Debug/net8.0-macos/osx-arm64/__Internal.dylib' (no such file), '/Users/marc-aurel/Documents/repos/Aetherium/Aetherium/bin/Debug/net8.0-macos/osx-arm64/__Internal.dylib' (no such file)
...

The error message suggests that the __Internal library or its dependencies cannot be loaded. I understand that currently, the only way to run the application is through the executable in the generated app bundle which provides these exported functions.

For my use case it would be helpful if the initialization process performed by the Xamarin wrapper could be replicated in a custom .NET runtime host, or even better all necessary bindings moved to a native portable dylib. This would enable the use of ObjCRuntime and other Xamarin bindings outside of the generated app bundle.

There seems to be a similar issue here ForNeVeR/AvaloniaRider#237 (comment).

Steps to Reproduce

1. Create a new macOS project using the command dotnet new macos.
2. Build the project using the command dotnet build.
3. Run the generated DLL directly using the command dotnet MyApp.dll.

Expected Behavior

Runs the application.

Actual Behavior

Exception described above.

Environment

Version information

Visual Studio Community 2022 for Mac
Version 17.5.6 (build 3)
Installation UUID: af06fdaa-a9cd-45fc-8889-e9912615f626

Runtime
.NET 7.0.1 (64-bit)
Architecture: Arm64
Microsoft.macOS.Sdk 12.3.2372; git-rev-head:754abbf6a3563f6267e5717ae832b4ac25b1f2fb; git-branch:release/7.0.1xx-xcode13.3

Roslyn (Language Service)
4.5.0-3.23056.2+97881342e427ff5cdcba8f12b12ff8e6f3564431

NuGet
Version: 6.4.0.117

.NET SDK (Arm64)
SDK: /usr/local/share/dotnet/sdk/7.0.302/Sdks
SDK Versions:
	8.0.100-preview.4.23260.5
	8.0.100-preview.1.23115.2
	8.0.100-preview.1.23109.10
	7.0.302
	6.0.408
MSBuild SDKs: /Applications/Visual Studio.app/Contents/MonoBundle/MSBuild/Current/bin/Sdks

.NET Runtime (Arm64)
Runtime: /usr/local/share/dotnet/dotnet
Runtime Versions:
	8.0.0-preview.4.23259.5
	8.0.0-preview.1.23110.8
	8.0.0-preview.1.23106.5
	7.0.5
	6.0.16

Xamarin Designer
Version: 17.5.3.47
Hash: e8b5d371c3
Branch: remotes/origin/d17-5
Build date: 2023-05-18 17:57:49 UTC

Xamarin.Profiler
Version: 1.8.0.49
Location: /Applications/Xamarin Profiler.app/Contents/MacOS/Xamarin Profiler

Updater
Version: 11

Xamarin.Android
Not Installed

Microsoft Build of OpenJDK
Java SDK: /Library/Java/JavaVirtualMachines/microsoft-11.jdk
11.0.16.1
Android Designer EPL code available here:
https://github.com/xamarin/AndroidDesigner.EPL

Eclipse Temurin JDK
Java SDK: Not Found

Android SDK Manager
Version: 17.5.0.33
Hash: f0c0c52
Branch: remotes/origin/d17-5~2
Build date: 2023-05-18 17:57:54 UTC

Android Device Manager
Version: 0.0.0.1245
Hash: 7f8a990
Branch: 7f8a990
Build date: 2023-05-18 17:57:55 UTC

Apple Developer Tools
Xcode: 14.3.1 21815
Build: 14E300c

Xamarin.Mac
Version: 9.1.0.5 Visual Studio Community
Hash: 7738c90c9
Branch: xcode14.2
Build date: 2023-01-25 15:56:14-0500

Xamarin.iOS
Not Installed

Build Information
Release ID: 1705060003
Git revision: 6472fea49fa01feff06feefde4e358c02df42934
Build date: 2023-05-18 17:56:09+00
Build branch: release-17.5
Build lane: release-17.5

Operating System
Mac OS X 13.4.0
Darwin 22.5.0 Darwin Kernel Version 22.5.0
    Mon Apr 24 20:53:19 PDT 2023
    root:xnu-8796.121.2~5/RELEASE_ARM64_T6020 arm64

Build Logs

N/A

Example Project (If Possible)

Described in steps to reproduce

[marzent]

I also stumbled upon https://github.com/veldrid/veldrid/blob/367f3ec52f92ffe65681cafd16cb65d9f89e0af0/src/Veldrid.MetalBindings/ObjectiveCRuntime.cs which seems to be the only other way to interact with the ObjectiveC runtime and seems to work fine so far.

It would still be nice to have easily accessible (and more complete) official bindings though.

[rolfbjarne]

We might be able to ship as only two separate files: a native dylib and the managed Microsoft.macOS.dll assembly, and then the consumer would need to some API to load/initialize the native library and/or Microsoft.macOS.dll before being able to use any other API in Microsoft.macOS.dll.

However, this is a rather big undertaking, and is unlikely to happen any time soon.

There are thus two options:

1. Someone can contribute the required changes.
2. Get enough people to support this proposal to make our priorities change.

In any case I'm leaving this issue open, since it's a totally valid request.

[jwosty]

Could you kindly give a general breakdown of what tasks would be required to make this happen? I'd love to know where to start experimenting with stuff.

Does NativeAOT impact this, now that it's supported? Would you be able to statically link against the native code (built as a dylib) and have it "just work" without needing a new API to initialize said library?

[rolfbjarne]

Does NativeAOT impact this, now that it's supported? Would you be able to statically link against the native code (built as a dylib) and have it "just work" without needing a new API to initialize said library?

No, unfortunately NativeAOT does not make things simpler at the moment. In order to make NativeAOT work we currently we require the trimmer to run with some custom linker steps that transforms our existing codebase into something that works with NativeAOT, and that would not be easy to implement outside our app bundling logic.

Could you kindly give a general breakdown of what tasks would be required to make this happen? I'd love to know where to start experimenting with stuff.

The first step would be to define exactly what you want to do (from a developer/user perspective).

From the top of my head here are a few ideas, with the immediate drawbacks I can think of:

1. Do you want a single, self-contained macOS executable?
   • This requires putting the assemblies themselves into the executable, and all the problems related to not having the assembly on disk (which introduces its own set of incompatibilities).
2. Do you want a console app you can dotnet exec?
   • This requires having an installed .NET somewhere accessible.
3. Do you want a self-contained console app you can dotnet publish -r osx-arm64
   • AFAIK there's currently no support for creating universal console apps like this in .NET, so if you have to choose whether you want to execute on both x64 and arm64 (and then publish for x64), or if you're ok with only supporting arm64.

Personally I feel that option 2 and 3 are what people are most likely to want, and I believe it's also much easier to implement.

I think I'd go for making something like this work:

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net8.0</TargetFramework>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Microsoft.macOS.Console" Version="1.2.3.4" />       
  </ItemGroup>
</Project>

Features that would have to be dropped:

• The registrar: we register all managed types that subclass NSObject with the Objective-C runtime at startup. We've made this fast (with our static registrar), but that would be rather difficult to port to this scenario (it involves generating Objective-C code and linking it into the native executable). For this type of console apps it might not be necessary to register all managed types at startup, but there might be surprises if we don't (one thing I know won't work is loading xibs/storyboards that reference any custom subclasses).
• NativeAOT won't work: related to the above point about the registrar, we have a dynamic registrat that's capable of registering types as they're used, but it doesn't work with NativeAOT. The dynamic registrar would either have to be modified (which might not be possible), or NativeAOT would have to be fixed.
• (probably lots of other things as well)

This would be my way to create a proof-of-concept:

1. Create a test project (see example csproj) above, and try to make it work.
   2. Create a test NuGet named "Microsoft.macOS.Console", put Microsoft.macOS.dll in it, together with libxamarin-dotnet-coreclr.dylib from the existing "Microsoft.macOS.Runtime.osx-x64" NuGet.
2. Clone this repository (xamarin-macios) and build it locally. If you need a modified Microsoft.macOS.dll or libxamarin-dotnet-coreclr.dylib you'll be able to build and copy them to your test NuGet.

[SimonWeinbergerEnscape]

I'm looking for a way to use the Xamarin bindings for MacOS without having a fully fledged AppBundle. Our software is just a (unmanaged) plugin to another software and hosts the .NET runtime itself (as described in https://learn.microsoft.com/en-us/dotnet/core/tutorials/netcore-hosting). I tried switching the TargetFramework to net8.0-macos, but that doesn't work. I'm stuck with using net8.0.

So if there was a way to reference Microsoft.macOS.dll, and a way to initialize things correctly, that would be great. Right now we write bindings to every ObjC type that we want to use manually, and that gets tedious.

[Houfeng]

How is the problem now? Do you have a plan to fix it?

[rolfbjarne]

How is the problem now? Do you have a plan to fix it?

There has been no progress, and it's not on the roadmap at the moment either.