Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[mono] Build native osx-arm64 versions of the cross compilers #74175

Closed
lambdageek opened this issue Aug 18, 2022 · 12 comments · Fixed by #74428
Closed

[mono] Build native osx-arm64 versions of the cross compilers #74175

lambdageek opened this issue Aug 18, 2022 · 12 comments · Fixed by #74428
Assignees
Milestone

Comments

@lambdageek
Copy link
Member

In .NET 6 we ship mono-aot-cross targeting our various platforms as an osx-x64 executable and rely on Rosetta to run it.

We should investigate if we can ship alternative runtime packs that include osx-arm64 compilers.

First step is to figure out if there are any issues building osx-arm64 hosted versions of the cross-compilers.

@ghost ghost added the untriaged New issue has not been triaged by the area owner label Aug 18, 2022
@lambdageek lambdageek added this to the 8.0.0 milestone Aug 18, 2022
@ghost ghost removed the untriaged New issue has not been triaged by the area owner label Aug 18, 2022
@lambdageek lambdageek changed the title Build native osx-arm64 versions of the cross compilers [mono] Build native osx-arm64 versions of the cross compilers Aug 18, 2022
@eerhardt
Copy link
Member

I stumbled on this today trying to build a .NET MAUI app on my M1 mac:

/usr/local/share/dotnet/packs/Microsoft.MacCatalyst.Sdk/15.4.447/targets/Xamarin.Shared.Sdk.targets(1000,3): error MSB4018: The "AOTCompile" task failed unexpectedly. [/Users/eerhardt/git/EFCore/EFCore/EFCore.csproj]
/usr/local/share/dotnet/packs/Microsoft.MacCatalyst.Sdk/15.4.447/targets/Xamarin.Shared.Sdk.targets(1000,3): error MSB4018: System.AggregateException: One or more errors occurred. (One or more errors occurred. (An error occurred trying to start process '/usr/local/share/dotnet/packs/Microsoft.NETCore.App.Runtime.AOT.osx-x64.Cross.maccatalyst-arm64/6.0.8/Sdk/../tools/mono-aot-cross' with working directory '/Users/eerhardt/git/EFCore/EFCore'. Bad CPU type in executable)) [/Users/eerhardt/git/EFCore/EFCore/EFCore.csproj]

It would be great if our tools worked natively for arm64. Even the Apple support doc says:

In most cases, you won't notice any difference in the performance of an app that needs Rosetta. But you should contact the app developer to inquire about a version that can natively use the full power and performance of Apple silicon.

@directhex
Copy link
Contributor

directhex commented Aug 19, 2022

arm64, arm, wasm: builds
x64:

  In file included from /Users/directhex/Projects/runtime/src/mono/mono/mini/../../mono/metadata/exception-internals.h:11:
  In file included from /Users/directhex/Projects/runtime/src/mono/mono/mini/../../mono/metadata/handle.h:24:
  In file included from /Users/directhex/Projects/runtime/src/mono/mono/mini/../../mono/utils/mono-threads.h:16:
  In file included from /Users/directhex/Projects/runtime/src/mono/mono/mini/../../mono/utils/mono-stack-unwinding.h:12:
  /Users/directhex/Projects/runtime/src/mono/mono/mini/../../mono/utils/mono-context.h:289:27: error: array has incomplete element type 'MonoContextSimdReg' (aka 'struct __darwin_xmm_reg')
          MonoContextSimdReg fregs [AMD64_XMM_NREG];
                                   ^
  /Users/directhex/Projects/runtime/src/mono/mono/mini/../../mono/utils/mono-context.h:30:16: note: forward declaration of 'struct __darwin_xmm_reg'
  typedef struct __darwin_xmm_reg MonoContextSimdReg;

x86:

  In file included from /Users/directhex/Projects/runtime/src/mono/mono/mini/../../mono/metadata/exception-internals.h:11:
  In file included from /Users/directhex/Projects/runtime/src/mono/mono/mini/../../mono/metadata/handle.h:24:
  In file included from /Users/directhex/Projects/runtime/src/mono/mono/mini/../../mono/utils/mono-threads.h:16:
  In file included from /Users/directhex/Projects/runtime/src/mono/mono/mini/../../mono/utils/mono-stack-unwinding.h:12:
  /Users/directhex/Projects/runtime/src/mono/mono/mini/../../mono/utils/mono-context.h:164:30: error: array has incomplete element type 'MonoContextSimdReg' (aka 'struct __darwin_xmm_reg')
      MonoContextSimdReg fregs [X86_XMM_NREG];
                               ^
  /Users/directhex/Projects/runtime/src/mono/mono/mini/../../mono/utils/mono-context.h:25:16: note: forward declaration of 'struct __darwin_xmm_reg'
  typedef struct __darwin_xmm_reg MonoContextSimdReg;

This is VASTLY better than I expected, tbh. The build system appears to already do everything right - the problem area is x86/x64 targets. @lambdageek can you help with the runtime internals issues?

@eerhardt
Copy link
Member

We will need to create new "packs" for this work, correct? Today building a MAUI app uses /usr/local/share/dotnet/packs/Microsoft.NETCore.App.Runtime.AOT.osx-x64.Cross.maccatalyst-arm64, we will need to create Microsoft.NETCore.App.Runtime.AOT.osx-arm64.Cross.XXX packs, correct?

@directhex
Copy link
Contributor

@eerhardt correct. That's the easy part.

@directhex
Copy link
Contributor

Wait, I'm wrong, it's less good than I thought.

My tests were on an M1 Macbook - it does the right thing there for 3 target arches. It absolutely does not do the right thing on x64 (i.e. how our CI is set up).

@directhex
Copy link
Contributor

Made some initial progress (compiling android-arm64 cross compiler for linux-arm64 on linux-x64 - compiling works but linking fails due to us trying to use linux-x64 LLVM)

@lewing
Copy link
Member

lewing commented Aug 22, 2022

Windows (win-arm64) has the same issue, we're about to add emulated cross compilers for that case.

@directhex
Copy link
Contributor

directhex commented Aug 22, 2022

We currently have:

  • linux-x64 building browser-wasm cross-compiler for linux-x64
  • linux-x64 building android-x64 cross-compiler for linux-x64
  • linux-x64 building android-arm64 cross-compiler for linux-x64
  • linux-x64 building android-x86 cross-compiler for linux-x64
  • linux-x64 building android-arm cross-compiler for linux-x64
  • win-x64 building browser-wasm cross-compiler for win-x64
  • win-x64 building android-x64 cross-compiler for win-x64
  • win-x64 building android-arm64 cross-compiler for win-x64
  • win-x64 building android-x86 cross-compiler for win-x64
  • win-x64 building android-arm cross-compiler for win-x64
  • osx-x64 building browser-wasm cross-compiler for osx-x64
  • osx-x64 building android-x64 cross-compiler for osx-x64
  • osx-x64 building android-arm64 cross-compiler for osx-x64
  • osx-x64 building android-x86 cross-compiler for osx-x64
  • osx-x64 building android-arm cross-compiler for osx-x64
  • osx-x64 building ios-arm64 cross-compiler for osx-x64
  • osx-x64 building iossimulator-arm64 cross-compiler for osx-x64
  • osx-x64 building iossimulator-x64 cross-compiler for osx-x64
  • osx-x64 building iossimulator-x86 cross-compiler for osx-x64
  • osx-x64 building ios-arm cross-compiler for osx-x64
  • osx-x64 building tvos-arm64 cross-compiler for osx-x64
  • osx-x64 building tvossimulator-arm64 cross-compiler for osx-x64
  • osx-x64 building tvossimulator-x64 cross-compiler for osx-x64
  • osx-x64 building maccatalyst-arm64 cross-compiler for osx-x64
  • osx-x64 building maccatalyst-x64 cross-compiler for osx-x64

My aim is to additionally support:

  • linux-x64 building browser-wasm cross-compiler for linux-arm64
  • linux-x64 building android-x64 cross-compiler for linux-arm64
  • linux-x64 building android-arm64 cross-compiler for linux-arm64
  • linux-x64 building android-x86 cross-compiler for linux-arm64
  • linux-x64 building android-arm cross-compiler for linux-arm64
  • win-x64 building browser-wasm cross-compiler for win-arm64
  • win-x64 building android-x64 cross-compiler for win-arm64
  • win-x64 building android-arm64 cross-compiler for win-arm64
  • win-x64 building android-x86 cross-compiler for win-arm64
  • win-x64 building android-arm cross-compiler for win-arm64
  • osx-x64 building browser-wasm cross-compiler for osx-arm64
  • osx-x64 building android-x64 cross-compiler for osx-arm64
  • osx-x64 building android-arm64 cross-compiler for osx-arm64
  • osx-x64 building android-x86 cross-compiler for osx-arm64
  • osx-x64 building android-arm cross-compiler for osx-arm64
  • osx-x64 building ios-arm64 cross-compiler for osx-arm64
  • osx-x64 building iossimulator-arm64 cross-compiler for osx-arm64
  • osx-x64 building iossimulator-x64 cross-compiler for osx-arm64
  • osx-x64 building iossimulator-x86 cross-compiler for osx-arm64
  • osx-x64 building ios-arm cross-compiler for osx-arm64
  • osx-x64 building tvos-arm64 cross-compiler for osx-arm64
  • osx-x64 building tvossimulator-arm64 cross-compiler for osx-arm64
  • osx-x64 building tvossimulator-x64 cross-compiler for osx-arm64
  • osx-x64 building maccatalyst-arm64 cross-compiler for osx-arm64
  • osx-x64 building maccatalyst-x64 cross-compiler for osx-arm64

However, nothing I'm writing is hardcoded enough that we couldn't whimsically decide to build... a maccatalyst-arm64 cross-compiler for linux-ppc64le or something. We can hammer out the details once we have cross compilers building for $(TargetArchitecture) instead of $(Architecture). We don't need any of those Linux targets really (i.e. linux-arm64 isn't a key product for us) but it's my main test development target so removing it will add a few days at best to delivery.

@directhex
Copy link
Contributor

Currently, I am trying to rearchitect something I was never particularly happy with - to wit, we have MonoLLVMDir and MonoAOTLLVMDir which both contain exactly one different thing - what the build system thinks is an LLVM suitable for compiling an LLVM JIT for the current platform, and what the build system thinks is an LLVM suitable for compiling an LLVM AOT for the current platform. It's full of workarounds and hacks. I am now trying to a) simplify the hacks and b) make the logic more generic, so we just have MonoLLVMDir\arm64, MonoLLVMDir\x64, etc, and just use the appropriate one in each given case. Replacing the old logic which contorts itself in knots trying to make MonoAOTLLVMDir be the right thing early on for compiling something later on.

I have successfully produced Microsoft.NETCore.App.Runtime.AOT.linux-arm64.Cross.android-arm64.8.0.0-ci.nupkg with the current state of art of my branch. I'm sure I broke a lot of things (especially WASM) so tomorrow I'll set up a draft PR and see just how many things

@ghost ghost added the in-pr There is an active PR which will close this issue when it is merged label Aug 23, 2022
@directhex
Copy link
Contributor

Removing windows-arm64 cross-compilers from scope for this issue. Right now we can't even build a basic JIT for windows-arm64, and trying to fix that is a distinct issue from trying to build cross-compilers there.

@directhex
Copy link
Contributor

directhex commented Aug 26, 2022

(Update @lambdageek says: a newer version of this table is in #82495 (comment))

Build machine Cross-compiler target Executing machine Status
linux-x64 browser-wasm linux-arm64 ☑ In PR
linux-x64 android-x64 linux-arm64 ❌ Does not build
linux-x64 android-arm64 linux-arm64 ⚠ Builds, disabled in PR
linux-x64 android-x86 linux-arm64 ❌ Does not build
linux-x64 android-arm linux-arm64 ⚠ Builds, disabled in PR
win-x64 browser-wasm win-arm64 ☠ Does not build, removed from scope
win-x64 android-x64 win-arm64 ☠ Does not build, removed from scope
win-x64 android-arm64 win-arm64 ☠ Does not build, removed from scope
win-x64 android-x86 win-arm64 ☠ Does not build, removed from scope
win-x64 android-arm win-arm64 ☠ Does not build, removed from scope
osx-x64 browser-wasm osx-arm64 ☑ In PR
osx-x64 android-x64 osx-arm64 ❌ Does not build
osx-x64 android-arm64 osx-arm64 ⚠ Builds, disabled in PR
osx-x64 android-x86 osx-arm64 ❌ Does not build
osx-x64 android-arm osx-arm64 ⚠ Builds, disabled in PR
osx-x64 ios-arm64 osx-arm64 ⚠ Builds, disabled in PR
osx-x64 iossimulator-arm64 osx-arm64 ⚠ Builds, disabled in PR
osx-x64 iossimulator-x64 osx-arm64 ❌ Does not build
osx-x64 iossimulator-x86 osx-arm64 ❌ Does not build
osx-x64 ios-arm osx-arm64 ⚠ Builds, disabled in PR
osx-x64 tvos-arm64 osx-arm64 ⚠ Builds, disabled in PR
osx-x64 tvossimulator-arm64 osx-arm64 ⚠ Builds, disabled in PR
osx-x64 tvossimulator-x64 osx-arm64 ❌ Does not build
osx-x64 maccatalyst-arm64 osx-arm64 ⚠ Builds, disabled in PR
osx-x64 maccatalyst-x64 osx-arm64 ❌ Does not build

The way the cross-compiler build is structured, we have an all-or-nothing approach for all architectures supported by a given OS - which is to say, if you say /p:MonoCrossAOTTargetOS=MacCatalyst it will try to build a cross compiler for both maccatalyst-x64 and maccatalyst-arm64. So, as long as the issues in #74175 (comment) remain, a single failing build for a given OS means I have every arch for that OS turned off. Hence only browser-wasm right now.

Reproducing the issue locally

You'll need some offset files for a broken target. Go to the latest build on https://dev.azure.com/dnceng/public/_build?definitionId=686&_a=summary&branchFilter=149480 and click on the published artifacts. Scroll down to Mono_Offsets_Android, click the meatball menu, and click Download artifacts.

Unzip the zip file into artifacts/obj/mono/offsetfiles, i.e. you should have a file artifacts/obj/mono/offsetfiles/Android.x64.Release/cross/offsets-x64-android.h

Now, you can do an AOT build:

Linux:

docker run -e ROOTFS_DIR=/crossrootfs/arm64 -w `pwd` --platform linux/amd64 -v `pwd`:`pwd` -it mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-cross-arm64-20220427171722-6e40d49 ./build.sh -subset mono+packs -c release -arch arm64 -ci  /p:MonoCrossAOTTargetOS=Android /p:SkipMonoCrossJitConfigure=true /p:BuildMonoAOTCrossCompilerOnly=true /bl

Mac:

arch -x86_64 ./build.sh -subset mono+packs -c release -arch arm64 -ci  /p:MonoCrossAOTTargetOS=Android /p:SkipMonoCrossJitConfigure=true /p:BuildMonoAOTCrossCompilerOnly=true /bl

You can also use this method to build your own arm64 cross-compiler for browser-wasm, by using /p:MonoCrossAOTTargetOS=Browser and offsets from Mono_Offsets_Browser

@BretJohnson
Copy link
Member

FYI - Without this, one consequence is that the UI below shows up as part of the VSMac install.

image

directhex added a commit that referenced this issue Jan 5, 2023
This change enables building arbitrary cross-compilers for non-x64 hosts. Right now, only osx-arm64->browser-wasm and linux-arm64->browser-wasm are actually enabled, as runtime bugs prevent us from building any arm64->x86 or arm64->x64 cross-compilers. Getting this framework in will unblock workload work related to "native" wasm on M1 Mac.

== BREAKING CHANGE FOR DEVELOPERS ==
MonoAOTLLVMDir no longer exists.

Old:
`/p:MonoLLVMDir=/path/to/llvm` specifies the directory with LLVM binaries compiled for `$(TargetArchitecture)`, for compiling the Mono JIT on that target architecture
`/p:MonoAOTLLVMDir=/path/to/llvm` specifies the directory with LLVM binaries compiled for `$(BuildArchitecture)`, for compiling the mono-aot-cross executable
```

New:
`/p:MonoLLVMDir=/path/to/llvm` specifies a parent-level directory with LLVM binaries compiled for relevant architectures - e.g. `$(MonoLLVMDir)\x64` contains LLVM built for `x64`, current OS. `$(MonoLLVMDir)\arm64 contains LLVM built for arm64`, current OS.

You do not need to specify `/p:MonoLLVMDir=` manually, unless you are a developer who may be using custom, non-nupkg LLVM.

This change is necessary to stop hardcoding the assumptions that a) `$(BuildArchitecture)` is always `x64` and b) AOT binaries should be compiled for $(BuildArchitecture)

Closes: #74175
@ghost ghost removed the in-pr There is an active PR which will close this issue when it is merged label Jan 5, 2023
@ghost ghost locked as resolved and limited conversation to collaborators Feb 5, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants