Cannot get cross-targeting to work w/Native AOT #88971
Comments
|
Tagging subscribers to this area: @agocke, @MichalStrehovsky, @jkotas Issue DetailsFollowing the instructions at:
My Dockerfile: https://github.com/richlander/dotnet-docker/blob/dotnet-8-samples/samples/releasesapi-aot/Dockerfile.cross-target Host: Docker build: I get this error: Generating native code
ILC: Method '[releasesapi]Program.<Main>$(string[])' will always throw because: Failed to load assembly 'Microsoft.AspNetCore'
/usr/bin/ld.bfd: unrecognised emulation mode: aarch64linux
Supported emulations: elf_x86_64 elf32_x86_64 elf_i386 elf_iamcu elf_l1om elf_k1om
clang-16 : error : linker command failed with exit code 1 (use -v to see invocation) [/source/releasesapi.csproj]
/root/.nuget/packages/microsoft.dotnet.ilcompiler/8.0.0-preview.6.23329.7/build/Microsoft.NETCore.Native.targets(364,5): error MSB3073: The command ""clang-16" "obj/Release/net8.0/linux-arm64/native/releasesapi.o" -o "bin/Release/net8.0/linux-arm64/native/releasesapi" -fuse-ld=bfd /root/.nuget/packages/runtime.linux-arm64.microsoft.dotnet.ilcompiler/8.0.0-preview.6.23329.7/sdk/libbootstrapper.a /root/.nuget/packages/runtime.linux-arm64.microsoft.dotnet.ilcompiler/8.0.0-preview.6.23329.7/sdk/libRuntime.ServerGC.a /root/.nuget/packages/runtime.linux-arm64.microsoft.dotnet.ilcompiler/8.0.0-preview.6.23329.7/sdk/libeventpipe-enabled.a /root/.nuget/packages/runtime.linux-arm64.microsoft.dotnet.ilcompiler/8.0.0-preview.6.23329.7/sdk/libstdc++compat.a /root/.nuget/packages/runtime.linux-arm64.microsoft.dotnet.ilcompiler/8.0.0-preview.6.23329.7/framework/libSystem.Native.a /root/.nuget/packages/runtime.linux-arm64.microsoft.dotnet.ilcompiler/8.0.0-preview.6.23329.7/framework/libSystem.IO.Compression.Native.a /root/.nuget/packages/runtime.linux-arm64.microsoft.dotnet.ilcompiler/8.0.0-preview.6.23329.7/framework/libSystem.Net.Security.Native.a /root/.nuget/packages/runtime.linux-arm64.microsoft.dotnet.ilcompiler/8.0.0-preview.6.23329.7/framework/libSystem.Security.Cryptography.Native.OpenSsl.a --sysroot=/crossrootfs/arm64 --target=aarch64-linux-gnu -g '-Wl,-rpath,$ORIGIN' -Wl,--build-id=sha1 -Wl,--as-needed -pthread -ldl -lz -lrt -lm -pie -Wl,-pie -Wl,-z,relro -Wl,-z,now -Wl,--eh-frame-hdr -Wl,--discard-all -Wl,--gc-sections" exited with code 1. [/source/releasesapi.csproj]
The command '/bin/sh -c dotnet publish -a arm64 -o /app -p:CppCompilerAndLinker=clang-16 -p:SysRoot=/crossrootfs/arm64 releasesapi.csproj' returned a non-zero code: 1
|
|
--- a/samples/releasesapi-aot/Dockerfile.cross-target
+++ b/samples/releasesapi-aot/Dockerfile.cross-target
@@ -2,7 +2,6 @@
# https://github.com/dotnet/dotnet-docker/blob/main/samples/README.md
FROM mcr.microsoft.com/dotnet-buildtools/prereqs:cbl-mariner-2.0-cross-arm64 AS build
COPY --from=mcr.microsoft.com/dotnet/sdk:8.0-preview-cbl-mariner2.0 /usr/share/dotnet /usr/share/dotnet
-RUN tdnf install -y build-essential
RUN ln -s /usr/share/dotnet/dotnet /usr/bin/dotnet
WORKDIR /source
@@ -12,7 +11,7 @@ RUN dotnet restore -a arm64
# copy and publish app and libraries
COPY . .
-RUN dotnet publish -a arm64 -o /app -p:CppCompilerAndLinker=clang-16 -p:SysRoot=/crossrootfs/arm64 releasesapi.csproj
+RUN dotnet publish -a arm64 -o /app -p:CppCompilerAndLinker=clang-16 -p:SysRoot=/crossrootfs/arm64 -p:LinkerFlavor=lld releasesapi.csproj
RUN rm /app/*.dbg /app/*.Development.jsonIn my testing, current version of |
|
in my case cross compiling works for arm64 on amd64 with a little hack for i cant use ubuntu images since i cant install |
|
There is a simpler |
|
dn-vm/dnvm@c429bf6 demonstrates a working solution. |
|
Here's a Dockerfile that works. Imagine all the apt commands being in the SDK layer. # Learn about building .NET container images:
# https://github.com/dotnet/dotnet-docker/blob/main/samples/README.md
FROM --platform=$BUILDPLATFORM mcr.microsoft.com/dotnet/sdk:8.0-preview AS build
ARG TARGETARCH
RUN dpkg --add-architecture arm64
RUN apt update && apt install -y clang zlib1g-dev zlib1g-dev:arm64 \
gcc-aarch64-linux-gnu
WORKDIR /source
# copy csproj and restore as distinct layers
COPY *.csproj .
RUN dotnet restore -a $TARGETARCH
# copy and publish app and libraries
COPY . .
RUN if [ "$TARGETARCH" = "arm64" ]; then \
export OBJCOPY=aarch64-linux-gnu-objcopy
else \
export OBJCOPY=objcopy
fi; \
dotnet publish -a $TARGETARCH --self-contained -o /app releasesapi.csproj -p:ObjCopyName=$OBJCOPY
RUN rm /app/*.dbg /app/*.Development.json
# final stage/image
FROM mcr.microsoft.com/dotnet/runtime-deps:8.0-preview
WORKDIR /app
COPY --from=build /app .
# USER $APP_UID
ENTRYPOINT ["./releasesapi"]Given this Dockerfile, on x64, both of these commands work: docker build -t app .
docker build -t app --platform linux/arm64 .The Is there a way we could move that if statement into the SDK somehow? For example, I could imagine this (the SDK would pick the one that matches dotnet publish -a $TARGETARCH --self-contained -o /app releasesapi.csproj -p:ObjCopyName-x64=objcopy -p:ObjCopyName-arm64=aarch64-linux-gnu-objcopy |
|
We should be able to get rid of the manually specified |
|
I tried using This: dotnet publish -a $TARGETARCH --self-contained -o /app -p:LinkerFlavor=lld releasesapi.csprojResults in: clang : error : invalid linker name in argument '-fuse-ld=lld' |
|
The original issue (top post) was CBL mariner image with clang-16 missing cross-platform binutils, for which the fix is to simply use
clang 7 and above support |
User can have a non-default version of binutils or multiple toolchains installed side-by-side (e.g. different versions of binutils or different toolchain flavors). With cmake, we can easily search for tools belonging to the selected toolchain efficiently, I think the right way would be to either take dependency on cmake for cross scenarios or implement the probing logic in a shell script to locate the matching linker and objcopy. Also, these days it is much easier to run arch-specific docker image and bypass the complexities of cross toolchain: # build arm64 image with NativeAOT prereqs:
$ docker build -t dotnet8-aot-arm64-prereqs - <<EOF
FROM arm64v8/ubuntu:latest
RUN apt update && apt install -y clang build-essential libicu-dev curl libz-dev
RUN curl -sSL https://dot.net/v1/dotnet-install.sh | bash /dev/stdin -quality preview -channel 8.0
ENV PATH=\${PATH}:/root/.dotnet
# warmup the image with NativeAOT and R2R packs
RUN dotnet new console -o /tmp/warmup
RUN dotnet publish -p:PublishAot=true /tmp/warmup
RUN dotnet publish -p:PublishReadyToRun=true /tmp/warmup
EOF
# usage:
$ docker run --rm -v$(pwd):/myproject dotnet8-aot-arm64-prereqs \
dotnet publish -p:PublishAot=true /myproject |
|
I'm using the $ clang --version
Debian clang version 14.0.6
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
$ cat /etc/os-release | head -n 1
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"My goal here is to enable users to create a multi-platform-friendly Dockerfile. We do that with the SDK today. Context: https://devblogs.microsoft.com/dotnet/improving-multiplatform-container-support/ It seems like we are really close here to enabling the same experience for native AOT. I like the warmup you are doing. Very nice. |
|
OK. I figured the most critical issue. |
|
This works. Again, imagine that the two This is very similar to what @agocke is doing in his working example. FROM --platform=$BUILDPLATFORM mcr.microsoft.com/dotnet/sdk:8.0-preview AS build
ARG TARGETARCH
RUN dpkg --add-architecture arm64
RUN apt update && apt install -y clang zlib1g-dev zlib1g-dev:arm64 gcc-aarch64-linux-gnu llvm
WORKDIR /source
# copy csproj and restore as distinct layers
COPY *.csproj .
RUN dotnet restore -a $TARGETARCH
# copy and publish app and libraries
COPY . .
RUN dotnet publish -a $TARGETARCH --self-contained -o /app releasesapi.csproj
RUN rm /app/*.dbg /app/*.Development.json
# final stage/image
FROM mcr.microsoft.com/dotnet/runtime-deps:8.0-preview
WORKDIR /app
COPY --from=build /app .
USER $APP_UID
ENTRYPOINT ["./releasesapi"] |
|
I'm thinking -- given the size of these dependencies -- that we could offer three
|
Ok, it is different than:
that one is using CBL-mariner with llvm toolchain installed, incl. lld and llvm-objcopy (without the version suffix), which is why #88971 (comment) applies. Since the official .NET CI in .NET 8 is using the same CBL-based images with cross-toolchain installed, to validate PR changes and create official builds; I think it would be a good idea for end-user to consume the same base images. If a user, who prefers the $other distro, wants to build the cross-toolchain, they are free to build a custom image. Alternatively, IMO the best way in this day and age is to not engage with cross-compilation at all, and instead use cross-arch docker image. On win, macOS, and some linux distros, we don't need to install extra packages to enable cross-arch'ness anymore; just install docker and all the dependencies get in place. Using this approach will make sure our project file and/or command-line arguments are no more unusual than |
|
@am11 dotnet sdk does not support qemu thats why cross compilation (amd64, arm64) is needed for now. |
|
@ivanjx, .NET 8 SDK works with QEMU just fine. Make sure to disable the write-xor-execute feature;
emulation is usually slower than cross, but on CI build machine, it doesn't matter that much (as long as the builds are succeeding). |
Right, so evidentially docs could use a lighter tone and avoid calling stuff "unsupported" when runtime (the thing which basically matters most for platform support) does not enforce any restrictions for emulators, and neither does the SDK specifically disallows them. If note about unsupported stuff is really necessary in the docs, it should read something like: " |
|
|
I will go back to the cbl-mariner approach next. As is likely obvious, the Debian and Ubuntu latest approaches will not produce equivalent results to the cbl-mariner one. However, the cbl-mariner one isn't a great default option. |
|
FYI: I believe I have everything I need working now. Thanks MUCH for the help. It was tremendously useful. The final results were simpler than where I started, which is great. I also learned a fair bit. I'll be updating my samples PR in the next 24 hours or so with my results. |
ghost
locked as resolved and limited conversation to collaborators
Following the instructions at:
My Dockerfile: https://github.com/richlander/dotnet-docker/blob/dotnet-8-samples/samples/releasesapi-aot/Dockerfile.cross-target
Host:
Linux kelowna 6.1.38-1-MANJARO #1 SMP PREEMPT_DYNAMIC Wed Jul 5 23:49:30 UTC 2023 x86_64 GNU/LinuxDocker build:
docker build --pull -t app -f Dockerfile.cross-target .I get this error:
Related: Is this guidance about which NuGet packages to work correct?
https://github.com/dotnet/runtime/blob/main/src/coreclr/nativeaot/docs/compiling.md#cross-architecture-compilation
Follow up to #88942
The text was updated successfully, but these errors were encountered: