Live apphost not being used by crossgen2 with source build #77667

ayakael · 2022-10-31T07:05:26Z

Description

.NET 7 in source-build does not currently use NativeAOT. Thus, when building Crossgen2 from source on platforms without apphost in nuget feeds, build of runtime fails with:

/var/build/dotnet7/testing/dotnet7-stage0/src/bootstrap/sdk/7.0.100/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(135,5): error NETSDK1084: There is no application host available for the specified RuntimeIdentifier 'linux-musl-x86'. [/var/build/dotnet7/testing/dotnet7-stage0/src/dotnet-e6dd91c290b808f971a1ac69c2fb29395bbf1051/src/runtime/src/coreclr/tools/aot/crossgen2/crossgen2.csproj]

Reproduction Steps

Crossbuild from x64 to x86, crossgen2 build will fail.

Expected behavior

Crossgen2 should be able to use to live apphost built previously by runtime

Actual behavior

Build fails due to no apphost for that RID

Regression?

Per #7335 and #66866, there was a time when this was a non-issue. With move to NativeAOT, UseLiveBuiltDotNetHost was dropped, and this use-case seems to have been removed.

Known Workarounds

None for now. I attempted to build runtime in two steps, to no avail:

ROOTFS_DIR="$CBUILDROOT" ./build.sh $args -subset Clr.Native+Host.Native
ROOTFS_DIR="$CBUILDROOT" ./build.sh $args /p:AppHostSourcePath="$builddir"/src/runtime/artifacts/obj/linux-musl-$_dotnet_target.Release/apphost/standalone/apphost

Configuration

.NET 7.0.100-rtm.22519.39
Alpine Linux Edge
x64 to x86 crossbuild

Other information

No response

The text was updated successfully, but these errors were encountered:

dotnet-issue-labeler · 2022-10-31T07:05:29Z

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

am11 · 2022-10-31T07:26:37Z

@agocke might be able to help but AFAIK, these days crossgen2 uses live build of ilc (NativeAOT) when available or fallback to live build of corehost on platforms where NativeAOT is not available. The use of prebuilt host (from nuget package) was completely removed. Looks like you are seeing a different behavior on x86; something is still expecting prebuilt host. Perhaps we are (unintentionally) skipping live build host on x86?

Could you try git clean -xdf && ./build.sh clr+host+libs+packs just to be sure?

ayakael · 2022-10-31T08:07:57Z

Building just those subsets gives the same result.

am11 · 2022-10-31T08:51:33Z

In 9a3626e, MicrosoftNETCoreDotNetHostVersion etc. were removed so no prebuilt apphost get restored. crossgen2 is published as a single-file using the live corehost build (when NativeAOT is not supported).

Maybe this patch will help:

--- a/src/coreclr/tools/aot/crossgen2/crossgen2.csproj
+++ b/src/coreclr/tools/aot/crossgen2/crossgen2.csproj
@@ -9,7 +9,7 @@
     <!-- Trimming is not currently working, but set the appropriate feature flags for NativeAOT -->
     <PublishTrimmed Condition="'$(NativeAotSupported)' == 'true'">true</PublishTrimmed>
     <RuntimeIdentifiers Condition="'$(RunningPublish)' != 'true' and '$(DotNetBuildFromSource)' != 'true'">linux-x64;linux-musl-x64;linux-arm;linux-musl-arm;linux-arm64;linux-musl-arm64;freebsd-x64;osx-x64;osx-arm64;win-x64;win-x86;win-arm64;win-arm</RuntimeIdentifiers>
-    <RuntimeIdentifiers Condition="'$(DotNetBuildFromSource)' == 'true'">$(PackageRID)</RuntimeIdentifiers>
+    <RuntimeIdentifiers Condition="'$(RunningPublish)' != 'true' and '$(DotNetBuildFromSource)' == 'true'">$(PackageRID)</RuntimeIdentifiers>
     <SelfContained>false</SelfContained>
     <SelfContained Condition="'$(RunningPublish)' == 'true'">true</SelfContained>
   </PropertyGroup>

agocke · 2022-10-31T17:36:50Z

This seems like it's specific to linux x86, right? I don't see any source build problems for other architectures.

ghost · 2022-10-31T17:37:42Z

Tagging subscribers to this area: @hoyosjs
See info in area-owners.md if you want to be subscribed.

Issue Details

Description

.NET 7 in source-build does not currently use NativeAOT. Thus, when building Crossgen2 from source on platforms without apphost in nuget feeds, build of runtime fails with:

/var/build/dotnet7/testing/dotnet7-stage0/src/bootstrap/sdk/7.0.100/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(135,5): error NETSDK1084: There is no application host available for the specified RuntimeIdentifier 'linux-musl-x86'. [/var/build/dotnet7/testing/dotnet7-stage0/src/dotnet-e6dd91c290b808f971a1ac69c2fb29395bbf1051/src/runtime/src/coreclr/tools/aot/crossgen2/crossgen2.csproj]

Reproduction Steps

Crossbuild from x64 to x86, crossgen2 build will fail.

Expected behavior

Crossgen2 should be able to use to live apphost built previously by runtime

Actual behavior

Build fails due to no apphost for that RID

Regression?

Per #7335 and #66866, there was a time when this was a non-issue. With move to NativeAOT, UseLiveBuiltDotNetHost was dropped, and this use-case seems to have been removed.

Known Workarounds

None for now. I attempted to build runtime in two steps, to no avail:

ROOTFS_DIR="$CBUILDROOT" ./build.sh $args -subset Clr.Native+Host.Native
ROOTFS_DIR="$CBUILDROOT" ./build.sh $args /p:AppHostSourcePath="$builddir"/src/runtime/artifacts/obj/linux-musl-$_dotnet_target.Release/apphost/standalone/apphost

Configuration

.NET 7.0.100-rtm.22519.39
Alpine Linux Edge
x64 to x86 crossbuild

Other information

No response

Author:	ayakael
Assignees:	-
Labels:	`area-Infrastructure-coreclr`
Milestone:	Future

ayakael · 2022-10-31T19:38:53Z

This seems like it's specific to linux x86, right? I don't see any source build problems for other architectures.

Indeed. I believe this is specific to x86 as crossgen2 isn't built on other community platforms (like ppc64le and s390x) as they use mono.

In 9a3626e, MicrosoftNETCoreDotNetHostVersion etc. were removed so no prebuilt apphost get restored. crossgen2 is published as a single-file using the live corehost build (when NativeAOT is not supported).

Maybe this patch will help:

--- a/src/coreclr/tools/aot/crossgen2/crossgen2.csproj
+++ b/src/coreclr/tools/aot/crossgen2/crossgen2.csproj
@@ -9,7 +9,7 @@
     <!-- Trimming is not currently working, but set the appropriate feature flags for NativeAOT -->
     <PublishTrimmed Condition="'$(NativeAotSupported)' == 'true'">true</PublishTrimmed>
     <RuntimeIdentifiers Condition="'$(RunningPublish)' != 'true' and '$(DotNetBuildFromSource)' != 'true'">linux-x64;linux-musl-x64;linux-arm;linux-musl-arm;linux-arm64;linux-musl-arm64;freebsd-x64;osx-x64;osx-arm64;win-x64;win-x86;win-arm64;win-arm</RuntimeIdentifiers>
-    <RuntimeIdentifiers Condition="'$(DotNetBuildFromSource)' == 'true'">$(PackageRID)</RuntimeIdentifiers>
+    <RuntimeIdentifiers Condition="'$(RunningPublish)' != 'true' and '$(DotNetBuildFromSource)' == 'true'">$(PackageRID)</RuntimeIdentifiers>
     <SelfContained>false</SelfContained>
     <SelfContained Condition="'$(RunningPublish)' == 'true'">true</SelfContained>
   </PropertyGroup>

Unfortunately that patch didn't work. Very curious that it insists on the apphost not existing despite the apphost being built.

ayakael · 2022-10-31T20:34:40Z

Is there a subset for only building and packaging apphost? I'm thinking a workaround might be to build apphost, and push it to a local nuget feed via dotnet nuget push, that is then referenced by NuGet.conf for it to be pulled full the full build. -subset Clr.Native+Host.Native builds Apphost but does not package it.

ayakael · 2022-10-31T20:54:16Z

sigh -subset Host packages apphost, but despite that crossgen2 refuses to pull runtime.linux-musl-x86.Microsoft.NETCore.DotNetAppHost.7.0.0-rtm.22511.4.nupkg from the local NuGet feed.

am11 · 2022-11-01T08:33:20Z

I cannot test linux-musl-x86 atm as we don't have a prereq docker image handy (and build-rootfs is lacking its support too).

Does source-build fail for linux-x86 too, as it does for linux-musl-x86? asking because the regular cross build for linux-x86 has no issue:

$ git clone https://github.com/dotnet/runtime -b release/7.0 --single-branch --depth 1

$ docker run -v$(pwd)/runtime:/runtime -e ROOTFS_DIR=/crossrootfs/x86 --rm \
    mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-cross-x86-linux-20221024215143-3fc5553 \
    /runtime/build.sh clr+libs+host+packs -arch x86 -cross

__DistroRid: linux-x86
... a few minutes later
Build succeeded.

ayakael · 2022-11-01T14:03:38Z

I cannot test linux-musl-x86 atm as we don't have a prereq docker image handy (and build-rootfs is lacking its support too).

Does source-build fail for linux-x86 too, as it does for linux-musl-x86? asking because the regular cross build for linux-x86 has no issue:
$ git clone https://github.com/dotnet/runtime -b release/7.0 --single-branch --depth 1

$ docker run -v$(pwd)/runtime:/runtime -e ROOTFS_DIR=/crossrootfs/x86 --rm \
    mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-cross-x86-linux-20221024215143-3fc5553 \
    /runtime/build.sh clr+libs+host+packs -arch x86 -cross

__DistroRid: linux-x86
... a few minutes later
Build succeeded.

I could try to figure out a linux-x86 environment. I suspect it worked for you as you were not building with DotNetBuildFromSource=true, thus crossgen would not look for apphost given build with NativeAOT. The reason we build with that flag on Alpine is to weed out issues that would likely appear in source-build (like this one). I'm trying with DotNetBuildFromSource=false on linux-musl-x86 to confirm this.

am11 · 2022-11-01T14:17:35Z

thus crossgen would not look for apphost given build with NativeAOT

NativeAOT doesn't support linux-x86, it builds with apphost (singlefilehost to be specific) when NativeAOT does not support the target platform.

ayakael · 2022-11-01T14:22:49Z

clr+libs+host+packs

Interesting, I didn't know that x86 uses singlefilehost. Indeed, building with DotNetBuildFromSource=false yields a successful build on linux-musl-x86. This confirms this as being principally a source-build issue.

ayakael · 2022-11-01T15:31:43Z

I workaround for now, I figure, is to build a Mono flavored runtime as to avoid building crossgen. Seems kind of like a big roundabout for what seems like a bug that's likely due to a misplaced if statement somewhere. Alas, I also want to check if Mono will hit RAM limits seen by other people when trying to build x86 from source.

ayakael · 2022-11-04T22:37:59Z

Forcing build of Mono-flavored runtime fails with:

  [100%] Linking C shared library libcoreclr.so
  /usr/bin/ld: skipping incompatible /var/build/sysroot-x86//usr/lib/libssp_nonshared.a when searching for -lssp_nonshared
  /usr/bin/ld: skipping incompatible /var/build/sysroot-x86//usr/lib/libgcc_s.so.1 when searching for libgcc_s.so.1
  /usr/bin/ld: skipping incompatible /var/build/sysroot-x86//usr/lib/libgcc_s.so.1 when searching for libgcc_s.so.1
  /usr/bin/ld: skipping incompatible /var/build/sysroot-x86//usr/lib/libc.so when searching for -lc
  /usr/bin/ld: skipping incompatible /var/build/sysroot-x86//usr/lib/libc.a when searching for -lc
  /usr/bin/ld: skipping incompatible /var/build/sysroot-x86//usr/lib/libgcc_s.so.1 when searching for libgcc_s.so.1
  /usr/bin/ld: skipping incompatible /var/build/sysroot-x86//usr/lib/libgcc_s.so.1 when searching for libgcc_s.so.1
  [100%] Built target monosgen-static
  [100%] Building C object mono/mini/CMakeFiles/mono-sgen.dir/main.c.o
  [100%] Linking C executable mono-sgen
  /usr/bin/ld: skipping incompatible /var/build/sysroot-x86//usr/lib/libssp_nonshared.a when searching for -lssp_nonshared
  /usr/bin/ld: skipping incompatible /var/build/sysroot-x86//usr/lib/libgcc_s.so.1 when searching for libgcc_s.so.1
  /usr/bin/ld: skipping incompatible /var/build/sysroot-x86//usr/lib/libgcc_s.so.1 when searching for libgcc_s.so.1
  [100%] Built target monosgen-shared
  /usr/bin/ld: skipping incompatible /var/build/sysroot-x86//usr/lib/libc.so when searching for -lc
  /usr/bin/ld: skipping incompatible /var/build/sysroot-x86//usr/lib/libc.a when searching for -lc
  /usr/bin/ld: skipping incompatible /var/build/sysroot-x86//usr/lib/libgcc_s.so.1 when searching for libgcc_s.so.1
  /usr/bin/ld: skipping incompatible /var/build/sysroot-x86//usr/lib/libgcc_s.so.1 when searching for libgcc_s.so.1
  /usr/bin/ld: libmonosgen-2.0.a(pal_icushim.c.o): in function `GlobalizationNative_LoadICU':
  pal_icushim.c:(.text+0x8f): undefined reference to `__dlsym_time64'
  /usr/bin/ld: pal_icushim.c:(.text+0xc9): undefined reference to `__dlsym_time64'
  /usr/bin/ld: pal_icushim.c:(.text+0x103): undefined reference to `__dlsym_time64'
  /usr/bin/ld: pal_icushim.c:(.text+0x13d): undefined reference to `__dlsym_time64'
  /usr/bin/ld: pal_icushim.c:(.text+0x177): undefined reference to `__dlsym_time64'
  /usr/bin/ld: libmonosgen-2.0.a(pal_icushim.c.o):pal_icushim.c:(.text+0x1b1): more undefined references to `__dlsym_time64' follow
  clang-15: error: linker command failed with exit code 1 (use -v to see invocation)
  make[2]: *** [mono/mini/CMakeFiles/mono-sgen.dir/build.make:194: mono/mini/mono-sgen] Error 1
  make[1]: *** [CMakeFiles/Makefile2:660: mono/mini/CMakeFiles/mono-sgen.dir/all] Error 2
  make: *** [Makefile:136: all] Error 2

Likely a toolchain.cmake issue, as we had to force a sysroot-based crossbuild as Alpine doesn't support -m32compilation mode very well.

ayakael · 2023-03-15T15:30:41Z

This doesn't seem to be an issue anymore. I've crossbuilt version 7.0.102 to 8.0.100-preview.1 without having to deal with this.

I do have an issue with the build artifacts, though. I'm getting an execution failure when trying to run the resulting SDK. Running dotnet --info with the following SDK in linux-musl-x86 chroot yields a freeze, and then eventually a Failed to create CoreCLR, HRESULT: 0x8007000E. It seems like CoreCLR tries to allocate all of the available memory for some reason. @am11 I see that you got the build-rootfs.sh script working for linux-musl-x86, have you managed a successful crossbuild to linux-musl-x86?

I tried stracing it, but could only attach the process after execution.
strace.txt

Edit: The following patchset is used to build the SDK: https://lab.ilot.io/ayakael/dotnet-stage0

am11 · 2023-03-15T18:59:32Z

@ayakael, I have only tried building runtime, didn't test the binaries.

Running dotnet --info with the following SDK

I don't have permission to download it. :)

Failed to create CoreCLR, HRESULT: 0x8007000E

BTW, were you testing under VM / baremetal install or QEMU? CoreCLR-PAL has some issues with QEMU (even on 64-bit archs).

ayakael · 2023-03-15T23:16:35Z

Woops, here is a functional link: dotnet-sdk-8.0.100-preview.2.23157.25-r1-linux-musl-x86.tar.xz

I tested first within a chroot of alpine-x86 that was in LXC, and then within a chroot in a qemu based env. Both behave the same.

am11 · 2023-03-16T11:36:29Z

Opened #83509 for the different issue.

This doesn't seem to be an issue anymore. I've crossbuilt version 7.0.102 to 8.0.100-preview.1 without having to deal with this.

Good to know that this issue no longer repros. Closing.

ghost added the untriaged New issue has not been triaged by the area owner label Oct 31, 2022

am11 added the area-Infrastructure-installer label Oct 31, 2022

agocke added this to the Future milestone Oct 31, 2022

agocke added area-Infrastructure-coreclr and removed untriaged New issue has not been triaged by the area owner area-Infrastructure-installer labels Oct 31, 2022

am11 mentioned this issue Mar 16, 2023

Failed to create CoreCLR, HRESULT: 0x8007000E on linux-musl-x86 #83509

Open

am11 closed this as completed Mar 16, 2023

ghost locked as resolved and limited conversation to collaborators Apr 15, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Live apphost not being used by crossgen2 with source build #77667

Live apphost not being used by crossgen2 with source build #77667

ayakael commented Oct 31, 2022

dotnet-issue-labeler bot commented Oct 31, 2022

am11 commented Oct 31, 2022

ayakael commented Oct 31, 2022

am11 commented Oct 31, 2022

agocke commented Oct 31, 2022

ghost commented Oct 31, 2022

Description

Reproduction Steps

Expected behavior

Actual behavior

Regression?

Known Workarounds

Configuration

Other information

ayakael commented Oct 31, 2022 •

edited

Loading

ayakael commented Oct 31, 2022

ayakael commented Oct 31, 2022 •

edited

Loading

am11 commented Nov 1, 2022

ayakael commented Nov 1, 2022

am11 commented Nov 1, 2022

ayakael commented Nov 1, 2022

ayakael commented Nov 1, 2022 •

edited

Loading

ayakael commented Nov 4, 2022

ayakael commented Mar 15, 2023 •

edited

Loading

am11 commented Mar 15, 2023

ayakael commented Mar 15, 2023

am11 commented Mar 16, 2023

Live apphost not being used by crossgen2 with source build #77667

Live apphost not being used by crossgen2 with source build #77667

Comments

ayakael commented Oct 31, 2022

Description

Reproduction Steps

Expected behavior

Actual behavior

Regression?

Known Workarounds

Configuration

Other information

dotnet-issue-labeler bot commented Oct 31, 2022

am11 commented Oct 31, 2022

ayakael commented Oct 31, 2022

am11 commented Oct 31, 2022

agocke commented Oct 31, 2022

ghost commented Oct 31, 2022

Description

Reproduction Steps

Expected behavior

Actual behavior

Regression?

Known Workarounds

Configuration

Other information

ayakael commented Oct 31, 2022 • edited Loading

ayakael commented Oct 31, 2022

ayakael commented Oct 31, 2022 • edited Loading

am11 commented Nov 1, 2022

ayakael commented Nov 1, 2022

am11 commented Nov 1, 2022

ayakael commented Nov 1, 2022

ayakael commented Nov 1, 2022 • edited Loading

ayakael commented Nov 4, 2022

ayakael commented Mar 15, 2023 • edited Loading

am11 commented Mar 15, 2023

ayakael commented Mar 15, 2023

am11 commented Mar 16, 2023

ayakael commented Oct 31, 2022 •

edited

Loading

ayakael commented Oct 31, 2022 •

edited

Loading

ayakael commented Nov 1, 2022 •

edited

Loading

ayakael commented Mar 15, 2023 •

edited

Loading