Building net*-android apps always restore Microsoft.NETCore.App.Host.linux

[jonathanpeppers]

Describe the bug

> dotnet new android
> dotnet restore

Appears to always download:

This is related to: #19891

Since these are on NuGet.org, everything works.

However, we started noticing this with net7.0-android apps:

Since these are not on NuGet, you get errors like:

Errors
    C:\src\foo\foo.csproj error NU1102: Unable to find package Microsoft.NETCore.App.Host.linux-arm with version (= 7.0.0-preview.5.22271.4)
  - Found 88 version(s) in nuget.org [ Nearest version: 7.0.0-preview.4.22229.4 ]
  - Found 0 version(s) in Microsoft Visual Studio Offline Packages [C:\src\foo\foo.csproj]
    C:\src\foo\foo.csproj error NU1102: Unable to find package Microsoft.NETCore.App.Host.linux-arm64 with version (= 7.0.0-preview.5.22271.4)
  - Found 88 version(s) in nuget.org [ Nearest version: 7.0.0-preview.4.22229.4 ]
  - Found 0 version(s) in Microsoft Visual Studio Offline Packages [C:\src\foo\foo.csproj]
    C:\src\foo\foo.csproj error NU1102: Unable to find package Microsoft.NETCore.App.Host.linux-x64 with version (= 7.0.0-preview.5.22271.4)
  - Found 88 version(s) in nuget.org [ Nearest version: 7.0.0-preview.4.22229.4 ]
  - Found 0 version(s) in Microsoft Visual Studio Offline Packages [C:\src\foo\foo.csproj]

And so anyone dogfooding .NET 7 would have to add a nuget.config:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <packageSources>
    <add key="dotnet7" value="https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet7/nuget/v3/index.json" />
  </packageSources>
</configuration>

To Reproduce

1. Install the .NET 7 android workload
2. dotnet new android
3. dotnet restore

Exceptions (if any)

Binlog: msbuild.zip

Further technical details

.NET SDK:
 Version:   7.0.100-preview.5.22273.1
 Commit:    a3acdcf34e

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.19044
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\src\xamarin-android\bin\Release\dotnet\sdk\7.0.100-preview.5.22273.1\

global.json file:
  Not found

Host:
  Version:      7.0.0-preview.5.22271.4
  Architecture: x64
  Commit:       621cd59436

.NET SDKs installed:
  7.0.100-preview.5.22273.1 [C:\src\xamarin-android\bin\Release\dotnet\sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 7.0.0-preview.5.22272.3 [C:\src\xamarin-android\bin\Release\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 7.0.0-preview.5.22271.4 [C:\src\xamarin-android\bin\Release\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 7.0.0-preview.5.22269.1 [C:\src\xamarin-android\bin\Release\dotnet\shared\Microsoft.WindowsDesktop.App]

[dotnet-issue-labeler]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[marcpopMSFT]

@dsplaisted I believe you were looking into this before with Jonathan.

[jonathanpeppers]

We had a new version of this appear in: dotnet/android#7071

(Restore target) -> 
    App1.csproj : error NU1101: Unable to find package Microsoft.NETCore.App.Host.linux-bionic-arm. No packages exist with this id in source(s): dotnet-eng, dotnet-public, xamarin.android util
    App1.csproj : error NU1101: Unable to find package Microsoft.NETCore.App.Host.linux-bionic-arm64. No packages exist with this id in source(s): dotnet-eng, dotnet-public, xamarin.android util
    App1.csproj : error NU1101: Unable to find package Microsoft.NETCore.App.Host.linux-bionic-x86. No packages exist with this id in source(s): dotnet-eng, dotnet-public, xamarin.android util
    App1.csproj : error NU1101: Unable to find package Microsoft.NETCore.App.Host.linux-bionic-x64. No packages exist with this id in source(s): dotnet-eng, dotnet-public, xamarin.android util

[marcpopMSFT]

@jonathanpeppers does setting DisableTransitiveFrameworkReferenceDownloads per #25358 work around this issue now or is it separate?

[LoopedBard3]

Trying /p:DisableTransitiveFrameworkReferenceDownloads=true out locally, does not seem to fix the problem when building Maui targeting android-arm64.

[jonathanpeppers]

I think there might be a couple different things going on here:

1. linux-bionic was added to the RID graph, somehow android inherits from it. This might be correct, because "bionic" is some subset of libc, if I understand correctly?
2. linux-bionic-arm and linux-bionic-x86 do not actually exist, only the 64-bit versions exist

@directhex was going to look at the RIDs and/or the packages. Did we come up with something there? Maybe we should delete the 32-bit ones that shouldn't exist?

[directhex]

bionic is the Android C library, equivalent to glibc on Linux. linux-bionic is basically a RID for "Linux, but using Bionic libc instead of GNU libc". It was created for an internal customer, who only wanted 64-bit versions - but I was told we should have the 32-bit versions in the RID graph too so android-x86 and android-x64 would have matching dependency graphs, not android-x86->linux-x86.

The bug here is the needless restore. The existence or lack thereof of a runtime pack for linux-bionic-x86 should not be affecting Android apps. We need to figure out why that dependency is being considered as needed by the SDK.

[LoopedBard3]

I am not super familiar with how the SDK uses the RID Graph, although it looks like linux-bionic* are included as imports for the respective RID's based on the RuntimeIdentifierGraph.json file used when building the Maui app:

    "android": {
      "#import": [
        "linux-bionic"
      ]
    },
    "android-arm": {
      "#import": [
        "android",
        "linux-bionic-arm"
      ]
    },
    "android-arm64": {
      "#import": [
        "android",
        "linux-bionic-arm64"
      ]
    },
    "android-x64": {
      "#import": [
        "android",
        "linux-bionic-x64"
      ]
    },
    "android-x86": {
      "#import": [
        "android",
        "linux-bionic-x86"
      ]
    },

Are these imports not treated as download/install requirements? Or is there a different place I should be looking to track down this bug? Looking at dotnet/runtime@b5f9c22#diff-4cae40f80b790c66e01955edbd41e3ea51cec84088d1cbf6b461a5be8daee777R279-R307, it looks like android was previously importing linux, would changing it back to that fix the issue? Thanks!

[directhex]

The RID graph should only be traversed like that when it comes to calculating NuGet fallbacks - i.e. "if there's no .so for android in this package, try the .so for linux-bionic, followed by the .so for Linux"

[steveisok]

@dsplaisted pointed to #22658 where we can exclude certain rids from the task that tries to download hosting packs.

[marcpopMSFT]

Change went into installer for 6.0.400 already and flowing to main here: dotnet/installer#14288

[stulda]

I have this issue newly on my m1 macbook after sdk 6.0.400 update.

.NET SDK (reflecting any global.json):
Version: 6.0.400
Commit: 7771abd

Runtime Environment:
OS Name: Mac OS X
OS Version: 12.5
OS Platform: Darwin
RID: osx.12-arm64
Base Path: /usr/local/share/dotnet/sdk/6.0.400/

global.json file:
Not found

Host:
Version: 6.0.8
Architecture: arm64
Commit: 55fb7ef977

/Users/stulda/Temp/test/test.csproj : error NU1102: Unable to find package Microsoft.NETCore.App.Host.linux-arm with version (= 6.0.8)
/Users/stulda/Temp/test/test.csproj : error NU1102:   - Found 94 version(s) in nuget.org [ Nearest version: 7.0.0-preview.1.22076.8 ]
/Users/stulda/Temp/test/test.csproj : error NU1102:   - Found 0 version(s) in /usr/local/share/dotnet/library-packs
/Users/stulda/Temp/test/test.csproj : error NU1102: Unable to find package Microsoft.NETCore.App.Host.linux-arm64 with version (= 6.0.8)
/Users/stulda/Temp/test/test.csproj : error NU1102:   - Found 94 version(s) in nuget.org [ Nearest version: 7.0.0-preview.1.22076.8 ]
/Users/stulda/Temp/test/test.csproj : error NU1102:   - Found 0 version(s) in /usr/local/share/dotnet/library-packs
/Users/stulda/Temp/test/test.csproj : error NU1102: Unable to find package Microsoft.NETCore.App.Host.linux-x64 with version (= 6.0.8)
/Users/stulda/Temp/test/test.csproj : error NU1102:   - Found 94 version(s) in nuget.org [ Nearest version: 7.0.0-preview.1.22076.8 ]
/Users/stulda/Temp/test/test.csproj : error NU1102:   - Found 0 version(s) in /usr/local/share/dotnet/library-packs

[jonathanpeppers]

These are on NuGet.org: https://www.nuget.org/packages/Microsoft.NETCore.App.Host.linux-arm/6.0.8

They shipped yesterday, I think, so maybe it just works now?

[stulda]

No change

I also tried this:

dotnet add package Microsoft.NETCore.App.Host.linux-arm --version 6.0.8

and here is what i got:

dotnet add package Microsoft.NETCore.App.Host.linux-arm --version 6.0.8
  Determining projects to restore...
  Writing /var/folders/m9/l6lwz36j0wxbc2lxybg9htkr0000gn/T/tmp2R0UrT.tmp
info : X.509 certificate chain validation will use the fallback certificate bundle at '/usr/local/share/dotnet/sdk/6.0.400/trustedroots/codesignctl.pem'.
info : Adding PackageReference for package 'Microsoft.NETCore.App.Host.linux-arm' into project '/Users/stulda/Temp/test/test.csproj'.
info : Restoring packages for /Users/stulda/Temp/test/test.csproj...
info :   GET https://api.nuget.org/v3-flatcontainer/microsoft.netcore.app.host.linux-arm/index.json
info :   OK https://api.nuget.org/v3-flatcontainer/microsoft.netcore.app.host.linux-arm/index.json 342ms
info :   GET https://api.nuget.org/v3-flatcontainer/microsoft.netcore.app.host.linux-arm64/index.json
info :   GET https://api.nuget.org/v3-flatcontainer/microsoft.netcore.app.host.linux-x64/index.json
info :   OK https://api.nuget.org/v3-flatcontainer/microsoft.netcore.app.host.linux-arm64/index.json 264ms
info :   OK https://api.nuget.org/v3-flatcontainer/microsoft.netcore.app.host.linux-x64/index.json 358ms
error: NU1103: Unable to find a stable package Microsoft.NETCore.App.Host.linux-arm with version (>= 6.0.8)
error:   - Found 94 version(s) in nuget.org [ Nearest version: 7.0.0-preview.1.22076.8 ]
error:   - Found 0 version(s) in /usr/local/share/dotnet/library-packs
error: NU1102: Unable to find package Microsoft.NETCore.App.Host.linux-arm with version (= 6.0.8)
error:   - Found 94 version(s) in nuget.org [ Nearest version: 7.0.0-preview.1.22076.8 ]
error:   - Found 0 version(s) in /usr/local/share/dotnet/library-packs
error: NU1102: Unable to find package Microsoft.NETCore.App.Host.linux-arm64 with version (= 6.0.8)
error:   - Found 94 version(s) in nuget.org [ Nearest version: 7.0.0-preview.1.22076.8 ]
error:   - Found 0 version(s) in /usr/local/share/dotnet/library-packs
error: NU1102: Unable to find package Microsoft.NETCore.App.Host.linux-x64 with version (= 6.0.8)
error:   - Found 94 version(s) in nuget.org [ Nearest version: 7.0.0-preview.1.22076.8 ]
error:   - Found 0 version(s) in /usr/local/share/dotnet/library-packs
error: Package 'Microsoft.NETCore.App.Host.linux-arm' is incompatible with 'all' frameworks in project '/Users/stulda/Temp/test/test.csproj'.

[stulda]

Change went into installer for 6.0.400 already and flowing to main here: dotnet/installer#14288

This workaround is missing in 6.0.400 release:
https://github.com/xamarin/xamarin-android/pull/7237/files