Tests fail for Linux local build on latest commits: "The framework 'Microsoft.NETCore.App', version '2.1.0' was not found."

[dagood]

• This issue is blocking
• This issue is causing unreasonable pain

When I try to clone latest arcade and build with tests, it fails. Repro command:

docker run -it --rm mcr.microsoft.com/dotnet/core/sdk:3.1 bash -c '
  git clone https://github.com/dotnet/arcade
  cd arcade
  git checkout 342467b # Latest as of writing, for reproducibility
  ./build.sh -test'

Also repros with the container used in the PR validation build (no global .NET installed):

arcade/azure-pipelines.yml

Lines 20 to 21 in 342467b

	- container: LinuxContainer
	image: microsoft/dotnet-buildtools-prereqs:ubuntu-14.04-cross-0cd4667-20170319080304

docker run -it --rm microsoft/dotnet-buildtools-prereqs:ubuntu-14.04-cross-0cd4667-20170319080304 bash -c '
  git clone https://github.com/dotnet/arcade
  cd arcade
  git checkout 342467b # Latest as of writing, for reproducibility
  ./build.sh -test'

They hit errors at the end of the build:

...
  Running tests: /arcade/artifacts/bin/Microsoft.DotNet.SignTool.Tests/Debug/netcoreapp2.1/Microsoft.DotNet.SignTool.Tests.dll [netcoreapp2.1|x64]
  Running tests: /arcade/artifacts/bin/Microsoft.DotNet.RemoteExecutor.Tests/Debug/netcoreapp2.1/Microsoft.DotNet.RemoteExecutor.Tests.dll [netcoreapp2.1|x64]
  Running tests: /arcade/artifacts/bin/Microsoft.DotNet.Build.Tasks.Feed.Tests/Debug/netcoreapp2.1/Microsoft.DotNet.Build.Tasks.Feed.Tests.dll [netcoreapp2.1|x64]
XUnit : error : Tests failed: /arcade/artifacts/log/Debug/Microsoft.DotNet.Build.Tasks.Feed.Tests_netcoreapp2.1_x64.log [netcoreapp2.1|x64] [/arcade/src/Microsoft.DotNet.Build.Tasks.Feed.Tests/Microsoft.DotNet.Build.Tasks.Feed.Tests.csproj]
XUnit : error : Tests failed: /arcade/artifacts/log/Debug/Microsoft.DotNet.SignTool.Tests_netcoreapp2.1_x64.log [netcoreapp2.1|x64] [/arcade/src/Microsoft.DotNet.SignTool.Tests/Microsoft.DotNet.SignTool.Tests.csproj]

...

$ cat artifacts/log/Debug/Microsoft.DotNet.Build.Tasks.Feed.Tests_netcoreapp2.1_x64.log
It was not possible to find any compatible framework version
The framework 'Microsoft.NETCore.App', version '2.1.0' was not found.
  - The following frameworks were found:
      5.0.0-preview.6.20305.6 at [/arcade/.dotnet/shared/Microsoft.NETCore.App]

You can resolve the problem by installing the specified framework and/or SDK.

The specified framework can be found at:
  - https://aka.ms/dotnet-core-applaunch?framework=Microsoft.NETCore.App&framework_version=2.1.0&arch=x64&rid=ubuntu.14.04-x64
=== COMMAND LINE ===
"/arcade/.dotnet/dotnet" exec --depsfile "/arcade/artifacts/bin/Microsoft.DotNet.Build.Tasks.Feed.Tests/Debug/netcoreapp2.1/Microsoft.DotNet.Build.Tasks.Feed.Tests.deps.json" --runtimeconfig "/arcade/artifacts/bin/Microsoft.DotNet.Build.Tasks.Feed.Tests/Debug/netcoreapp2.1/Microsoft.DotNet.Build.Tasks.Feed.Tests.runtimeconfig.json"  "/root/.nuget/packages/xunit.runner.console/2.4.1/tools/netcoreapp2.0/xunit.console.dll" "/arcade/artifacts/bin/Microsoft.DotNet.Build.Tasks.Feed.Tests/Debug/netcoreapp2.1/Microsoft.DotNet.Build.Tasks.Feed.Tests.dll" -noautoreporters -xml "/arcade/artifacts/TestResults/Debug/Microsoft.DotNet.Build.Tasks.Feed.Tests_netcoreapp2.1_x64.xml" -html "/arcade/artifacts/TestResults/Debug/Microsoft.DotNet.Build.Tasks.Feed.Tests_netcoreapp2.1_x64.html"  > "/arcade/artifacts/log/Debug/Microsoft.DotNet.Build.Tasks.Feed.Tests_netcoreapp2.1_x64.log" 2>&1

[garath]

Triage: @dagood can you explain more about this scenario? We're not quite sure if this is meant to be a supported scenario or not.

FYI @tkapin

[dagood]

I don't know either. But it's very weird that there are tests running and succeeding on Linux in PR validation jobs, but a simple -test fails locally.

[dagood]

I get the same failure on Windows, for what it's worth.

[dagood]

Adding <RollForward>Major</RollForward> is a direct fix for the error I copied here, but then I run into more test failures. I'd like to know if walking up to the repo and running build.{sh/cmd} -test is meant to be supported before trying anything else though.