Updating SDK version causes dotnet-install failure for aspnetcore repo

[andriipatsula]

• This issue is blocking
• This issue is causing unreasonable pain

Upgrade from 7.0.100-rc.1.22375.2 to 7.0.100-rc.1.22403.8

InstallDotNetCore.targets(15,5): error : dotnet-install failed [/__w/1/s/artifacts/source-build/self/package-cache/microsoft.dotnet.arcade.sdk/7.0.0-beta.22379.10/tools/Tools.proj]

original PR
link to MS Teams discussion

2 pipelines are failing: source-build & aspnetcore-components-e2e and other green.

• https://dev.azure.com/dnceng/public/_build/results?buildId=1922815&view=results
• https://dev.azure.com/dnceng/public/_build/results?buildId=1922816&view=results

[andriipatsula]

@MichaelSimons , you have tagged @dotnet/source-build-internal in PR, link. Do you know if someone looked at the problem?

[MichaelSimons]

No I don't think anyone from source-build has taken a look. I just took a peak at the error message and at first glance it doesn't appear source-build specific.

[andriipatsula]

/source-build/self/src/eng/SourceBuild.props(60,5): error MSB3073: The command "./eng/build.sh --only-build-repo-tasks -bl " exited with code 1.

it is failing on SourceBuild.props(60, 5) and there is nothing interesting in msbuild bin logs. what steps could we perform?

[MichaelSimons]

Earlier in the build output there is this -

/__w/1/s/artifacts/source-build/self/package-cache/microsoft.dotnet.arcade.sdk/7.0.0-beta.22379.10/tools/InstallDotNetCore.targets(15,5): error : dotnet-install failed [/__w/1/s/artifacts/source-build/self/package-cache/microsoft.dotnet.arcade.sdk/7.0.0-beta.22379.10/tools/Tools.proj]
    Build failed with exit code 1. Check errors above.

[riarenas]

So it looks like both problems are the same. Installation for that preview sdk is failing in two legs running in ubuntu 18.04.

Is there something known about this particular SDK and problems to install in that OS? @marcpopMSFT might know / be interested from the SDK side, and @YuliiaKovalova / @bekir-ozturk might be interested from the scripts side

EDIT: It looks to be a bug in the task as per following comments

[riarenas]

For whoever picks this up in FR, we should investigate why the

arcade/src/Microsoft.DotNet.Arcade.Sdk/src/InstallDotNetCore.cs

Lines 144 to 147 in be70926

	if (process.ExitCode != 0)
	{
	Log.LogError("dotnet-install failed");
	}

task is failing, even after the scripts are saying that the SDK was installed successfully:

https://dev.azure.com/dnceng/public/_build/results?buildId=1922816&view=logs&j=fe94c0c9-bb8c-5d6f-3b51-887173cc2f5c&t=7877c2d9-a5b1-59a3-dc2b-0d7e9d9db055&l=34

[riarenas]

@MattGal is looking at this and found:

when you turn up the verbosity it logs this exception

    
 The "InstallDotNetCore" task failed unexpectedly. [/aspnetcore/artifacts/source-build/self/package-cache/microsoft.dotnet.arcade.sdk/7.0.0-beta.22379.10/tools/Tools.proj] (TaskId:2) (TaskId:6)System.InvalidOperationException: An equivalent project (a project with the same global properties and tools version) is already present in the project collection, with the path "/aspnetcore/artifacts/source-build/self/src/eng/Versions.props". To load an equivalent into this project collection, unload this project first. [/aspnetcore/artifacts/source-build/self/package-cache/microsoft.dotnet.arcade.sdk/7.0.0-beta.22379.10/tools/Tools.proj] (TaskId:2) (TaskId:6)
   at Microsoft.Build.Shared.ErrorUtilities.ThrowInvalidOperation(String resourceName, Object[] args) [/aspnetcore/artifacts/source-build/self/package-cache/microsoft.dotnet.arcade.sdk/7.0.0-beta.22379.10/tools/Tools.proj] (TaskId:2) (TaskId:6)
   at Microsoft.Build.Evaluation.ProjectCollection.LoadedProjectCollection.AddProject(Project project) [/aspnetcore/artifacts/source-build/self/package-cache/microsoft.dotnet.arcade.sdk/7.0.0-beta.22379.10/tools/Tools.proj] (TaskId:2) (TaskId:6)
   at Microsoft.Build.Evaluation.ProjectCollection.OnAfterRenameLoadedProject(String oldFullPathIfAny, Project project) [/aspnetcore/artifacts/source-build/self/package-cache/microsoft.dotnet.arcade.sdk/7.0.0-beta.22379.10/tools/Tools.proj] (TaskId:2) (TaskId:6)
   at Microsoft.Build.Evaluation.Project.ProjectImpl.Initialize(IDictionary`2 globalProperties, String toolsVersion, String subToolsetVersion, ProjectLoadSettings loadSettings, EvaluationContext evaluationContext) [/aspnetcore/artifacts/source-build/self/package-cache/microsoft.dotnet.arcade.sdk/7.0.0-beta.22379.10/tools/Tools.proj] (TaskId:2) (TaskId:6)
   at Microsoft.Build.Evaluation.Project..ctor(String projectFile, IDictionary`2 globalProperties, String toolsVersion, String subToolsetVersion, ProjectCollection projectCollection, ProjectLoadSettings loadSettings, EvaluationContext evaluationContext, IDirectoryCacheFactory directoryCacheFactory) [/aspnetcore/artifacts/source-build/self/package-cache/microsoft.dotnet.arcade.sdk/7.0.0-beta.22379.10/tools/Tools.proj] (TaskId:2) (TaskId:6)
   at Microsoft.Build.Evaluation.Project.FromFile(String file, ProjectOptions options) [/aspnetcore/artifacts/source-build/self/package-cache/microsoft.dotnet.arcade.sdk/7.0.0-beta.22379.10/tools/Tools.proj] (TaskId:2) (TaskId:6)
   at Microsoft.DotNet.Arcade.Sdk.InstallDotNetCore.Execute() in /_/src/Microsoft.DotNet.Arcade.Sdk/src/InstallDotNetCore.cs:line 86 [/aspnetcore/artifacts/source-build/self/package-cache/microsoft.dotnet.arcade.sdk/7.0.0-beta.22379.10/tools/Tools.proj] (TaskId:2) (TaskId:6)
   at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute() [/aspnetcore/artifacts/source-build/self/package-cache/microsoft.dotnet.arcade.sdk/7.0.0-beta.22379.10/tools/Tools.proj] (TaskId:2) (TaskId:6)
   at Microsoft.Build.BackEnd.TaskBuilder.ExecuteInstantiatedTask(ITaskExecutionHost taskExecutionHost, TaskLoggingContext taskLoggingContext, TaskHost taskHost, ItemBucket bucket, TaskExecutionMode howToExecuteTask) [/aspnetcore/artifacts/source-build/self/package-cache/microsoft.dotnet.arcade.sdk/7.0.0-beta.22379.10/tools/Tools.proj] (TaskId:2) (TaskId:6

[MattGal]

@rainersigwald and I are going to look at this today... it should be possible to non-destructively load a "duplicate" project with different APIs, it's just hard to try out changes inside linux docker.

[riarenas]

Is the aspnetcore-components-e2e build also running in docker? Perhaps that's an easier repro to iterate on?

https://dev.azure.com/dnceng/public/_build/results?buildId=1922816&view=logs&j=fe94c0c9-bb8c-5d6f-3b51-887173cc2f5c&t=57cf1669-6335-4911-bd22-363246b5c623

[MattGal]

Yup I'll try that too, I'm just in a meeting.

[MattGal]

I think https://github.com/dotnet/arcade/pull/10318/files will do it, I just made a mistake while testing and logged a warning, which due to the nature of source build is very hard to suppress. I'll try the other build Ricardo's recommending.

[MattGal]

@lbussell helped me with the testing and we think that #10318 should address it once merged.

[MattGal]

#10318 didn't help, I am back on this.

[MattGal]

I spent a bunch of time on this today and made minimal good progress. I ended up filing dotnet/msbuild#7870, as I am fairly confident at this point that this is a regression in MSBuild functionality included in the .NET Core SDK and I have a fairly self-contained repro.

[MattGal]

After isolating this to msbuild and filing a bug, it turns out to be dotnet/sdk#26965, which is fixed by dotnet/msbuild#7860. I verified this using the DOTNET_CLI_DO_NOT_USE_MSBUILD_SERVER suggested by the MSBuild team.

Adding DOTNET_CLI_DO_NOT_USE_MSBUILD_SERVER=true to the build can unblock it, but you probably don't want to take an SDK with this bug so unless there's a very pressing need, I would advise just waiting for the next SDK that has this change in it before updating.

[MattGal]

As it's now clear this is an MSBuild bug in this SDK (and thus not one users want to adopt) I am closing this issue.

[dougbu]

@rainersigwald do you have a commit we should be hunting for in https://maestro-prod.westus2.cloudapp.azure.com/2237/https:%2F%2Fgithub.com%2Fdotnet%2Fsdk/latest/graph to avoid needing the DOTNET_CLI_DO_NOT_USE_MSBUILD_SERVER workaround❔

SDK is currently pulling in a msbuild build that's 5 days old and contains commit https://github.com/dotnet/msbuild/commits/2db11c256ade886f673ed56d12780fb70e6ef92e. Wondering if that's enough❔

[MattGal]

@rainersigwald do you have a commit we should be hunting for in https://maestro-prod.westus2.cloudapp.azure.com/2237/https:%2F%2Fgithub.com%2Fdotnet%2Fsdk/latest/graph to avoid needing the DOTNET_CLI_DO_NOT_USE_MSBUILD_SERVER workaround❔

SDK is currently pulling in a msbuild build that's 5 days old and contains commit https://github.com/dotnet/msbuild/commits/2db11c256ade886f673ed56d12780fb70e6ef92e. Wondering if that's enough❔

Based off my experiments, depending on whether or not you also do the behavior that triggers the bug, taking an arcade update featuring the change in #10629 may fully remove the need to make this change (and is included in the 7.0 release branch of arcade)