MSB4242: SDK Resolver Failure due to System.Threading.AbandonedMutexException

[breidikl]

Describe the bug

Updating our build environment to use .NET 6 SDK 6.0.402 (from 6.0.401) and the associated 6.0.10 runtime packages is now resulting in intermittent failures as follows during dotnet restore calls:

Name.csproj : error MSB4242: SDK Resolver Failure: "The SDK resolver "NuGetSdkResolver" failed while attempting to resolve the SDK "Office.Build.Before". Exception: "System.Threading.AbandonedMutexException: The wait completed due to an abandoned mutex.
Name.csproj : error MSB4242: at System.Threading.WaitHandle.WaitOneNoCheck(Int32 millisecondsTimeout)
Name.csproj : error MSB4242: at System.Threading.WaitHandle.WaitOne(TimeSpan timeout, Boolean exitContext)
Name.csproj : error MSB4242: at NuGet.Common.Migrations.MigrationRunner.Run()
Name.csproj : error MSB4242: at Microsoft.Build.NuGetSdkResolver.NuGetSdkResolver.Resolve(SdkReference sdkReference, SdkResolverContext resolverContext, SdkResultFactory factory)
Name.csproj : error MSB4242: at Microsoft.Build.BackEnd.SdkResolution.SdkResolverService.TryResolveSdkUsingSpecifiedResolvers(IList`1 resolvers, Int32 submissionId, SdkReference sdk, LoggingContext loggingContext, ElementLocation sdkReferenceLocation, String solutionPath, String projectPath, Boolean interactive, Boolean isRunningInVisualStudio, SdkResult& sdkResult)"

"Name" varies build to build and appears to be a random set of csproj files being restored for each build. We have not been able to repro when rebuilding the same projects locally for the same environment.

Using 6.0.401 (with the 6.0.9 runtime packages) did not hit this with everything else in our environment being the same.

Further technical details

We are currently using MSBuild 17.3.4, and running "dotnet --info" shows the following:

.NET SDK (reflecting any global.json):
Version: 6.0.402
Commit: 6862418

Runtime Environment:
OS Name: Windows
OS Version: 10.0.22621
OS Platform: Windows
RID: win10-x64
Base Path: dotnetsdkbootstrapper\sdk\sdk\6.0.402\

global.json file:
\global.json

Host:
Version: 6.0.10
Architecture: x64
Commit: 5a400c212a

.NET SDKs installed:
3.1.424 [dotnetsdkbootstrapper\sdk\sdk]
5.0.408 [dotnetsdkbootstrapper\sdk\sdk]
6.0.402 [dotnetsdkbootstrapper\sdk\sdk]

.NET runtimes installed:
Microsoft.AspNetCore.App 3.1.30 [dotnetsdkbootstrapper\sdk\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 5.0.17 [dotnetsdkbootstrapper\sdk\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 6.0.10 [dotnetsdkbootstrapper\sdk\shared\Microsoft.AspNetCore.App]
Microsoft.NETCore.App 3.1.30 [dotnetsdkbootstrapper\sdk\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 5.0.17 [dotnetsdkbootstrapper\sdk\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 6.0.10 [dotnetsdkbootstrapper\sdk\shared\Microsoft.NETCore.App]
Microsoft.WindowsDesktop.App 3.1.30 [dotnetsdkbootstrapper\sdk\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 5.0.17 [dotnetsdkbootstrapper\sdk\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 6.0.10 [dotnetsdkbootstrapper\sdk\shared\Microsoft.WindowsDesktop.App]

[dfederm]

Likely related to: NuGet/NuGet.Client@3c1bf9d by @kartheekp-ms

[dfederm]

I believe the solution is to add somethign like this:

            catch (AbandonedMutexException)
            {
                // Abandoned by another process, we acquired it.
            }

AbandonedMutexException is seemingly thrown even in some success cases.

From: https://learn.microsoft.com/en-us/dotnet/api/system.threading.mutex?view=net-6.0#remarks

If a thread terminates while owning a mutex, the mutex is said to be abandoned. The state of the mutex is set to signaled, and the next waiting thread gets ownership. Beginning in version 2.0 of the .NET Framework, an AbandonedMutexException is thrown in the next thread that acquires the abandoned mutex. Before version 2.0 of the .NET Framework, no exception was thrown.

In the case of a system-wide mutex, an abandoned mutex might indicate that an application has been terminated abruptly (for example, by using Windows Task Manager).

[marcpopMSFT]

@jeffkl

[jeffkl]

Looks like a bug in our new MigrationRunner code, I will follow up with my team.

[kartheekp-ms]

NuGet side tracking bug NuGet/Home#12159

[kartheekp-ms]

@breidikl - Sorry for the inconvenience caused by this bug.
As a workaround, would it be possible to execute dotnet nuget --version command before running the restore. In that way we don't hit the race condition that is causing these failures intermittently.

[marcpopMSFT]

This issue will be tracked with the nuget side issue 12159. Closing for now.