[BUG]: dotnet format randomly fails with unable to locate dotnet CLI

[mslukebo]

Task name

DotNetCoreCLI@2

Task version

2.221.0

Environment type (Please select at least one enviroment where you face this issue)

• Self-Hosted
• Microsoft Hosted
• VMSS Pool
• Container

Azure DevOps Server type

dev.azure.com (formerly visualstudio.com)

Azure DevOps Server Version (if applicable)

No response

Operation system

windows-latest

Task log

==============================================================================
Task         : .NET Core
Description  : Build, test, package, or publish a dotnet application, or run a custom dotnet command
Version      : 2.221.0
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/build/dotnet-core-cli
==============================================================================
C:\Windows\system32\chcp.com 65001
Active code page: 65001
Info: .NET Core SDK/runtime 2.2 and 3.0 are now End of Life(EOL) and have been removed from all hosted agents. If you're using these SDK/runtimes on hosted agents, kindly upgrade to newer versions which are not EOL, or else use UseDotNet task to install the required version.
C:\hostedtoolcache\windows\dotnet\dotnet.exe format --no-restore --verify-no-changes --verbosity detailed
  The dotnet runtime version is '7.0.12'.
Unable to locate dotnet CLI. Ensure that it is on the PATH.
##[error]Error: The process 'C:\hostedtoolcache\windows\dotnet\dotnet.exe' failed with exit code 4
Info: Azure Pipelines hosted agents have been updated and now contain .Net 5.x SDK/Runtime along with the older .Net Core version which are currently lts. Unless you have locked down a SDK version for your project(s), 5.x SDK might be picked up which might have breaking behavior as compared to previous versions. You can learn more about the breaking changes here: https://docs.microsoft.com/en-us/dotnet/core/tools/ and https://docs.microsoft.com/en-us/dotnet/core/compatibility/ . To learn about more such changes and troubleshoot, refer here: https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/build/dotnet-core-cli?view=azure-devops#troubleshooting
##[error]Dotnet command failed with non-zero exit code on the following projects : [ '' ]
Finishing: Check code style

Relevant log output

Unable to locate dotnet CLI. Ensure that it is on the PATH.
##[error]Error: The process 'C:\hostedtoolcache\windows\dotnet\dotnet.exe' failed with exit code 4

Aditional info

We have a build pipeline with the following steps:

- task: UseDotNet@2
  displayName: Install .NET 6
  inputs:
    packageType: 'sdk'
    version: '6.x'
    performMultiLevelLookup: true

- task: NuGetToolInstaller@1
  displayName: 'Use NuGet 6.x.x'
  inputs:
    versionSpec: 6.x.x

- task: DotNetCoreCLI@2
  displayName: Restore Packages
  inputs:
    command: restore
    projects: 'OUR_SLN'
    verbosityRestore: minimal

- task: DotNetCoreCLI@2
  displayName: 'Build solution'
  inputs:
    command: 'build'
    projects: 'OUR_SLN'
    arguments: '--no-restore'

- task: DotNetCoreCLI@2
  displayName: 'Check code style'
  inputs:
    command: custom
    custom: format
    arguments: >-
        --no-restore
        --verify-no-changes
        --verbosity detailed

The issue is the final task (running dotnet format) will sometimes randomly fail with the above error ("Unable to locate dotnet CLI. Ensure that it is on the PATH").

Here are logs from the UseDotNet@2 task for a pipeline run where we observed the error:

Starting: Install .NET 6
==============================================================================
Task         : Use .NET Core
Description  : Acquires a specific version of the .NET Core SDK from the internet or the local cache and adds it to the PATH. Use this task to change the version of .NET Core used in subsequent tasks. Additionally provides proxy support.
Version      : 2.228.0
Author       : Microsoft Corporation
Help         : https://aka.ms/AA4xgy0
==============================================================================
Tool to install: .NET Core sdk version 6.x.
Found version 6.0.416 in channel 6.0 for user specified version spec: 6.x
Version 6.0.416 was not found in cache.
Getting URL to download .NET Core sdk version: 6.0.416.
Detecting OS platform to find correct download package for the OS.
C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -NoLogo -Sta -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -Command "& 'D:\a\_work\_tasks\UseDotNet_b0ce7256-7898-45d3-9cb5-176b752bfea6\2.228.0\externals\get-os-platform.ps1'"
Primary:win-x64
Detected platform (Primary): win-x64
Downloading: https://download.visualstudio.microsoft.com/download/pr/e5f3c5bd-fb2d-4070-9893-fa3e852c5545/49dc01b583c1007561a3e51f898abe77/dotnet-sdk-6.0.416-win-x64.zip
Extracting downloaded package D:\a\_work\_temp\8f3c7e6f-2b89-4da9-bddf-c2615ca271c9.
Extracting archive
C:\Windows\system32\chcp.com 65001
Active code page: 65001
C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -NoLogo -Sta -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -Command "$ErrorActionPreference = 'Stop' ; try { Add-Type -AssemblyName System.IO.Compression.FileSystem } catch { } ; [System.IO.Compression.ZipFile]::ExtractToDirectory('D:\a\_work\_temp\8f3c7e6f-2b89-4da9-bddf-c2615ca271c9', 'D:\a\_work\_temp\a443')"
Successfully installed .NET Core sdk version 6.0.416.
Creating global tool path and pre-pending to PATH.
Finishing: Install .NET 6

The logs from the UseDotNet@2 task on a successful run are completely identical.

Based on our pipeline runs, we see this failure about 50% of the time.

I'm unsure if this is an issue with UseDotNet, dotnet format, or DotNetCoreCLI.

[mslukebo]

I did some investigating and it turns out there's a race condition in dotnet format. This is hitting on build agents more than local machines since build agents are a bit slower.

dotnet/format#2000