Nuget 5.4.0 in ADO - PluginException: Problem starting the plugin /..\CredentialProvider.Microsoft.exe'. Plugin 'CredentialProvider.Microsoft' ... with exit code -1

[rchouser]

Nuget 5.4.0 downloaded into our ADO build
Worked before? Yes with Nuget 4.7.0

Detailed repro steps so we can see the same problem

We are simply running our build which hasn't had many recent changes to the tasks itself. We started getting some build breaks by a behind the scenes change which necessitated our moving to newer task versions for several of our nuget and build related tasks. We initially had success but started hitting this particular error very soon afterwards and now we hit it in about 50% of our builds. When it occurs trying again 2 to 3 times usually gets it to work again. I initially thought that clearing out the list of authenticated nuget feeds was having an effect but I believe that was a false positive. We host our own build VMs and it's hitting the same VM in these cases where simply retrying works. I'll also point out that restarting the build VM only temporarily fixed this.
...

Verbose Logs

Detected NuGet version 5.4.0.6315 / 5.4.0+d790b66be476cd901a56bd46ada037162097ee21.d790b66be476cd901a56bd46ada037162097ee21
SYSTEMVSSCONNECTION exists true
d76576b0-e896-43c7-9012-7185c7fb24a1 exists true
5e02afaa-297f-4717-af44-0c64dd196d4e exists true
926bf1c5-b53e-4291-99d3-1fbf84ca8c8b exists true
E:\a_w_tool\NuGet\5.4.0\x64\nuget.exe restore E:\a_w\3\s\Applications\Applications.sln -PackagesDirectory E:\a_w\3\s\Packages -Verbosity Detailed -NonInteractive -ConfigFile E:\a_w\3\s\nuget.config
NuGet Version: 5.4.0.6315
MSBuild auto-detection: using msbuild version '16.5.0.12403' from 'C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Current\bin'. Use option -MSBuildVersion to force nuget to use a specific version of MSBuild.
MSBuild P2P timeout [ms]: 120000
C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Current\bin\msbuild.exe "C:\Users\rhouser\AppData\Local\Temp\NuGetScratch\ayfaynja.gyn.nugetinputs.targets" /t:GenerateRestoreGraphFile /nologo /nr:false /v:q /p:NuGetRestoreTargets="C:\Users\rhouser\AppData\Local\Temp\NuGetScratch\qma4cg5c.cyw.nugetrestore.targets" /p:RestoreUseCustomAfterTargets="True" /p:RestoreTaskAssemblyFile="E:\a_w_tool\NuGet\5.4.0\x64\nuget.exe" /p:RestoreSolutionDirectory="E:\a_w\3\s\Applications\" /p:RestoreConfigFile="E:\a_w\3\s\nuget.config" /p:RestorePackagesPath="E:\a_w\3\s\Packages" /p:SolutionDir="E:\a_w\3\s\Applications\" /p:SolutionName="Applications"

All packages listed in packages.config are already installed.
Running restore with 4 concurrent jobs.
...

Restoring packages for E:\a_w\3\s\Applications\InputIntel\CommandServiceManager\CommandServiceManager.csproj...
Restoring packages for .NETCoreApp,Version=v3.0...
Restoring packages for .NETCoreApp,Version=v3.0...
Restoring packages for .NETCoreApp,Version=v3.0...
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NuGet.Commands.RestoreCommand.d__32.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NuGet.Commands.RestoreRunner.d__7.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NuGet.Commands.RestoreRunner.d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NuGet.Commands.RestoreRunner.d__9.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at NuGet.Commands.RestoreRunner.d__2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NuGet.Commands.RestoreRunner.d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NuGet.Commands.RestoreRunner.d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NuGet.CommandLine.RestoreCommand.d__52.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NuGet.CommandLine.Command.Execute()
at NuGet.CommandLine.Program.MainCore(String workingDirectory, String[] args))
##[error]Packages failed to restore

Sample Project

I've not been able to distill this yet into a simple project

Any ideas?

[rrelyea]

engaging in teams messaging with filer.

we just shipped 5.5.0 - can you try that version too?
you are using the old credential provider infrastructure. another option is the v2 credential provider mentioned here: https://docs.microsoft.com/en-us/nuget/consume-packages/consuming-packages-authenticated-feeds#nugetexe

[rrelyea]

@rchouser has moved to the latest generation of auth plugins and nuget.exe. Still has a problem.
he has also turned on logging for auth (plugin diagnostic logging - https://github.com/NuGet/Home/wiki/Plugin-Diagnostic-Logging)

@nkolev92 said I looked at the logs and they don’t indicate any problem whatsoever.
Unfortunately I’m not an expert in the Azure DevOps tasks, but what you have done seems pretty standard, so I don’t see anything obvious that’s incorrectly configured.

Conversation continues...

[rrelyea]

Answer from Azure Artifacts team:
This seems like a disconnect between two ways of consuming nuget.
A – use nuget authenticate and script your call to nuget.exe. This flow is documented here: https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/package/nuget-authenticate?view=azure-devops
B – use the nuget command (heavyweight tasks). This flow is documented here: https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/package/nuget?view=azure-devops

The heavyweight tasks do:

1. Getting the correct version of nuget,
2. Auth
3. Calling the command (restore).

When using lightweight tasks, 1 is done with the nuget tool installer https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/tool/nuget?view=azure-devops, 2 is the lightweight task above, and 3 would be using a powershell script to call nuget that was installed by 1.

Those two flows are not meant to be mixed. We recommend flow A btw as it is the most flexible.

Notice in the NuGet Authenticate task it puts the cred provider here: %userprofile%\.nuget\plugins\netcore\CredentialProvider.Microsoft

Yet in the heavyweight tasks it is trying to consume from here: E:\a_w_tasks\NuGetCommand_333b11bd-d341-40d9-afcf-b32d5ce6f23b\2.166.0\CredentialProviderV2\plugins\netfx\CredentialProvider.Microsoft\CredentialProvider.Microsoft.exe