[BUG]: Azure File Copy Task (Version 6) is breaking on pipeline using Self hosted agent with error 'The term 'azcopy' is not recognized.

[Varaprasad91]

New issue checklist

• I searched for existing GitHub issues
• I read pipeline troubleshooting guide
• I checked how to collect logs

Task name

Azure file copy

Task version

6.241.0

Issue Description

We recently upgraded the Azure DevOps (ADO) service connection from using an Azure Resource Manager service principal (automatic) to Azure Resource Manager using Workload Identity federation with OpenID Connect (automatic). But, we got an issue with Azure file copy task (Task version 6) when running it on pipeline with self-hosted agent. The error message received was: 'The term 'azcopy' is not recognized as the name of a cmdlet, function, script file, or operable program. Please check the spelling of the name, or verify that the path is correct and try again.' Also, the task works correctly when using an Azure agent, but fails on the self-hosted agent.

Environment type (Please select at least one enviroment where you face this issue)

• Self-Hosted
• Microsoft Hosted
• VMSS Pool
• Container

Azure DevOps Server type

dev.azure.com (formerly visualstudio.com)

Azure DevOps Server Version (if applicable)

No response

Operation system

windows

Relevant log output

[debug]Script stack trace:
[debug]at ThrowError, C:\Agent\vsts-agent-win-x64-2.182.1\_work\_tasks\AzureFileCopy_eb72cb01-a7e5-427b-a8a1-1b31ccac8a43\6.241.0\Utility.ps1: line 121
[debug]at Upload-FilesToAzureContainer, C:\Agent\vsts-agent-win-x64-2.182.1\_work\_tasks\AzureFileCopy_eb72cb01-a7e5-427b-a8a1-1b31ccac8a43\6.241.0\Utility.ps1: line 272
[debug]at <ScriptBlock>, C:\Agent\vsts-agent-win-x64-2.182.1\_work\_tasks\AzureFileCopy_eb72cb01-a7e5-427b-a8a1-1b31ccac8a43\6.241.0\AzureFileCopy.ps1: line 181
[debug]at <ScriptBlock>, <No file>: line 1
[debug]at <ScriptBlock>, <No file>: line 22
[debug]at <ScriptBlock>, <No file>: line 18
[debug]at <ScriptBlock>, <No file>: line 1
[debug]Exception:
System.Management.Automation.RuntimeException: Upload to container: 'sitemap-generator' in storage account: 'sitemapextractor' with blob prefix: '' failed with error: 'AzCopy.exe exited with non-zero exit code while uploading files to blob storage.' For more info please refer to https://aka.ms/azurefilecopyreadme
2024-06-13T07:48:44.0282744Z ##[error]Upload to container: 'sitemap-generator' in storage account: 'sitemapextractor' with blob prefix: '' failed with error: 'AzCopy.exe exited with non-zero exit code while uploading files to blob storage.' For more info please refer to https://aka.ms/azurefilecopyreadme
[debug]Processed: ##vso[task.logissue type=error;source=TaskInternal]Upload to container: 'sitemap-generator' in storage account: 'sitemapextractor' with blob prefix: '' failed with error: 'AzCopy.exe exited with non-zero exit code while uploading files to blob storage.' For more info please refer to https://aka.ms/azurefilecopyreadme
[debug]Processed: ##vso[task.complete result=Failed]
[section]Finishing: AzureBlob File Copy

Full task logs with system.debug enabled

 [REPLACE THIS WITH YOUR INFORMATION] 

Repro steps

No response

Tasks

Preview Give feedback

No tasks being tracked yet.

[v-schhabra]

Hi @Varaprasad91
Could you please share us the complete logs of the pipeline by adding variable system.debug to "true".

[Varaprasad91]

error logs1.txt
@v-schhabra could you please check this

[v-schhabra]

Hi @Varaprasad91
Thanks for sharing the logs. I hope @v-snalawade already connected with you reg this issue and she has started investigation.

[tomaeda]

Hi, @v-snalawade
I'm having trouble with the same issue. Are there any updates?
Auto-login appears to fail because azcopy.exe isn't available for the oauth token task has retrieved, even though task has running Connect-AzAccount.
Same issue as #19910 ?

[v-schhabra]

Hi, @v-snalawade I'm having trouble with the same issue. Are there any updates? Auto-login appears to fail because azcopy.exe isn't available for the oauth token task has retrieved, even though task has running Connect-AzAccount. Same issue as #19910 ?

We are already investigating on this issue. Will share the latest updates here.
Could you please share us the complete logs of the pipeline by adding variable system.debug to "true".

[v-snalawade]

@tomaeda - Can you check if this suggestion resolves the issue

[tomaeda]

Thank you for your response.

@v-schhabra
I'm sorry, the logs I'm seeing are provided by the end user. I can't share it here.
I haven't been able to reproduce this issue, so I don't have my own logs.
However, the information from line 5599 onwards in the error.logs1.txt shared by Varaprasad91 is consistent with the logs I am seeing, except for specific information such as the storage account name.

@v-snalawade
End user can work around the issue with a different pool.
The issue only occurs with the Self-hosted agent pool, not with MS-hosted pool.
But end user wants to use Self-hosted, so I'd like to know how to resolve the inconsistency between powershell and pwsh. (If this is same issue)

[v-schhabra]

Thank you for your response.

@v-schhabra I'm sorry, the logs I'm seeing are provided by the end user. I can't share it here. I haven't been able to reproduce this issue, so I don't have my own logs. However, the information from line 5599 onwards in the error.logs1.txt shared by Varaprasad91 is consistent with the logs I am seeing, except for specific information such as the storage account name.

@v-snalawade End user can work around the issue with a different pool. The issue only occurs with the Self-hosted agent pool, not with MS-hosted pool. But end user wants to use Self-hosted, so I'd like to know how to resolve the inconsistency between powershell and pwsh. (If this is same issue)

If it is fine for you please share the logs at v-schhabra@microsoft.com

[tomaeda]

I can't upload the logs I'm looking at. I'm trying to reproduce it in my environment so that I can provide the logs, but unfortunately for now I keep failing to reproduce.

[v-schhabra]

#20117
Did fixes for the above issue and deployments are completed to all the rings.