Copying files from Azure File Share to local directory consistently failing with v10.27.0

[sgerace]

We started noticing intermittent failures in one of our GitHub actions pipelines which uses azcopy to download a number of files (34200 files, 1223 folders, 5,213,971,256 bytes) from an Azure File Share to the local build machine. After looking at the logs, we noticed that the failures are occurring whenever the build agent has v10.27.0 of azcopy installed. Since some agents were still running v10.26.0, the job succeeds "intermittently" whenever we are assigned a GitHub actions runner that still has the older version of azcopy installed (we're using the ubuntu-latest build agent image). We've been seeing the failure rate slowly increase over the last several days, and now it is failing every time (because all runners now have the azcopy v10.27.0 installed).

We are able to consistently reproduce the error on a local development machine (running macOS); the operation succeeds without any issues when running v10.26.0 and fails every time when running v10.27.0.

The command we are running is:

azcopy copy "https://$STORAGE_ACCOUNT_NAME.file.core.windows.net/$FILE_SHARE_NAME/" "$BACKUP_DIR" --recursive

We are leveraging AZCOPY_AUTO_LOGIN_TYPE=SPN and setting AZCOPY_SPA_APPLICATION_ID, AZCOPY_SPA_CLIENT_SECRET, and AZCOPY_TENANT_ID to provide the command the SPN credentials.

Looking through the logs, it seems like there are a lot of DOWNLOADFAILED errors related to "no such file or directory":

2024/11/07 12:28:44 ERR: [P#0-T#72] DOWNLOADFAILED: https://#################.file.core.windows.net/##########/index.html : 000 : File Creation Error open /###########/temp/##########/.azDownload-49f6fa0f-894d-ad45-682c-b96d8ee5b525-index.html: no such file or directory
   Dst: /###########/temp/##########/index.html
2024/11/07 12:28:44 INFO: [P#0-T#72] https://#################.file.core.windows.net/##########/index.html Deleting incomplete destination file Dst: /###########/temp/##########/index.html
2024/11/07 12:28:44 ERR: [P#0-T#72] /###########/temp/##########/index.html: 0: Delete File Error -remove /###########/temp/##########/.azDownload-49f6fa0f-894d-ad45-682c-b96d8ee5b525-index.html: no such file or directory. X-Ms-Request-Id:

2024/11/07 12:28:44 ERR: [P#0-T#73] DOWNLOADFAILED: https://#################.file.core.windows.net/###########/582.2ac69e076b3048b7.js : 000 : File Creation Error open /###########/temp/###########/.azDownload-49f6fa0f-894d-ad45-682c-b96d8ee5b525-582.2ac69e076b3048b7.js: no such file or directory
   Dst: /###########/temp/###########/582.2ac69e076b3048b7.js
2024/11/07 12:28:44 INFO: [P#0-T#73] https://#################.file.core.windows.net/###########/582.2ac69e076b3048b7.js Deleting incomplete destination file Dst: /###########/temp/###########/582.2ac69e076b3048b7.js
2024/11/07 12:28:44 ERR: [P#0-T#73] /###########/temp/###########/582.2ac69e076b3048b7.js: 0: Delete File Error -remove /###########/temp/###########/.azDownload-49f6fa0f-894d-ad45-682c-b96d8ee5b525-582.2ac69e076b3048b7.js: no such file or directory. X-Ms-Request-Id:

Note that I've redacted sensitive the storage account and file share names as well as the file paths (hence the ####s in the logs).

This is a major issue for us at the moment as none of our pipelines are able to function properly. Thanks!

[gapra-msft]

Hi @sgerace would you be able to share azcopy debug logs with us to our email azcopydev AT microsoft.com? This would help expediate investigation.

In the meantime, here is a link to the 10.26 executable.

[kayoshino]

Just to add - my team is also seeing similar issues from v10.27.0.

[sgerace]

@gapra-msft, I'm just coming back to this today and it looks like you've already created a PR. Do you still need our debug logs? If so, are there any specific arguments we should add to the azcopy command to get more detailed logs?

[gapra-msft]

@sgerace --log-level=DEBUG would be the most verbose logs.

Yes, sharing them would be helpful. We are still working on the fix.

[sgerace]

@gapra-msft, I finally got around to generating and downloading the logs with --log-level=DEBUG and the main log file is 105.8 MB. Should I go ahead and truncate? I just don't want to give you a bunch of repeat logs.

[gapra-msft]

@sgerace If you could zip it, that would be ideal.

[sgerace]

@gapra-msft, I just tried to email the compressed log file to azcopydev@microsoft.com, but received the following error:

[gapra-msft]

@sgerace could you try my email? gapra AT microsoft.com

[sgerace]

@gapra-msft Did that work?

[gapra-msft]

Hi @sgerace Thanks! Yes, I was able to access the logs in your email.

I believe this is the same issue we are currently working towards fixing. We will be sure to update you once a proper fix is available.

[gapra-msft]

Hi @sgerace We just released a patch which should resolve your problem. Please upgrade to 10.27.1

[sgerace]

Thanks @gapra-msft, I can confirm that upgrading to 10.27.1 has fixed our issue.