[C++][FS][Azure] SAS token authentication

[Tom-Newton]

Describe the enhancement requested

Child of #38598

Add support for Azure CLI auth. Probably just accept the SAS token as an argument and use AzureSasCredential https://github.com/Azure/azure-sdk-for-cpp/blob/101f20f2bbf3dd5f6438565cd9f709a231317f77/sdk/tables/azure-data-tables/inc/azure/data/tables/credentials/azure_sas_credential.hpp#L16C9-L16C27. This should make the implementation very similar to all the other Azure auths.

One possible complication I'm aware of is in CopyFile, because this is implemented by generating a federated SAS token from whatever the original authentication was. I don't know if its possible to get a federated SAS token when using SAS token auth.

Also I'm not sure we need to generate a federated SAS token to implement CopyFile. I think we should be able to use Copy Blob instead of Copy Blob From URL. I think the latter is only needed if the source is not in the same Azure blob storage account.

Component(s)

C++

[Tom-Newton]

take

[Tom-Newton]

I have it working to some extent, but as I feared there is a complexity with CopyFile because its implemented by creating a SAS token at runtime #41276. When authenticated with a SAS token already we cannot generate a new SAS token in the current way (I tested that it does work with Azure CLI auth so the problem is not just broken code that we still need to test #41315).

There are 2 options:

1. Stop using SAS token inside CopyFile
   1. With azcopy a SAS token is only needed when doing a copy from one Azure storage account to another Azure storage account. Within a single storage account we should be able to use the Copy Blob API instead of Copy Blob From URL but I'm not sure if this is actually exposed by the C++ SDK.
   2. I'm a bit concerned by the current implementation that generates a new SAS token for every blob. Azure tends to have tight rate limits on APIs to generate authentication tokens, so I suspect a call to CopyFiles on a large directory would have problems.
   3. We can close [C++][FS][Azure] Test CopyFile() with non account key credential #41315
2. Use the SAS token we already have
   1. Definitely possible.

[kou]

The option 1 is better, right? Let's try the option 1.

[Tom-Newton]

The option 1 is better, right? Let's try the option 1.

I thought so too but, after trying it, now I'm not so sure. I checked a few of the Azure SDKs (C++, Python, and golang) and it actually looks like none of them expose the Copy Blob API.

I found the SourceAuthorization option which should allow configuring different authentication to the source when using Copy Blob From URL. Plus some some examples of how to use it in azcopy. There also seem to be a string of changes in azcopy to try to get this working correctly, because of problems like making too many token requests or using a cached token after it expired. The current implementation adds a custom http policy when creating the Azure client, which intercepts requests that include an empty x-ms-copy-source-authorization and populate it https://github.com/Azure/azure-storage-azcopy/blob/7f5ab059c28b09841d009d28b7510d4fcb167dc8/ste/sourceAuthPolicy.go#L47-L77.

This is all looking rather complicated and I don't think we can generate the required bearer tokens with SAS or account key auth. So I think I'll stick with generating SAS tokens, and I think I will just bump their expiry a bit as a simple mitigation for the problem mentioned in azcopy where tokens expired during retries.

[Tom-Newton]

Ok, I think I finally worked it out StartCopyFromUri in the Azure C++ SDK can do copies within the same storage account any extra authentication required.

[kou]

Issue resolved by pull request 45021
#45021