object_store: Support `container@account.dfs.core.windows.net/path` URL style for `az` protocol

[daviewales]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

In adlfs (fsspec), there are two valid conventions for specifying a canonical path to Azure Data Lake Gen2 storage, including the account name:

az://container@account.dfs.core.windows.net/path-part/file
abfs://container@account.dfs.core.windows.net/path-part/file

While object_store supports both the az and abfs protocols, it does not support specifying the account in the URL for the az protocol.
However, it does support this for the abfs protocol.

Describe the solution you'd like
I would like to be able to use the following URL convention with object_store, which improves compatibility with adlfs:

az://container@account.dfs.core.windows.net/path-part/file

Describe alternatives you've considered
The workaround is to remember that I need to use the abfs protocol in object_store based tools such as polars.
However, it would be nice to be able to use the az protocol for everything.

[tustvold]

This seems fine to me and should be a relatively straightforward change to MicrosoftAzureBuilder::parse_url

[daviewales]

As an extension to the above feature request, I can see that object_store supports the following URL schemes:

• https://<account>.dfs.core.windows.net
• https://<account>.blob.core.windows.net
• https://<account>.blob.core.windows.net/<container>
• https://<account>.dfs.fabric.microsoft.com
• https://<account>.dfs.fabric.microsoft.com/<container>
• https://<account>.blob.fabric.microsoft.com
• https://<account>.blob.fabric.microsoft.com/<container>

However, it would be excellent if these could be extended to support the following generic structure:

• https://<account>.<type>.<service>/<container>/<path>

where:

• <account>: the name of the storage account
• <type>: dfs or blob
• <service>: core.windows.net or fabric.microsoft.com
• <container>: the name of the container
• <path>: the directory path within the container

This matches the URL structure used by Microsoft's AzCopy tool.
It also matches a supported URL structure for Azure Synapse's OPENROWSET function.

[kylebarron]

The only addition you're suggesting is the <path>? You can use the PrefixStore to handle that

[daviewales]

Also in the documentation, it lists

https://<account>.blob.core.windows.net/<container>

But not

https://<account>.dfs.core.windows.net/<container>

My goal is to be able to use the following in Polars:

pl.scan_csv('https://<account>.dfs.core.windows.net/<container>/path.csv')

But I've tried, and it didn't seem to work.

When I checked the object_store documentation, I saw that it supported https://<account>.dfs.core.windows.net, but not https://<account>.dfs.core.windows.net/<container>, so I thought that might be the issue.`

I also saw in the documentation that other URL schemes were specified as supporting the <path>, but not the https://<account>.dfs.core.windows.net one.

[ByteBaker]

@kylebarron @daviewales I'd like to take a shot at this once we have concluded what all we want here, and what is currently as it should be. I'm starting with the az://container@account.dfs.core.windows.net/path-part/file which is at the moment extremely simple.

What do you suggest?

[daviewales]

I think that we've only reached agreement on az://container@account.dfs.core.windows.net/path-part/file. There hasn't yet been any support for the form https://<account>.dfs.core.windows.net/<container>/path.

[ByteBaker]

Great. In this case, I've made the change already, apparently it was only a tiny flip, the code was already there. Will create the PR now and share.

[kylebarron]

I think that we've only reached agreement on az://container@account.dfs.core.windows.net/path-part/file. There hasn't yet been any support for the form https://<account>.dfs.core.windows.net/<container>/path.

This already exists, however the current code incorrectly interprets the container as part of the path. See #399 and #398