A simple working Azure Blob storage example?

[min-mwei]

Hi. Is there an example to list and read data from Azure blob? List, read, and ideally write.

I have some issues with the path scheme. For example, with Ray/ADLS, the pattern works "az://containerName/TopfolderPath"
ds = ray.data.read_parquet(
"az://containerName/path/",
filesystem=adlfs.AzureBlobFileSystem(account_name="x", account_key="y")
)

But this simple code with daft below gives a file not found error.
io_config = IOConfig(azure=AzureConfig(storage_account="x", access_key="y"))
daft.read_parquet("az://containerName/path/", io_config=io_config)

Thanks much

[jaychia]

Hi @min-mwei!

A few questions:

1. What version of Daft are you using?
2. What is the structure of your folder? Are you attempting to find files in a nested folder structure, or are all of your files on the same top-level of the folder?

Daft recognizes a filepath ending with / as a "folder", and automatically find all files in that folder but not recursively.

I.e. these are equivalent:

• daft.read_parquet("az://containerName/path/")
• daft.read_parquet("az://containerName/path/*")

If instead you want Daft to find data recursively, you can use our ** wildcard syntax to perform a recursive file search:

daft.read_parquet("az://containerName/path/**/*.parquet", io_config=io_config)

Let me know if this helps!

[min-mwei]

Thanks Jay. Here is the daft and python version

Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import daft
>>> daft.__version__
'0.2.12'
>>> 

Right now, I am only doing simple testing so no nested path. Is there some unit test code I could take a look?

[jaychia]

Yes, here are our file listing tests for Azure: https://github.com/Eventual-Inc/Daft/blob/main/tests/integration/io/test_list_files_azure.py#L38-L55

I also did some quick testing and found that on my machine I only get a FileNotFound error if the selected az://containerName/path/ was an empty folder. Otherwise, Daft seems to work as expected

Did you manage to try using the * wildcard to see if that helped?

[min-mwei]

Thanks Jay. I just spent sometime to debug this issue a bit. I think the root cause is that my storage account does not have "hierarchical namespace" enabled on creation. This Daft f"az://{CONTAINER}/path/**" scheme works on a new test storage account with that bit true on creation. It would be good to document this behavior.

[jaychia]

😵 good find!

Thanks for finding this, we had no idea that Azure allowed for that option! Our globber is heavily optimized to use hierarchical namespaces for embarrassingly parallel listing of large nested hierarchies in AWS S3 (it does a mix of recursing down the folder hierarchy + early-stopping to fallback onto prefix-listing)

I think it then makes sense that it would "fail to find a folder" in a non-hierarchical storage account.

I'll add some documentation as a stopgap, but we might be able to perform some autodetection + fallback behavior for non-hierarchical containers in azure blob store.

[jaychia]

Docs are merging!

#1851

We're also looking into what it would take to support non-heirarchical storage account. Ideally we would:

1. Detect whether the storage account is hierarchical or not
2. If it is, use the current algorithm
3. If it is not, fall back on a naive listing algorithm

Ticket: #1852