'FileNotFoundError' with '.compute()' operation on Dask dataframe read using 'read_deltalake'

[sartsha]

Hello. I am facing this issue on a local deltalake table on a single machine setup.

Environment:

• Windows 10
• Python 3.10.9
• dask-deltatable 0.3.3
• deltalake 0.20.2
• dask 2024.10.0
• pyarrow 17.0.0

Minimal code to reproduce:

from deltalake import write_deltalake
import pandas as pd
import numpy as np

some_strings = ['apple', 'banana', 'grape']
n = 10
df = pd.DataFrame({
    "Column1": np.random.choice(some_strings, n),
    "Column2": np.random.randint(1,100,n)
})
table_path = 'Relative path/random_table'
write_deltalake(table_path, df, mode='overwrite')



import dask_deltatable as ddt

ddf = ddt.read_deltalake(table_path, version=0)
print(list(ddf.columns)) # Output: ['Column1', 'Column2']

print(ddf.compute().size) # Error!

Error description: "FileNotFoundError: C:/...<full_path>.../Relative path/random_table/part-00001-...c000.snappy.parquet"
This was raised from _ensure_single_source() function in pyarrow/dataset.py

Whenever I try to do the .compute() operation to convert it to a pandas dataframe, the error above is raised, even though the files and folder are present.

I have tried setting the 'dataframe.query-planning' option to False, but no luck