You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I'm reading in the delta table as dask dataframe using dask-deltatable's read_deltalake('path', engine='pyarrow'). After performing some manipulations, I'm trying to write the dask dataframe as deltatable using to_deltalake('path', ddf).
However, I'm getting the following error. The parquet files are getting created in the destination but the delta log folder is not. The final "delta-commit" step fails when observing the dask dashboard.
AttributeError: 'pyarrow.lib.Schema' object has no attribute 'origin' corresponding to this line.
This is a Schema Violation exception and I can bypass this error when I explicitly mention the schema when writing (using to_deltalake('path', ddf, schema=schema). But specifying the schema everytime while writing is a tedious task and not a very good approach.
FYI, I'm using deltalake==0.13.0 (as only this supports dask-deltatable) and dask-deltatable==0.3.1
Related Issue(s)
Also, I assume this is related to this issue: #686
The text was updated successfully, but these errors were encountered:
I'm reading in the delta table as dask dataframe using dask-deltatable's read_deltalake('path', engine='pyarrow'). After performing some manipulations, I'm trying to write the dask dataframe as deltatable using to_deltalake('path', ddf).
However, I'm getting the following error. The parquet files are getting created in the destination but the delta log folder is not. The final "delta-commit" step fails when observing the dask dashboard.
AttributeError: 'pyarrow.lib.Schema' object has no attribute 'origin'
corresponding to this line.This is a Schema Violation exception and I can bypass this error when I explicitly mention the schema when writing (using to_deltalake('path', ddf, schema=schema). But specifying the schema everytime while writing is a tedious task and not a very good approach.
FYI, I'm using deltalake==0.13.0 (as only this supports dask-deltatable) and dask-deltatable==0.3.1
Related Issue(s)
Also, I assume this is related to this issue: #686
The text was updated successfully, but these errors were encountered: