-
Notifications
You must be signed in to change notification settings - Fork 25
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
dask 2021.8.0
breaks parquet io
#84
Comments
I think at some point, dask/pyarrow wrote metdata files without the proper file path. The latest release of dask switched to use |
Having the exact same issue with |
This is fixed by #92. Tests all pass with dask |
Opening issue here for visibility and to track any potential updates.
ALL software version info
(this library, plus any other relevant software, e.g. bokeh, python, notebook, OS, browser, etc)
Description of expected behavior and the observed behavior
Reading and writing to parquet is producing errors when used with the latest version of dask
2021.8.0
. I have tested reverting back to dask2021.7.2
and do not experience any of the issues outlined below.The dask team may already be aware of this issue:
_metadata
dask/dask#8030Complete, minimal, self-contained example code that reproduces the issue
The code block above and
ddf.pack_partitions_to_parquet
both produce the following error:The text was updated successfully, but these errors were encountered: