Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support pyarrow>=13 #1744

Closed
dhirschfeld opened this issue Oct 20, 2023 · 1 comment · Fixed by #1804
Closed

Support pyarrow>=13 #1744

dhirschfeld opened this issue Oct 20, 2023 · 1 comment · Fixed by #1804
Labels
enhancement New feature or request

Comments

@dhirschfeld
Copy link
Contributor

Description

As it say's in the title.

delta-rs/python/pyproject.toml

Line 20 in 1691030

"pyarrow>=8,<13",

xref: #1743
@dhirschfeld dhirschfeld added the enhancement New feature or request label Oct 20, 2023
@wjones127
Copy link
Collaborator

Started to look into this in #1602, FYI

@ion-elgreco ion-elgreco mentioned this issue Nov 4, 2023
Merged
wjones127 added a commit that referenced this issue Nov 5, 2023
@ion-elgreco @wjones127 @rtyler
fix(python): add support for pyarrow 13+ (#1804)
d98a0c2 
# Description

I build on top of the branch of @wjones127
#1602. In pyarrow v13+ the
ParquetWriter by default uses the `compliant_nested_types = True` (see
related PR: https://github.com/apache/arrow/pull/35146/files)and the
docs:
https://arrow.apache.org/docs/python/generated/pyarrow.parquet.ParquetWriter.html).

In arrow/parquet-rs it fails when it compares schemas because it
expected the old non-compliant ones. For now we can have pyarrow 13+
supported by disabling it or updating the file options provided by a
user.

# Related Issue(s)
<!---
For example:

- closes #106
--->

- Closes #1744

# Documentation

<!---
Share links to useful documentation
--->

---------

Co-authored-by: Will Jones <willjones127@gmail.com>
Co-authored-by: R. Tyler Croy <rtyler@brokenco.de>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix(python): add support for pyarrow 13+ ion-elgreco/delta-rs
2 participants
@dhirschfeld @wjones127