Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Investigate issue with zero-copy PyArrow to arrow-rs interface #125

Open
jonmmease opened this issue Jun 9, 2022 · 2 comments · Fixed by #264
Open

Investigate issue with zero-copy PyArrow to arrow-rs interface #125

jonmmease opened this issue Jun 9, 2022 · 2 comments · Fixed by #264
Labels
investigation

Comments

@jonmmease
Copy link
Collaborator

We disabled the use of the PyArrow -> arrow-rs zero-copy interface in #124 due to a correctness issue. We should more thoroughly investigate the cause of this, as it would be advantageous to re-adopt the zero-copy interface again in the future.
@jonmmease jonmmease mentioned this issue Mar 18, 2023
Merged
@jonmmease
Copy link
Collaborator Author

I tried to re-enable the zero-copy support in #264, but this was later (logically) reverted in #269 due to an issue issue where pre_transform_dataset would change. #269 added a regression test for what I was seeing.

I just noticed a fix in the arrow-rs 36 changelog: PyArrowConvert Leaks Memory. After we update to arrow-rs 36, I'd like to see if this takes care of the issue above.

@jonmmease jonmmease reopened this Mar 25, 2023
@jonmmease
Copy link
Collaborator Author

When revisiting this, let's look into the pyo3-arrow crate: https://docs.rs/pyo3-arrow/latest/pyo3_arrow/#

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
investigation
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Re-enable zero copy from pyarrow to arrow-rs vega/vegafusion
1 participant
@jonmmease