Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

PyArrowConvert Leaks Memory #3683

Closed
tustvold opened this issue Feb 9, 2023 · 1 comment · Fixed by #3685 or #3893
Closed

PyArrowConvert Leaks Memory #3683

tustvold opened this issue Feb 9, 2023 · 1 comment · Fixed by #3685 or #3893
Assignees
@tustvold
Labels
arrow Changes to the arrow crate bug

Comments

@tustvold
Copy link
Contributor

tustvold commented Feb 9, 2023

Describe the bug

PyArrowConvert calls ArrowArray::into_raw to convert the contents of ArrowArray::empty into raw pointers, it then passes these to python to populate, before passing them to ArrowArray::try_from_raw. This copies the memory contents and does not take ownership of the passed pointers, this appears to have been a change made in (#1449) by @viirya. The result is that the allocation of Arc<FFI_ArrowSchema> and Arc<FFI_ArrowArray> is never freed.

Some of the memory leaks appear to have been fixed in #1878 but not the ones present in the python interface.

I happen to think the into_raw method is extremely hard to use correctly, and I would propose removing it in favor of an interface that does not yield ownership of the pointers.

To Reproduce

Expected behavior

Additional context
@tustvold tustvold added the bug label Feb 9, 2023
@tustvold tustvold self-assigned this Feb 9, 2023
@tustvold tustvold mentioned this issue Feb 9, 2023
Closed
tustvold added a commit to tustvold/arrow-rs that referenced this issue Feb 9, 2023
@tustvold
Cleanup FFI interface (apache#3684) (apache#3683)
0ef2333
tustvold added a commit to tustvold/arrow-rs that referenced this issue Feb 9, 2023
@tustvold
Cleanup FFI interface (apache#3684) (apache#3683)
cf9f573
@tustvold tustvold mentioned this issue Feb 9, 2023
Merged
tustvold added a commit that referenced this issue Feb 9, 2023
@tustvold
Cleanup FFI interface (#3684) (#3683) (#3685)
bb4fc59 
* Cleanup FFI interface (#3684) (#3683)

* Add import example

* Tweak doc example

* Use ManuallyDrop to model external memory
@tustvold tustvold added the arrow Changes to the arrow crate label Feb 10, 2023
@tustvold
Copy link
Contributor Author

label_issue.py automatically added labels {'arrow'} from #3685

@tustvold tustvold reopened this Mar 21, 2023
tustvold added a commit to tustvold/arrow-rs that referenced this issue Mar 21, 2023
@tustvold
Fix pyarrow memory leak (apache#3683)
14122fc 
Remove ArrowArray::into_raw and try_from_raw
tustvold added a commit to tustvold/arrow-rs that referenced this issue Mar 21, 2023
@tustvold
Fix pyarrow memory leak (apache#3683)
5b72667 
Remove ArrowArray::into_raw and try_from_raw
@tustvold tustvold mentioned this issue Mar 21, 2023
Merged
tustvold added a commit to tustvold/arrow-rs that referenced this issue Mar 21, 2023
@tustvold
Fix pyarrow memory leak (apache#3683)
d2b50c4 
Remove ArrowArray::into_raw and try_from_raw
tustvold added a commit to tustvold/arrow-rs that referenced this issue Mar 21, 2023
@tustvold
Fix pyarrow memory leak (apache#3683)
8573b38 
Remove ArrowArray::into_raw and try_from_raw
tustvold added a commit to tustvold/arrow-rs that referenced this issue Mar 21, 2023
@tustvold
Fix pyarrow memory leak (apache#3683)
9028daa 
Remove ArrowArray::into_raw and try_from_raw
tustvold added a commit that referenced this issue Mar 21, 2023
@tustvold
Fix pyarrow memory leak (#3683) (#3893)
3a6f8bd 
* Fix pyarrow memory leak (#3683)

Remove ArrowArray::into_raw and try_from_raw

* Update docs

* Further deprecation

* Clippy
spebern pushed a commit to spebern/arrow-rs that referenced this issue Mar 25, 2023
@tustvold @spebern
Fix pyarrow memory leak (apache#3683) (apache#3893)
515772a 
* Fix pyarrow memory leak (apache#3683)

Remove ArrowArray::into_raw and try_from_raw

* Update docs

* Further deprecation

* Clippy
@jonmmease jonmmease mentioned this issue Mar 25, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tustvold tustvold

Labels
arrow Changes to the arrow crate bug
Projects
None yet
Milestone
No milestone
1 participant
@tustvold