Memory leak (with reproducer) when returning BTreeSet and HashSet from pyfunction
#3285
Comments
|
@adamreichold The change certainly has an impact that removes the net new memory on every single call, however (on commit 1a0c9be ): There is still an overall net growth that each function adds the first time it is called. Do you have an intuition about what that might be? The ~ 6 MB that the BTreeSet test adds, and the ~ 4.5 MB that the BTreeMap test adds both persist until the process terminates. |
|
For convenience, here is my test pulled out of the zip I attached: from contextlib import contextmanager
import gc
import pytest
import psutil
import pyo3mem
@contextmanager
def memlog(i: int):
"""Tool to repeatedly execute a block of code"""
p = psutil.Process()
m0 = p.memory_info().rss
yield
gc.collect()
m1 = p.memory_info().rss
print(f"{i:0>4} {m0:>16d} {m1:>16d} {(m1 - m0):>16d}")
@pytest.mark.parametrize('func,label,expected_type', [
(pyo3mem.demo1, "Vec", list),
(pyo3mem.demo2, "BTreeSet", set),
(pyo3mem.demo3, "HashSet", set),
(pyo3mem.demo4, "BTreeMap", dict),
(pyo3mem.demo5, "HashMap", dict),
])
def test_mem(func, label, expected_type):
print(f"\n\nReturning a {label}")
print(
f'{"iter":>4} {"mem before (b)":>16} {"mem after (b)":>16} {"leaked (bytes)":>16}'
)
n = 10
for i in range(n):
with memlog(i):
result = func()
assert type(result) is expected_type
del resultIt seems unlikely that e.g. the MB is being added to one of Python's long-lived arenas. I'll increase the test data size to see whether a larger allocation triggers an actual memory deallocation when the reference counts drop. In my test code, I have only one reference to the allocated data, which is |
|
Ok good news. I added a zero to the length of the test data returned by the various Takes a while but once we get to ~ 69 MB heap then deallocations back to the OS start to happen. We finish up pretty reliably on ~ 27.5 MB which is where we started (~ 25.8 MB). Looks good 👍🏼 |
|
Would really appreciate a backport and a quick release into 0.18.4 🙏🏼 |
Out of curiosity, why 0.18.x instead of 0.19.x which is what we would usually support? |
|
Even happier if you can do the 0.19 branch, but I just assumed that would take longer because other stuff would be included so it'd be a bigger release process overall. 0.19.x is fine by me if that is also quick! |
Bug Description
Memory is leaked when returning a
BTreeSetorHashSetfrom apyfunctionin an extension module. (Vecand*Mapare ok).Steps to Reproduce
Overview
maturin init)#[pyfunction]that returnsPyResult<BTreeSet<u64>>Detail
The attached archive (
pyo3mem.zip) contains a full project that can reproduce the issue.pyo3mem.zip
To execute the reproducer:
requirements_dev.txt:$ pip install -r requirements_dev.txt$ maturin develop && pytest -sThis is the output from stdout:
Notes
gc.collect()before measuring "after" memorydel resultBacktrace
No response
Your operating system and version
Linux pop-os 6.2.6-76060206-generic #202303130630
168547333822.04~995127e SMP PREEMPT_DYNAMIC Tue M x86_64 x86_64 x86_64 GNU/LinuxYour Python version (
python --version)3.10.6
Your Rust version (
rustc --version)rustc 1.70.0 (90c541806 2023-05-31)
Your PyO3 version
0.19.0
How did you install python? Did you use a virtualenv?
Python installed with apt, and reproducer run inside a venv. These are the contents of the venv (already provided in
requirements_dev.txtin the zip bundle above):Additional Info
I have extensive experience with Python and I can help provide any other assistance you might want.
The text was updated successfully, but these errors were encountered: