Memory leak in yt's test suite (Python 3.12.0rc1 to 3.12.0rc3)

[neutrinoceros]

Bug report

Bug description:

yt's test suite started crashing when switched to Python 3.12. rc1 was the earliest that we tried, and we're still seeing this with rc3. When run through GitHub Actions, jobs simply exit with Error: the operation was cancelled, and we get no traceback.

It appeared that this crash can only be reproduced on machines with little RAM available, so we tried profiling memory usage.
Here's an illustrative graph, provided by @yut23, of memory usage VS time for a sample of yt's test suite (all third-party packages in the environment are at the same version in both envs)

Here's an self-contained reproducer script, using memray to trace memory usage

set -euxo pipefail

git clone https://github.com/yt-project/yt --depth=1

python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install -e "./yt[test]"
python -m pip install memray

# setuptools is inspected at runtime in yt's pytest configuration, but it's not needed otherwise
python -m pip install setuptools

cd yt/
python -m memray run -m pytest yt/data_objects

yt uses weakref quite a lot, so we suspect that the issue here might be reminiscent of #108295 (this was pointed out by @Xarthisius).

CPython versions tested on:

3.12

Operating systems tested on:

Linux, macOS

[mdboom]

I attempted to bisect where this is coming from, but it's challenging since so many of the dependencies break along the 3.11-to-3.12 path.

A standalone, simple reproducer would be very helpful here. Perhaps the memray data offers some clues?

[Xarthisius]

I attempted to bisect where this is coming from, but it's challenging since so many of the dependencies break along the 3.11-to-3.12 path.

A standalone, simple reproducer would be very helpful here. Perhaps the memray data offers some clues?

I'm getting there. A simpler case, but still relying on yt:

import numpy as np
from yt import load_uniform_grid


def foo():
    arr = np.ones((16, 16 ,16))
    data = dict(density=arr)
    bbox = np.array([[-1, 1], [-1, 1], [-1, 1]])
    ds = load_uniform_grid(data, arr.shape, 1.0, bbox=bbox)

    data_source = ds.all_data()
    field = "density"
    min_val, max_val = data_source[field].min() / 2, data_source[field].max() / 2
    return min_val, max_val

for i in range(1000):
    foo()

[mdboom]

Thanks, this is really helpful.

[neutrinoceros]

All I'm actually able to get from memray is that numpy arrays are the thing we're allocating. So far so good, but what I'd like to know is where memory should be freed but isn't, and I'm not sure that memray can help with that.

[mdboom]

FWIW if you are using a Python from one of the python.org macOS installers or from MacPorts, you shouldn't encounter this as they include a private copy of ncurses.

Was this intended for another issue? I'm having trouble connecting the dots (as I don't think ncurses is involved in this bug).

[ned-deily]

Oops, so that's where that comment went! Sorry!

[neutrinoceros]

Update: I've spent a couple hours trying to reduce the reproducer and trying to slowly remove yt from it (easier said than done !)

I discovered that commenting this single line in yt made the issue disappear
https://github.com/yt-project/yt/blob/95e70702d35460ff39e1341fcfe191bb139af5ef/yt/data_objects/static_output.py#L695

So we're not there yet, but that felt like a hint worth sharing.

[mdboom]

Let me share my findings as well. I have a script that prints out "unreachable garbage" at the end of the run (objects where the reference count is larger than len(gc.get_referrers(obj)), indicating some sort of reference counting bug). The only unreachable garbage on 3.12 that's not on 3.11 is a bunch of traceback objects (which are hanging on to a bunch of other objects, hence consuming a lot of memory). So that concurs with your finding, @neutrinoceros, since load_all_plugins() internally raises and catches a lot of exceptions in its normal course of business.

Trying to turn that into a pure Python reproducer has also alluded me: simply raising and catching a bunch of exceptions isn't enough, but is probably somehow related to object cycles and/or weak references getting involved as well. I think a pure Python reproducer will unlock the answer, because then we can git bisect to find the breakage -- as it stands now, I can't get numpy/yt and the various other dependencies to build across the range of 3.12a1 to now.

[neutrinoceros]

I am getting close to a yt-less reproducer, though for now I am not trying to eliminate numpy from it. Of course having a pure Python reproducer would be even better, but since numpy is a self-contained dependency, I'm wondering if that would be enough. Thoughts @mdboom ?

[neutrinoceros]

Here's my reduced reproducer https://github.com/neutrinoceros/reprod_cpython_bug_109602
(I'm tagging the current version no-yt in case I can refine it further later)

In it's current state, it still depends on numpy at runtime and on Cython + setuptools at build time, but yt is out, and we're left with ~100 lines of Python and 3 lines of Cython.

I have faith that the remaining dependency on numpy can be eliminated (I think its only effect is to generate big objects), but I wanted to share my progress as removing yt was quite an undertaking.