sub-interpreters: significant memory leak on shutdown

[costasgambit]

Bug report

Bug description:

Hi all, congrats on the new Python release, very keen on the direction that sub-interpreters are taking!

I noticed that constructing and destructing sub-interpreters now causes memory to be leaked. This was not a problem in 3.11, but is as of 3.12 and occurs on latest main branch too (as of 8c07137). I figure it must relate to the changes made to split up the interpreter states.

I can consistently replicate the issue using simply:

import _xxsubinterpreters
while 1:
    interp = _xxsubinterpreters.create()
    _xxsubinterpreters.destroy(interp)

You'll notice the memory usage climb quite quickly if you look at top/htop.

valgrind/massif think the culprit are interned strings, specifically here:

==219777==    at 0x4848A13: calloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==219777==    by 0x2DA59E: PyMem_RawCalloc (obmalloc.c:671)
==219777==    by 0x2DA59E: arena_map_get (obmalloc.c:1021)
==219777==    by 0x2DA59E: arena_map_mark_used (obmalloc.c:1076)
==219777==    by 0x2DAA71: new_arena (obmalloc.c:1209)
==219777==    by 0x2DAA71: allocate_from_new_pool (obmalloc.c:1389)
==219777==    by 0x2DB00B: pymalloc_alloc (obmalloc.c:1553)
==219777==    by 0x2DB00B: _PyObject_Malloc (obmalloc.c:1564)
==219777==    by 0x424AF5: gc_alloc (gcmodule.c:2304)
==219777==    by 0x424AF5: _PyObject_GC_New (gcmodule.c:2319)
==219777==    by 0x2BDF3B: new_dict (dictobject.c:748)
==219777==    by 0x2BDF3B: PyDict_New (dictobject.c:851)
==219777==    by 0x34CF06: init_interned_dict (unicodeobject.c:248)
==219777==    by 0x34CF06: _PyUnicode_InitGlobalObjects (unicodeobject.c:14670)
==219777==    by 0x3EDE18: pycore_init_global_objects (pylifecycle.c:678)
==219777==    by 0x3EDE18: pycore_interp_init (pylifecycle.c:826)
==219777==    by 0x3F07A9: new_interpreter (pylifecycle.c:2079)
==219777==    by 0x3F07A9: Py_NewInterpreterFromConfig (pylifecycle.c:2114)
==219777==    by 0x4867A52: interp_create (_xxsubinterpretersmodule.c:516)
==219777==    by 0x2D0C22: cfunction_call (methodobject.c:537)
==219777==    by 0x2716A7: _PyObject_MakeTpCall (call.c:240)

I had a look around that code but I can't spot any obvious bug. I did verify that the interned strings dict created and destroyed in clear_interned_dict() is the same for each sub-interpreter, just in case.

I think every interned string is now also automatically immortal. Does that mean it wouldn't be cleaned up even if the refcount goes down to zero?

CPython versions tested on:

3.11, 3.12, CPython main branch

Operating systems tested on:

Linux

[costasgambit]

I can reproduce this from C as well, without using _xxsubinterpreters.

[itamaro]

cc @ericsnowcurrently

[ehenson]

I can reproduce as well. I'm using Go with CGO.

This is within a Gin route Post handler.

var config = &C.PyInterpreterConfig{
	use_main_obmalloc:             0,
	allow_fork:                    0,
	allow_exec:                    0,
	allow_threads:                 1,
	allow_daemon_threads:          0,
	check_multi_interp_extensions: 1,
	gil: C.PyInterpreterConfig_OWN_GIL,
}

		runtime.LockOSThread()
		// Create a new subinterpreter
		var tstate *C.PyThreadState

		status := C.Py_NewInterpreterFromConfig(&tstate, config)

// compile and execute a tiny Python script as if it were an extension point for my application.
// paying very close attention to reference counting.

		handleRequest(c)

		C.Py_EndInterpreter(tstate)
		runtime.UnlockOSThread()
})

When using a stress tester, the memory grows into gigabytes within a few seconds, consistently growing, and never recovers.

[dawmster]

On windows 11 too...

[ehenson]

I am now using 3.12.1, and it is still leaking. I am using MacOS 14.2 M2 and Linux amd64.

I have studied the documentation about using Py_NewInterpreterFromConfig, and I doubt that I am using it correctly. If someone can post here a C example of using Py_NewInterpreterFromConfig, the Python script is a simple print("Hello, World!"), the C example is in the proper form of using Py_NewInterpreterFromConfig concerning managing the GIL and every step needed until Py_EndInterpreter; I will confirm my code and gladly report my results.

[ericsnowcurrently]

Sorry, this slipped through the cracks for me. 😞

I expect the underlying cause is the same as for gh-113055 and kind of has the same fix (gh-113412). Basically, the obmalloc implementation has never freed the blocks we get from the system allocator. There's always been a leak! It was just much harder to notice before. The difference between 3.11 and 3.12+ is that each interpreter has its own obmalloc state rather than sharing a process-global one like we used to. Consequently, all the memory each (isolated) subinterpreter uses leaks now.

Let's circle back on this once gh-113412 is merged and maybe also gh-113601.