Valgrind lists memory leaks: dlopen() called without dlclose()

[vstinner]

Example on Python 3.12 when loading _crypt extension module:

$ PYTHONMALLOC=malloc valgrind --leak-check=full --show-leak-kinds=all --num-callers=50 ./python -c 'import _crypt'
==2444751== Memcheck, a memory error detector
==2444751== Copyright (C) 2002-2024, and GNU GPL'd, by Julian Seward et al.
==2444751== Using Valgrind-3.23.0 and LibVEX; rerun with -h for copyright info
==2444751== Command: ./python -c import\ _crypt
==2444751== 
==2444751== 
==2444751== HEAP SUMMARY:
==2444751==     in use at exit: 3,253 bytes in 8 blocks
==2444751==   total heap usage: 44,037 allocs, 44,029 frees, 6,464,804 bytes allocated
==2444751== 
==2444751== 21 bytes in 1 blocks are still reachable in loss record 1 of 7
==2444751==    at 0x484282F: malloc (vg_replace_malloc.c:446)
==2444751==    by 0x402587F: malloc (rtld-malloc.h:56)
==2444751==    by 0x402587F: strdup (strdup.c:42)
==2444751==    by 0x4014B58: _dl_load_cache_lookup (dl-cache.c:515)
==2444751==    by 0x4008F1F: _dl_map_object (dl-load.c:2116)
==2444751==    by 0x400287C: openaux (dl-deps.c:64)
==2444751==    by 0x4001522: _dl_catch_exception (dl-catch.c:237)
==2444751==    by 0x4002CDF: _dl_map_object_deps (dl-deps.c:232)
==2444751==    by 0x400CA34: dl_open_worker_begin (dl-open.c:638)
==2444751==    by 0x4001522: _dl_catch_exception (dl-catch.c:237)
==2444751==    by 0x400C10F: dl_open_worker (dl-open.c:803)
==2444751==    by 0x4001522: _dl_catch_exception (dl-catch.c:237)
==2444751==    by 0x400C563: _dl_open (dl-open.c:905)
==2444751==    by 0x49E9E23: dlopen_doit (dlopen.c:56)
==2444751==    by 0x4001522: _dl_catch_exception (dl-catch.c:237)
==2444751==    by 0x4001678: _dl_catch_error (dl-catch.c:256)
==2444751==    by 0x49E9912: _dlerror_run (dlerror.c:138)
==2444751==    by 0x49E9EDE: dlopen_implementation (dlopen.c:71)
==2444751==    by 0x49E9EDE: dlopen@@GLIBC_2.34 (dlopen.c:81)
==2444751==    (...)

Valgrind reports leaks on memory allocated by dlopen() in Python/dynload_shlib.c, because Python doesn't call dlclose().

I don't think that it's big deal to not call dlclose(), since there is little memory and it's not common to unload a dynamic library at runtime. Also, I don't see where/how dlclose() can be called.

Instead, I propose to suppress these warnings in the Valgrind suppressions file, to help spotting more important leaks.

Linked PRs

• gh-121096: Ignore dlopen() leaks in Valgrind suppression file #121097
• [3.13] gh-121096: Ignore dlopen() leaks in Valgrind suppression file (GH-121097) #121122
• [3.12] gh-121096: Ignore dlopen() leaks in Valgrind suppression file (GH-121097) #121123

[picnixz]

Also, I don't see where/how dlclose() can be called

AFAICT, dlopen() is used for the init function of modules. I don't know if it's possible, but I would assume that dlclose should be called once the module is unloaded or if there is not reference anymore. In addition, dlclose() does not decessarily unload the shared library, it only decrements the number of references (like Py_DECREF).

Is there a callback mechanism for (extension) modules that could be called in order to close the handle when needed? As for "how", you need to carry over the handle in the module objects probably...

[vstinner]

Is there a callback mechanism for (extension) modules that could be called in order to close the handle when needed?

I don't think that it does exist, and it would require to rewrite a lot of things. For now, I prefer to simply ignore ("suppress") the "leak" in Valgrind.