Py_Finalize() doesn't clear all Python objects at exit

[kylotan]

BPO	1635741
Nosy	@amauryfa, @orsenthil, @vstinner, @tiran, @phsilva, @encukou, @ambv, @ericsnowcurrently, @serhiy-storchaka, @phmc, @DimitrisJim, @corona10, @miss-islington, @shihai1991, @erlend-aasland, @koubaa, @h-vetinari, @csabesz88, @kumaraditya303, @CharlieZhao95
PRs	bpo-1635741: Port _json extension module to multiphase initialization(PEP 489) #17835 bpo-1635741: Remove redundant _Py_INC_TPALLOCS in _Py_NewReference() #17883 bpo-1635741: Port _abc extension module to multiphase initialization(PEP 489) #18030 bpo-1635741: Port _asyncio extension module to multiphase initialization(PEP 489) #18032 bpo-1635741: Port _bisect extension module to multiphase initialization(PEP 489) #18049 bpo-1635741: Port _bz2 extension module to multiphase initialization(PEP 489) #18050 bpo-1635741: Port _codecs extension module to multiphase initialization (PEP 489) #18065 bpo-1635741: Port _collections extension module to multiphase initialization(PEP 489) #18066 bpo-1635741: Port _locale extension module to multiphase initialization (PEP 489) #18358 bpo-1635741: Calling Py_INCREF() after PyModule_AddObject() success to run #18365 bpo-1635741: Port _contextvars extension module to multiphase initialization (PEP … #18374 bpo-1635741: Port _crypt extension module to multiphase initialization(PEP 489) #18404 bpo-1635741: Fix refleaks of timemoudle in PyInit_time() #18486 bpo-1635741: Port audioop extension module to multiphase initialization(PEP 489) #18608 bpo-1635741: Fix potential refleaks in binascii module #18613 bpo-1635741: Port _ctypes_test extension module to multiphase initialization(PEP … #19012 bpo-1635741: Port statistics module to multiphase initialization #19015 bpo-1635741: Fix possible refleaks in tupleobject.c #19018 [3.7] bpo-1635741: Fix a possible refleak in tupleobject.c (GH-19018) #19022 [3.8] bpo-1635741: Fix a possible refleak in tupleobject.c (GH-19018) #19021 bpo-1635741: Port itertools module to multiphase initialization. #19044 bpo-1635741: Port _heapq module to multiphase initialization. #19057 bpo-39337: Add a test case for normalizing of codec names #19069 [WIP] bpo-1635741: Port _collections module to multiphase initialization. #19071 bpo-1635741: Port _collections module to multiphase initialization.  #19074 bpo-1635741: Port _weakref extension module to multiphase initialization(PEP 489) #19084 bpo-1635741: Enhance get_binascii_state() #19100 bpo-1635741: Port time module to multiphase initialization (PEP 489) #19107 bpo-1635741: Port _datetime module to multiphase initialization (PEP 489) #19122 Revert "bpo-1635741: Port _weakref extension module to multiphase initialization (PEP 489) (GH-19084)" #19128 bpo-40050: Fix importlib._bootstrap_external #19135 bpo-1635741: Port _weakref extension module to multiphase initialization (PEP 489) (GH-19084) #19140 bpo-1635741: Port operator module to multiphase initialization (PEP 489) #19150 bpo-1635741: Port _functools module to multiphase initialization (PEP-489) #19151 bpo-1635741: Port _uuid module to multiphase initialization. #19242 bpo-1635741: Port math module to multiphase initialization #19243 bpo-1635741: Port resource extension module to multiphase initialization(PEP 489) #19252 bpo-1635741: Enhance PyModule_AddIntMacro by ADD_INT macro #19307 bpo-1635741: Port _lzma module to multiphase initialization  #19382 bpo-1635741: Port mmap module to multiphase initialization #19459 bpo-1635741: Port _stat module to multiphase initialization #19798 bpo-1635741: Fix compiler warning in _stat.c #19822 bpo-1635741: Port syslog module to multiphase initialization #19907 bpo-1635741: Port errno module to multiphase initialization #19923 bpo-1635741: Port fcntl module to multiphase initialization #20540 [WIP] bpo-1635741: Py_Finalize() finalizes builtin static types #20763 bpo-1635741: Port nis module to multiphase initialization. #20808 bpo-1635741: Port _dbm module to multiphase initialization #20848 bpo-1635741: Port _gdbm module to multiphase initialization #20920 bpo-1635741: Enable unicode_release_interned() without insure or valgrind. #21087 bpo-1635741: Port termios module to multiphase initialization #21168 bpo-1635741: Port sha256 module to multiphase init (PEP 489) #21189 bpo-1635741: Release Unicode interned strings at exit #21269 bpo-1635741: Fix unicode_dealloc() for mortal interned string #21270 bpo-1635741: Port faulthandler module to multiphase initialization #21294 bpo-1635741 - try to port pickle to multi-stage init #21319 bpo-1635741: port winapi to multi-stage init #21371 bpo-1635741 unicodedata #21375 bpo-1635741: port multiprocessing to multiphase init #21378 bpo-1635741: prepare for multi-phase init for unicodedata #21418 bpo-1635741: Clean sysdict and builtins of interpreter #21605 bpo-1635741: Port hashlib modules to multiphase init (PEP 489) #21818 bpo-1635741: port sha3 module to multi-phase init #21855 bpo-1635741: port blake2 module to multi-phase init #21856 bpo-1635741: Fix refleaks of encodings module by removing the encodings._aliases #21896 bpo-1635741: Fix reference cycle by calling explicit gc collection in main interpreter #21902 bpo-1635741 port some simple modules to multi-phase initialization #21985 bpo-1635741 port _curses_panel to multi-phase init (PEP 489) #21986 bpo-1635741 port zlib module to multi-phase init #21995 bpo-1635741 port _testbuffer to multi-phase init #22003 bpo-1635741 port signalmodule to multi-phase init (PEP 489) #22049 bpo-1635741: port opcode module to multi-phase init (PEP 489) #22050 bpo-1635741: - port overlapped to multi-phase #22051 bpo-1635741: testbuffer heap types #22131 bpo-1635741: sha256 heap types #22134 bpo-1635741: Port the termios to multi-phase init (PEP 489) #22139 bpo-1635741 - port unicodedata to multi-phase init #22145 bpo-1635741: Port _string module to multi-phase init #22148 bpo-1635741: Port mashal module to multi-phase init #22149 bpo-1635741: port scproxy to multi-phase init #22164 bpo-1635741 port cmath to multi-phase init (pep 489) #22165 bpo-1635741 port _lsprof to multi-phase init (PEP 489) #22220 bpo-1635741: port pyexpat to multi-phase init (PEP 489) #22222 bpo-1635741: convert an _lsprof method to argument clinic #22240 bpo-41790: update error in documentation #22242 bpo-1635741: replace UCD_Check with Py_IS_TYPE #22328 bpo-1635741: Port _bisect module to multi-phase init #22415 bpo-41861: Convert _sqlite3 CursorType and ConnectionType to heap types #22478 bpo-1635741: pyexpat error handling #22489 bpo-1635741: convert unicode ucd type to heap type #22490 bpo-1635741: Add a global module state to unicodedata #22712 bpo-1635741: Move _PyUnicode_Name_CAPI to the internal C API #22713 bpo-1635741: Port _csv to multi-phase init #22838 bpo-1635741: Fix NULL ptr deref in multiprocessing #22880 gh-103092: Prep curses module for multi-phase init #23091 bpo-1635741: Add PyModule_AddObjectRef() function #23122 bpo-1635741: _ast uses PyModule_AddObjectRef() #23146 bpo-1635741: _contextvars uses PyModule_AddType() #23147 bpo-1635741: _sqlite3 uses PyModule_AddObjectRef #23148 bpo-1635741: _warnings uses PyModule_AddObjectRef() #23151 bpo-1635741: sqlite3 now uses Py_NewRef and Py_XNewRef #23170 bpo-1635741: Enhance _datetime error handling #23139 bpo-1635741: port pickle to multi-phase init (PEP 489) #23188 bpo-1635741: Fix typo in PyModule_AddObjectRef() doc #23234 bpo-40170: Fix PyType_Ready() refleak on static type #23236 bpo-1635741: In pickle module, inject module state from class methods #23304 bpo-1635741: Port _hashlib to multiphase initialization (GH-23358) #23358 bpo-1635741: Port _random to multiphase initialization #23359 bpo-1635741: Port grp and pwd to multiphase initialization #23360 bpo-1635741: Port symtable module to multiphase initialization #23361 bpo-1635741: Port _queue to multiphase initialization #23376 bpo-1635741: Port gc module to multiphase initialization #23377 bpo-1635741: Convert _imp to multi-phase init #23378 bpo-1635741: Port _warnings to the multi-phase init #23379 bpo-1635741: Port _random to the multi-phase init #23381 bpo-1635741: Port spwd to multiphase initialization (GH-23390) #23390 bpo-1635741: Convert _sre types to heap types and establish module state (PEP 384) #23393 bpo-1635741: Port _struct to multiphase initialization #23398 Revert "bpo-1635741: Port _struct to multiphase initialization" #23401 bpo-1635741: Fix _struct for build bot error #23402 bpo-1635741: Port _posixshmem extension module to multiphase initialization #23404 bpo-1635741: Port _posixsubprocess module to multiphase init (GH-23406) #23406 bpo-1635741: Port select module to multiphase init (GH-23409) #23409 bpo-1635741: Port resource extension module to module state (GH-23462) #23462 bpo-42327: Add PyModule_Add() (smaller). #23443 bpo-42327: Add PyModule_Add() (smaller). #23443 bpo-1635741: port _elementtree to multi-phase init (PEP 489) #23535 Add symbols of the stable ABI to python3dll.c #23598 bpo-1635741: Refactor _threadmodule.c #23793 bpo-1635741: Port _thread to multiphase init #23811 bpo-1635741: Fix ref. leak introduced by bf64d90 #23972 bpo-1635741: Adapt _multibytecodec to multi-phase initialization #24095 bpo-1635741: Convert socket.SocketType to heap type, and establish global state #24175 bpo-1635741: Fix PyModule_AddObjectRef to use EXPORT_FUNC #24205 bpo-1635741: Port _datetime extension module to multiphase initialization (PEP 489) #30522 bpo-1635741: port _ctypes extension module to multiphase initialization  #30525 bpo-46417: Call _PyDebug_PrintTotalRefs() later #30744 bpo-1635741: test_embed cheks that Python does not leak #31555 bpo-1635741: Fix winreg reference leaks #31560

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2022-01-27.13:42:43.202>
created_at = <Date 2007-01-15.10:26:05.000>
labels = ['expert-subinterpreters', 'interpreter-core', '3.11', 'performance']
title = "Py_Finalize() doesn't clear all Python objects at exit"
updated_at = <Date 2022-02-25.11:34:18.230>
user = 'https://bugs.python.org/kylotan'

bugs.python.org fields:

activity = <Date 2022-02-25.11:34:18.230>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2022-01-27.13:42:43.202>
closer = 'vstinner'
components = ['Interpreter Core', 'Subinterpreters']
creation = <Date 2007-01-15.10:26:05.000>
creator = 'kylotan'
dependencies = []
files = []
hgrepos = []
issue_num = 1635741
keywords = ['patch']
message_count = 185.0
messages = ['61054', '110895', '111024', '130729', '248761', '355187', '355189', '355191', '355193', '355194', '355201', '359342', '359482', '359830', '360063', '361427', '361428', '361466', '361798', '362068', '362124', '362143', '362144', '362195', '363935', '363936', '363937', '363940', '363941', '364234', '364330', '364379', '364463', '364521', '364609', '364656', '364833', '364836', '364845', '364871', '364883', '364906', '364909', '364951', '364952', '364968', '364971', '364972', '364973', '364975', '364987', '365008', '365017', '365043', '365386', '365388', '365484', '365584', '365611', '367686', '367796', '368096', '368318', '370561', '370763', '371119', '371569', '371683', '372061', '372077', '372096', '372102', '372103', '372797', '372912', '372938', '372942', '375169', '375272', '375286', '375287', '375288', '375290', '375311', '375567', '376218', '376221', '376240', '376288', '376360', '376461', '376474', '376477', '376480', '376493', '376500', '376552', '376554', '376555', '376556', '376566', '376567', '376571', '376572', '376616', '376687', '376993', '377253', '377364', '377365', '377523', '378115', '378685', '379280', '379287', '379667', '380216', '380270', '380280', '380317', '380318', '380331', '380332', '380340', '380346', '380351', '380733', '380758', '380854', '380857', '381124', '381300', '381317', '381318', '381351', '381352', '381376', '381379', '381391', '381392', '381395', '381396', '381400', '381411', '381423', '381426', '381431', '381432', '381434', '381476', '381477', '381571', '381572', '383155', '383168', '383276', '383283', '383285', '383287', '383454', '383636', '383785', '383799', '383801', '383844', '383847', '383880', '384039', '384042', '384326', '384352', '384646', '384978', '384987', '385010', '388728', '410809', '411116', '411278', '411311', '411872', '411961', '411997', '413938', '413979']
nosy_count = 26.0
nosy_names = ['amaury.forgeotdarc', 'orsenthil', 'kylotan', 'vstinner', 'christian.heimes', 'tlesher', 'phsilva', 'petr.viktorin', 'ysj.ray', 'santoso.wijaya', 'lukasz.langa', 'python-dev', 'eric.snow', 'serhiy.storchaka', 'pconnell', 'isoschiz', 'Jim Fasarakis-Hilliard', 'corona10', 'miss-islington', 'shihai1991', 'erlendaasland', 'koubaa', 'h-vetinari', 'TCsaba', 'kumaraditya', 'CharlieZhao']
pr_nums = ['17835', '17883', '18030', '18032', '18049', '18050', '18065', '18066', '18358', '18365', '18374', '18404', '18486', '18608', '18613', '19012', '19015', '19018', '19022', '19021', '19044', '19057', '19069', '19071', '19074', '19084', '19100', '19107', '19122', '19128', '19135', '19140', '19150', '19151', '19242', '19243', '19252', '19307', '19382', '19459', '19798', '19822', '19907', '19923', '20540', '20763', '20808', '20848', '20920', '21087', '21168', '21189', '21269', '21270', '21294', '21319', '21371', '21375', '21378', '21418', '21605', '21818', '21855', '21856', '21896', '21902', '21985', '21986', '21995', '22003', '22049', '22050', '22051', '22131', '22134', '22139', '22145', '22148', '22149', '22164', '22165', '22220', '22222', '22240', '22242', '22328', '22415', '22478', '22489', '22490', '22712', '22713', '22838', '22880', '23091', '23122', '23146', '23147', '23148', '23151', '23170', '23139', '23188', '23234', '23236', '23304', '23358', '23359', '23360', '23361', '23376', '23377', '23378', '23379', '23381', '23390', '23393', '23398', '23401', '23402', '23404', '23406', '23409', '23462', '23443', '23443', '23535', '23598', '23793', '23811', '23972', '24095', '24175', '24205', '30522', '30525', '30744', '31555', '31560']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue1635741'
versions = ['Python 3.11']

[kylotan]

This C code:

#include <Python.h>
int main(int argc, char *argv[])
{
    Py_Initialize(); Py_Finalize();
    Py_Initialize(); Py_Finalize();
    Py_Initialize(); Py_Finalize();
    Py_Initialize(); Py_Finalize();
    Py_Initialize(); Py_Finalize();
    Py_Initialize(); Py_Finalize();
    Py_Initialize(); Py_Finalize();
}

Produces this output:
[7438 refs]
[7499 refs]
[7550 refs]
[7601 refs]
[7652 refs]
[7703 refs]
[7754 refs]

A similar program configured to call the Py_Initialize()/Py_Finalize() 1000 times ends up with:
...
[58295 refs]
[58346 refs]
[58397 refs]

This is with a fresh debug build of Python 2.5.0 on Windows XP, using Visual C++ 2003.

[BreamoreBoy]

Does the title of this issue accurately reflect the current status of the Python interpreter?

[amauryfa]

Yes, some objects are not cleaned in finalization.
This is not a problem in usual cases though, when the interpreter is
started only once.

[ysjray]

Does the title of this issue accurately reflect the current status of the Python interpreter?

Yes, here is the running result on current 3.3 latest code:
[37182 refs]
[39415 refs]
[41607 refs]
[43799 refs]
[45991 refs]
[48183 refs]
[50375 refs]

This seems to be a known bug that Py_Finalize() doesn't free all objects according doc http://docs.python.org/dev/c-api/init.html?highlight=py_finalize#Py_Finalize

[AlexBudovski]

Interestingly enough, some of the leaked memory came from the finalize routine itself! Here's one example:

0:004> !heap -p -a 0x000000DB144346F0
address 000000db144346f0 found in
_HEAP @ db0cae0000
HEAP_ENTRY Size Prev Flags UserPtr UserSize - state
000000db14434690 030a 0000 [00] 000000db144346c0 03074 - (busy)
7ffc55628b04 ntdll!RtlpCallInterceptRoutine+0x0000000000000040
7ffc555f9f36 ntdll!RtlAllocateHeap+0x0000000000079836
7ffc2a60c4da ucrtbased!calloc_base+0x000000000000123a
7ffc2a60c27d ucrtbased!calloc_base+0x0000000000000fdd
7ffc2a60f34f ucrtbased!malloc_dbg+0x000000000000002f
7ffc2a60fdde ucrtbased!malloc+0x000000000000001e
5a5e6ef9 python36_d!_PyMem_RawMalloc+0x0000000000000029
5a5e78c7 python36_d!_PyMem_DebugAlloc+0x0000000000000087
5a5e5e6f python36_d!_PyMem_DebugMalloc+0x000000000000001f
5a5e7230 python36_d!PyMem_Malloc+0x0000000000000030
5a582047 python36_d!new_keys_object+0x0000000000000077
5a57f7c5 python36_d!dictresize+0x0000000000000085
5a57a4b2 python36_d!PyDict_Merge+0x0000000000000112
5a57bf33 python36_d!PyDict_Update+0x0000000000000023
5a75fb1d python36_d!PyImport_Cleanup+0x000000000000045d
5a778f9e python36_d!Py_Finalize+0x000000000000005e

[vstinner]

I tested on the master branch of Python:
---

#include <Python.h>

void func()
{
    Py_Initialize(); Py_Finalize();
    Py_ssize_t cnt = _Py_GetRefTotal();
    printf("sys.gettotalrefcount(): %zd\n", cnt);
}

int main(int argc, char *argv[])
{
    Py_SetProgramName(L"./_testembed");
    for (int i=0; i < 10; i++) {
        func();
    }
}

---

Each iteration leaks around 5,000 Python objects:
---
sys.gettotalrefcount(): 15113
sys.gettotalrefcount(): 19527
sys.gettotalrefcount(): 23941
sys.gettotalrefcount(): 28355
sys.gettotalrefcount(): 32769
sys.gettotalrefcount(): 37183
sys.gettotalrefcount(): 41597
sys.gettotalrefcount(): 46011
sys.gettotalrefcount(): 50425
sys.gettotalrefcount(): 54839
---

[vstinner]

I marked bpo-6741 as a duplicate of this issue.

[vstinner]

I marked bpo-26888 as a duplicate of this issue.

[vstinner]

I marked bpo-21387 as a duplicate of this issue.

[vstinner]

One part of this issue is that all C extensions of the stdlib should be updated to implement the PEP-489 "Multi-phase extension module initialization".

[vstinner]

I marked bpo-32026 as a duplicate of this issue.

[shihai1991]

One part of this issue is that all C extensions of the stdlib should be updated to implement the PEP-489 "Multi-phase extension module initialization".

I try to port _json extension module to multiphase initialization module, but the baseline(using victor's code) in my vm not changed~

[shihai1991]

Compare to _Py_ForgetReference(), _Py_INC_REFTOTAL in _Py_NewReference() looks redundant.

REF: https://github.com/python/cpython/blob/master/Include/object.h#L442

master brach baseline in my vm:

sys.gettotalrefcount(): 18049
sys.gettotalrefcount(): 22463

after PR17883

sys.gettotalrefcount(): 17589
sys.gettotalrefcount(): 22000

[shihai1991]

FWIW, i counted the difference of each file's refs after Py_Finalize().

[('Objects/dictobject.c', 21434), ('Python/marshal.c', 8135), ('Objects/codeobject.c', 6245), ('Objects/listobject.c', 6037), ('Objects/tupleobject.c', 4169), ('Objects/boolobject.c', 2433), ('Objects/object.c', 2364), ('Objects/unicodeobject.c', 1541), ('Objects/longobject.c', 1387), ('Objects/funcobject.c', 528), ('Objects/classobject.c', 528), ('Objects/abstract.c', 463), ('Python/structmember.c', 369), ('./Include/objimpl.h', 277), ('Objects/stringlib/partition.h', 273), ('Python/import.c', 259), ('Python/codecs.c', 197), ('./Modules/signalmodule.c', 61), ('./Modules/_threadmodule.c', 59), ('Objects/exceptions.c', 15), ('Objects/bytesobject.c', 5), ('./Modules/_weakref.c', 4), ('Python/_warnings.c', 3), ('./Modules/timemodule.c', 1), ('./Modules/_codecsmodule.c', 1), ('Objects/bytearrayobject.c', 1), ('Python/compile.c', 1), ('Objects/sliceobject.c', 0), ('Objects/memoryobject.c', 0), ('Python/context.c', -1), ('Objects/clinic/longobject.c.h', -1), ('Objects/enumobject.c', -1), ('Modules/gcmodule.c', -1), ('Objects/namespaceobject.c', -1), ('Objects/stringlib/unicode_format.h', -2), ('Objects/rangeobject.c', -3), ('Python/pystate.c', -4), ('Objects/fileobject.c', -14), ('./Modules/_io/clinic/bufferedio.c.h', -17), ('./Modules/_io/iobase.c', -21), ('Python/modsupport.c', -28), ('./Modules/_io/fileio.c', -28), ('Python/pylifecycle.c', -37), ('./Modules/_io/textio.c', -39), ('Objects/genobject.c', -53), ('Objects/weakrefobject.c', -54), ('./Modules/_io/bufferedio.c', -56), ('./Python/sysmodule.c', -68), ('./Modules/_io/_iomodule.c', -82), ('Python/errors.c', -90), ('Objects/descrobject.c', -110), ('Objects/structseq.c', -113), ('Python/bltinmodule.c', -118), ('Objects/setobject.c', -339), ('Objects/moduleobject.c', -454), ('./Modules/posixmodule.c', -614), ('./Modules/_abc.c', -664), ('Objects/call.c', -755), ('Objects/typeobject.c', -2035), ('Objects/frameobject.c', -6538), ('Python/ceval.c', -7857), ('./Include/object.h', -48292)]

[vstinner]

New changeset ed154c3 by Victor Stinner (Hai Shi) in branch 'master':
bpo-1635741: Port _json extension module to multiphase initialization (PEP-489) (GH-17835)
ed154c3

[shihai1991]

i thinkt that not checking PyModule_AddObject()'s result may cause this probleam too.

1. python-ast.c have one question, i fix it in PR18358.
2. most of the questions in extension module, for example: https://github.com/python/cpython/blob/master/Modules/gcmodule.c#L2019-L2022

[shihai1991]

update the above info:

1. python-ast.c have one question, i fix it in PR18365.

[shihai1991]

1. python-ast.c have one question, i fix it in PR18365.
2. most of the questions in extension module, for example: https://github.com/python/cpython/blob/master/Modules/gcmodule.c#L2019-L2022

brandt does relevant work already in PR17276、PR38823.

[miss-islington]

New changeset 1ea45ae by Hai Shi in branch 'master':
bpo-1635741: Port _codecs extension module to multiphase initialization (PEP-489) (GH-18065)
1ea45ae

[shihai1991]

Leave a note for myself:
I check the remaining object roughly(though dump_refs function), most of remaining object is 'str', such as:
'0x7f779cf88880 [13] str'->'0x7f779cf88880 [26] str'

So far, I don't know which file and fileno create those object. MAYBE I need find a hack way to sign this mallocing operation?(not sure)

[vstinner]

New changeset b2b6e27 by Hai Shi in branch 'master':
bpo-1635741: Port _crypt extension module to multiphase initialization (PEP-489) (GH-18404)
b2b6e27

[vstinner]

New changeset 7d79568 by Hai Shi in branch 'master':
bpo-1635741: Port _contextvars module to multiphase initialization (PEP-489) (GH-18374)
7d79568

[vstinner]

I wrote PR 20763 to "finalize" static types in Py_Finalize(). It mostly works, but "./Programs/_testembed test_forced_io_encoding" crash. (...)

I created bpo-46417 follow-up issue: "[subinterpreters] Clear static types in Py_Finalize()".

[vstinner]

Using "./python -IsS" command, sys.modules now only has 3 extensions which are not created by PyModuleDef_Init():

• builtins
• _io
• sys

The builtins and sys extensions use many static types. Converting these static types to heap types is blocked by bpo-40601. Soon, bpo-46417 will clear these types. Py_Finalize() does its best to clear explicitly builtins and sys namespaces.

[vstinner]

In bpo-46417, I pushed a change to call _PyDebug_PrintTotalRefs() after destroying the interpreter. I backported locally this change to other Python versions to compare the progress on old Python versions:

./python -I -X showrefcount -c pass command:

• Python 3.7: [22636 refs, 7453 blocks]
• Python 3.8: [18082 refs, 5867 blocks]
• Python 3.9: [17786 refs, 5766 blocks]
• Python 3.10: [10725 refs, 3978 blocks]
• current main branch: [2957 refs, 1240 blocks]

Yeah, main is way better than Python 3.10! I pushed many changes in
bpo-46417 to clear most static types, it helps a lot!

[vstinner]

My work on bpo-46417 reduces the number of leak references from around 10k to ... minus 4 references :-) Clearing static types and a few "static" objects helped a lot!

New changeset a1444f4 by Victor Stinner in branch 'main':
bpo-46417: Fix _PyStaticType_Dealloc() (GH-30810)
a1444f4

At commit a1444f4, I get:

    $ ./python -I -X showrefcount -c pass
    [-4 refs, 61 blocks]

I have to investigate why it's negative. It may be caused by bpo-46449 issue.

See also: https://bugs.python.org/issue46417#msg411307

[vstinner]

Thanks to recent enhancements, epecially in bpo-46417, the last memory blocks are now released at Python exit! The initial issue has been fixed!!! This bug was 15 years old! Current main branch:

$ ./python -I -X showrefcount -c pass
[-5 refs, 0 blocks]

"0 blocks" means that Python no longer leaks memory at exit. You can use Valgrind to check it ;-)

The negative reference count is being discussed in bpo-46449.

I close this issue since it has an insane history: too many pull requests, changes, etc.

While the work is not 100% done (we need to convert remaining static types to heap types, and convert extensions to multi-phase init), I prefer to address remaining issues in (existing or new) separated issues. See for example the bpo-40077: "Convert static types to heap types: use PyType_FromSpec()".

I would like to thank everybody who helped to fix this issue! It's hard to list all names since this issue is a meta issues made of many sub-issues which also have sub-issues. Authors of PEP-573 and PEP-630 also helped fixing this issue!

[vstinner]

I marked bpo-35774 as a duplicate of this issue.

[vstinner]

The negative refcount issue has been fixed ;-)

At commit 9a24127, Python now leaks exactly 0 reference count and 0 memory block at exit:

$ ./python -I -X showrefcount -c pass
[0 refs, 0 blocks]

[vstinner]

New changeset c9c178f by Victor Stinner in branch 'main':
bpo-1635741: test_embed cheks that Python does not leak (GH-31555)
c9c178f

[vstinner]

New changeset 4657bf7 by Victor Stinner in branch 'main':
bpo-1635741: Fix winreg reference leaks (GH-31560)
4657bf7

[AlexSoft73]

I have teste Python 3.11 "3.11.0rc1 (main, Aug 8 2022, 11:30:54) [MSC v.1932 64 bit (AMD64)]" in Win10 64 bits and it leaks around 2Mb each time you call
Py_IsInitialized();
Py_FinalizeEx();

[vstinner]

This issue is closed, please open a new issue.

[vstinner]

I have teste Python 3.11 "3.11.0rc1 (main, Aug 8 2022, 11:30:54) [MSC v.1932 64 bit (AMD64)]" in Win10 64 bits and it leaks around 2Mb each time you call

Ah, it seems like it has been reported as a new issue: #96853