"import hashlib" crashes on aarch64 after Finalize and re-initialize

[rdoeffinger]

Crash report

What happened?

Compiling and linking below program results in a crash when run on aarch64 (macOS, Debian VM or Ubuntu 22 VM in Azure for example):

#include <Python.h>

int main(int argc, char *argv[])
{
Py_Initialize();
PyRun_SimpleString("import hashlib\nprint('test1')");
Py_Finalize();
Py_Initialize();
PyRun_SimpleString("import trace\ntrace.Trace().run('import hashlib')\nprint('test2')");
Py_Finalize();
return 0;
}

The trace in the second import is just for better context.
It seems not reproducible on Intel.
On a debug python build it asserts with:
testcase1: ../Python/getargs.c:2052: parser_init: Assertion `parser->kwtuple != NULL' failed.

The backtrace looks like this:
#7 0x0000fffff7906fec [PAC] in _PyArg_UnpackKeywords (
args=args@entry=0xfffff7fec5d8, nargs=nargs@entry=0,
kwargs=kwargs@entry=0x0, kwnames=kwnames@entry=0xfffff6620be0,
parser=parser@entry=0xfffff6f70790 <_parser.16>, minpos=minpos@entry=0,
maxpos=maxpos@entry=1, minkw=minkw@entry=0, buf=buf@entry=0xffffffffe180)
at ../Python/getargs.c:2371
#8 0x0000fffff6f57500 [PAC] in _hashlib_openssl_md5 (module=0xfffff6553d70,
args=args@entry=0xfffff7fec5d8, nargs=nargs@entry=0,
kwnames=kwnames@entry=0xfffff6620be0)
at ../Modules/clinic/_hashopenssl.c.h:343
#9 0x0000fffff780c010 [PAC] in cfunction_vectorcall_FASTCALL_KEYWORDS (
func=0xfffff656c050, args=0xfffff7fec5d8, nargsf=,
kwnames=0xfffff6620be0) at ../Objects/methodobject.c:438
#10 0x0000fffff77b6c18 [PAC] in _PyObject_VectorcallTstate (
tstate=0xfffff7f6e1f0 <_PyRuntime+475736>,
callable=callable@entry=0xfffff656c050, args=args@entry=0xfffff7fec5d8,
nargsf=nargsf@entry=9223372036854775808,
kwnames=kwnames@entry=0xfffff6620be0)
at ../Include/internal/pycore_call.h:92

Looking at the hashlib_openssl_md5 function, the use of static non-const structures seems very suspicious at least.

CPython versions tested on:

3.12

Operating systems tested on:

Linux, macOS

Output from running 'python -VV' on the command line:

Python 3.12.4 (main, Jun 8 2024, 04:51:36) [Clang 15.0.0 (clang-1500.1.0.2.5)]

[diegorusso]

@rdoeffinger thanks for raising the bug, I can take a look at this.

[dz3995]

I have just run into this problem on x64 Windows PC, so I can confirm, that the issue is reproducible on Intel.

In my case I'm running this minimum example script:

import _hashlib

def test():
    return _hashlib.openssl_md5(b"hello world!")

in a similar sequence from embedded CPython:

• Py_Initialize
• Load module with above example and call test function
• Py_Finalize
• Py_Initialize
• Load module with above example and call test function
• Py_Finalize

On the second attempt the same assertion fails on debug build, and NULL pointer crash happens on non-debug build.

The problem seems to be the initialization and cleanup of statically initialized _PyArg_Parser structures. The above example will call (through _PyArg_UnpackKeywords) the parser_init function. This function will check _PyArg_Parser.initialized and when that flag is not set it will eventually set _PyArg_Parser.initialized and _PyArg_Parser.kwtuple.

Later, as a result of Py_Finalize call the function parser_clear will be called on the statically initialized parser. This function will only clean the _PyArg_Parser.kwtuple but will leave the _PyArg_Parser.initialized set.

The next call to parser_init will see that the _PyArg_Parser.initialized is still set even thought the _PyArg_Parser.kwtuple is not initialized this time and left as NULL. This leads to the NULL pointer exception.

CPython version: 3.12.4
Operating system: Windows 10 64bit

[rdoeffinger]

I did not test on Windows.
Only macOS Arm, Linux Arm, Linux x86_64
If you need to call the test function containing use of openssl function (vs. just importing/loading) then it might not be quite the same issue...

[dz3995]

I'm pretty certain that this is the exact same issue just by the location of the assertion that failed:

../Python/getargs.c:2052: parser_init: Assertion 'parser->kwtuple != NULL' failed.

combined with the double initialization. There are more ways to trigger that same symptoms because the static parser initialization is used in many places in the same way.

EDIT:

Just to be sure, I tested on Windows with the script:

import hashlib
print('test1')

and have exactly the same result. The function _hashlib_openssl_md5 is being internally called as a result of just importing hashlib (it can even be seen in the stacktrace from the first post).

[diegorusso]

I confirm that this is still happening on 3.13 and 3.14 (main)

[diegorusso]

I have a fix that basically set to 0 the parser->once.v in the parser_clear. I ran all the tests but I'm not sure if I'm missing something else.
Unfortunately I'll be away for some time and I'm not able to fix it now.

[diegorusso]

This is actually been fixed in the meantime by @neonene with #122481

I tested the code in this issue and now it is working OK. I guess we can close this issue but I leave it to @rdoeffinger to verify.
The fix went into 3.12.5.

[diegorusso]

I think this issue can be closed.

[picnixz]

Closing since the OP did not reply and the original issue is not reproducible anymore.