Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ImportError: cannot import name 'encodings' from 'psycopg2._psycopg' (again) #915

Open
parmentelat opened this issue Nov 20, 2024 · 9 comments

Comments

@parmentelat
Copy link

this issue resonates with #753
however I have not been able to solve my problem from that issue, so here is a new one

in my case I have a plain fedora41 box freshly installed, trying to run an application that was working fine within fedora40, but that fails in this new environment

I have python3 and python3-psycopg2 installed through dnf from fedora
and as far as the terminal, this works fine, in that I can do

python -c "import psycopg2"

however when invoked through wsgi I am getting the error quoted in the title, and more specifically

Traceback (most recent call last):
  File "/usr/share/plc_api/apache/plc.wsgi", line 35, in application
    api = PLCAPI()
  File "/usr/share/plc_api/PLC/API.py", line 160, in __init__
    from PLC.PostgreSQL import PostgreSQL
  File "/usr/share/plc_api/PLC/PostgreSQL.py", line 15, in <module>
    import psycopg2
  File "/usr/lib64/python3.13/site-packages/psycopg2/__init__.py", line 67, in <module>
    from psycopg2 import extensions as _ext
  File "/usr/lib64/python3.13/site-packages/psycopg2/extensions.py", line 50, in <module>
    from psycopg2._psycopg import (                             # noqa
    ...<5 lines>...
        set_wait_callback, get_wait_callback, encrypt_password, )
ImportError: cannot import name 'encodings' from 'psycopg2._psycopg' (/usr/lib64/python3.13/site-packages/psycopg2/_psycopg.cpython-313-x86_64-linux-gnu.so)
11-20 17:39:41 ERROR plc.wsgi:56 INTERNAL ERROR !!

the apache config reads

<VirtualHost *:443>

   # SSL
   SSLEngine On
   SSLCertificateFile /etc/planetlab/www_ssl.crt
   SSLCertificateKeyFile /etc/planetlab/www_ssl.key
   SSLCertificateChainFile /etc/planetlab/www_ca_ssl.crt

   WSGIScriptAlias /PLCAPI /usr/share/plc_api/apache/plc.wsgi
   WSGIDaemonProcess plcapi-wsgi-ssl user=apache group=apache processes=1 threads=25
   WSGIProcessGroup plcapi-wsgi-ssl

   <Directory "/usr/share/plc_api/apache">
      Options +ExecCGI
      Require all granted
   </Directory>

</VirtualHost>

the puzzling thing is that again this was working fine within fedora40 - which was using python-3.12 instead of 3.13 here, but well that does not seem too relevant..

there is no virtualenv in this box, just one plain system python install

let me know if I can make anything clearer

@GrahamDumpleton
Copy link
Owner

Try changing to use:

   WSGIScriptAlias /PLCAPI /usr/share/plc_api/apache/plc.wsgi
   WSGIDaemonProcess plcapi-wsgi-ssl user=apache group=apache processes=1 threads=25
   WSGIProcessGroup plcapi-wsgi-ssl
   WSGIApplicationGroup %{GLOBAL}

In other words, add within the VirtualHost definition:

   WSGIApplicationGroup %{GLOBAL}

I don't know why but some versions of psycopg2 in the past haven't worked properly when used in a Python sub interpreter.

That extra directive will force the Python WSGI application to run in the main interpreter context of the mod_wsgi daemon process instead of a sub interpreter context.

@parmentelat
Copy link
Author

That did the trick indeed ! thanks a million :)

for the record the app was working just fine in fedora 39, and I ran into this when upgrading to fedora41

here are a few possibly relevant changes:

  • one notable change is the move from python-3.12 to 3.13
  • httpd is 2.4.62 in both cases
  • mod_wsgi moved from 4.9.4-6 to 5.0.0.-6
  • also psycopg2 went from 2.9.6-3 to 2.9.9-7 - that seems a very small move but well just in case...

@OlegGirko
Copy link

I've encountered exactly the same problem. Severity of this bug is quite serious. Essentially, every WSGI application using Django framework stopped working in Fedora 41. To complicate things even more, the error above is masked by another exception there, so I had to spend some time to uncover the real problem. To reproduce it, you just need to have any simple WSGI app that does import psycopg2 and try to run in in Fedora 41.

Although the workaround with WSGIApplicationGroup proposed above solves this problem, I think, it makes sense to dig a little bit deeper.

Running a WSGI app with PSYCOPG_DEBUG environment variable set to 1 in Fedora 40 produces the following log:

[15449] psycopgmodule: initializing psycopg 2.9.9 (dt dec pq3 ext lo64)
[15449] psycopgmodule: configuring libpq libcrypto callbacks 
[15449] psycopgmodule: initializing module constants
[15449] psycopgmodule: initializing module types
[15449] psycopgmodule: initializing datetime module
[15449] psycopgmodule: initializing encodings table
[15449] psycopgmodule: initializing adapters
[15449] psycopgmodule: initializing basic exceptions
[15449] psycopgmodule: initializing sqlstate exceptions
[15449] psycopgmodule: module initialization complete

However, running the same WSGI application with the same environment variable in Fedora 41 produces something different:

[23652] psycopgmodule: initializing psycopg 2.9.9 (dt dec pq3 ext lo64)
[23652] psycopgmodule: configuring libpq libcrypto callbacks 
[23652] psycopgmodule: initializing module constants
[23652] psycopgmodule: initializing module types
[23652] psycopgmodule: initializing datetime module
[23652] psycopgmodule: initializing encodings table
[23652] psycopgmodule: initializing adapters
[23652] psycopgmodule: initializing basic exceptions
[23652] psycopgmodule: initializing psycopg 2.9.9 (dt dec pq3 ext lo64)
[23652] psycopgmodule: configuring libpq libcrypto callbacks 
[23652] psycopgmodule: initializing module constants
[23652] psycopgmodule: initializing module types
[23652] psycopgmodule: initializing datetime module
[23652] psycopgmodule: initializing encodings table
[23652] encodings_init(): already called
[23652] psycopgmodule: initializing adapters
[23652] psycopgmodule: initializing basic exceptions
[23652] psycopgmodule: initializing sqlstate exceptions
[23652] psycopgmodule: module initialization complete
[23652] psycopgmodule: initializing sqlstate exceptions
[23652] sqlstate_errors_init(): already called
[23652] psycopgmodule: module initialization complete

As you can see, during initialising basic exceptions, psycopgmodule starts initialising again. And there is a static variable in the code that indicates whether encodings_init() was already called and if so, doesn't initialise encodings again and produces a message instead.

So, I've added abort() call to the place in the code path that is run if encodings_init() was already called. Here is the resulting stack trace:

#0  __pthread_kill_implementation (threadid=<optimized out>, 
    signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
#1  0x00007f34f585e793 in __pthread_kill_internal (threadid=<optimized out>, 
    signo=6) at pthread_kill.c:78
#2  0x00007f34f5805d0e in __GI_raise (sig=sig@entry=6)
    at ../sysdeps/posix/raise.c:26
#3  0x00007f34f57ed942 in __GI_abort () at abort.c:79
#4  0x00007f34f4228e85 in encodings_init (module=0x7f34f075d120)
    at psycopg/psycopgmodule.c:570
#5  PyInit__psycopg () at psycopg/psycopgmodule.c:1023
#6  0x00007f34f462278c in _PyImport_RunModInitFunc (
    p0=p0@entry=0x7f34f4240d10 <PyInit__psycopg>, 
    info=info@entry=0x7f34c71fcde0, p_res=p_res@entry=0x7f34c71fcd30)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Python/importdl.c:423
#7  0x00007f34f4622373 in import_run_extension (
    tstate=tstate@entry=0x7f34b800e1f0, 
    p0=p0@entry=0x7f34f4240d10 <PyInit__psycopg>, 
    info=info@entry=0x7f34c71fcde0, spec=spec@entry=0x7f34f0742350, 
    modules=<optimized out>)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Python/import.c:1987
#8  0x00007f34f465513a in _imp_create_dynamic_impl (module=<optimized out>, 
    spec=0x7f34f0742350, file=<optimized out>)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Python/import.c:4715
#9  _imp_create_dynamic (module=<optimized out>, args=<optimized out>, 
    nargs=<optimized out>)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Python/clinic/import.c.h:485
#10 0x00007f34f457349b in cfunction_vectorcall_FASTCALL (func=<optimized out>, 
    args=0x7f34f0755c48, nargsf=1, kwnames=<optimized out>)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Objects/methodobject.c:425
#11 0x00007f34f455206a in PyObject_Call (callable=0x7f34f43c1b70, 
    args=0x7f34f0755c30, kwargs=0x7f34f075a580)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Objects/call.c:373
#12 PyCFunction_Call (callable=0x7f34f43c1b70, args=0x7f34f0755c30, 
    kwargs=0x7f34f075a580)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Objects/call.c:381
#13 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, 
    throwflag=<optimized out>)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Python/generated_cases.c.h:1355
#14 0x00007f34f456edd8 in _PyObject_VectorcallTstate (tstate=0x7f34b800e1f0, 
    callable=0x7f34f43d0400, args=0x7f34c71fd3b0, nargsf=<optimized out>, 
    kwnames=0x0)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Include/internal/pycore_call.h:168
#15 object_vacall (tstate=tstate@entry=0x7f34b800e1f0, base=<optimized out>, 
    callable=0x7f34f43d0400, vargs=0x7f34c71fd440)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Objects/call.c:819
#16 0x00007f34f45a59be in PyObject_CallMethodObjArgs (obj=<optimized out>, 
    name=<optimized out>)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Objects/call.c:880
#17 0x00007f34f45a4c10 in import_find_and_load (tstate=0x7f34b800e1f0, 
    abs_name=0x7f34f075a230)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Python/import.c:3683
#18 PyImport_ImportModuleLevelObject (name=0x7f34f075a230, 
    globals=<optimized out>, locals=<optimized out>, fromlist=0x7f34f11f18b0, 
    level=0)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Python/import.c:3765
#19 0x00007f34f4553f43 in import_name (tstate=<optimized out>, 
    frame=<optimized out>, name=0x7f34f075a230, fromlist=0x7f34f11f18b0, 
    level=0x7f34f48f4db0 <_PyRuntime+14000>)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Python/ceval.c:2693
#20 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, 
    throwflag=<optimized out>)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Python/generated_cases.c.h:3201
#21 0x00007f34f461b58b in PyEval_EvalCode (co=0x7f34b8097940, 
    globals=<optimized out>, locals=0x7f34f075a080)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Python/ceval.c:596
#22 0x00007f34f4636104 in builtin_exec_impl (module=<optimized out>, 
    source=0x7f34b8097940, globals=0x7f34f075a080, locals=0x7f34f075a080, 
    closure=<optimized out>)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Python/bltinmodule.c:1145
#23 builtin_exec (module=<optimized out>, args=<optimized out>, 
    nargs=<optimized out>, kwnames=<optimized out>)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Python/clinic/bltinmodule.c.h:556
#24 0x00007f34f4566f27 in cfunction_vectorcall_FASTCALL_KEYWORDS (
    func=0x7f34f439e1b0, args=0x7f34f0759ed8, nargsf=<optimized out>, 
    kwnames=<optimized out>)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Include/cpython/methodobject.h:50
#25 0x00007f34f455206a in PyObject_Call (callable=0x7f34f439e1b0, 
    args=0x7f34f0759ec0, kwargs=0x7f34f075a100)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Objects/call.c:373
#26 PyCFunction_Call (callable=0x7f34f439e1b0, args=0x7f34f0759ec0, 
    kwargs=0x7f34f075a100)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Objects/call.c:381
#27 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, 
    throwflag=<optimized out>)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Python/generated_cases.c.h:1355
#28 0x00007f34f456edd8 in _PyObject_VectorcallTstate (tstate=0x7f34b800e1f0, 
    callable=0x7f34f43d0400, args=0x7f34c71fdb40, nargsf=<optimized out>, 
    kwnames=0x0)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Include/internal/pycore_call.h:168
#29 object_vacall (tstate=tstate@entry=0x7f34b800e1f0, base=<optimized out>, 
    callable=0x7f34f43d0400, vargs=0x7f34c71fdbd0)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Objects/call.c:819
#30 0x00007f34f45a59be in PyObject_CallMethodObjArgs (obj=<optimized out>, 
    name=<optimized out>)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Objects/call.c:880
#31 0x00007f34f45a4c10 in import_find_and_load (tstate=0x7f34b800e1f0, 
    abs_name=0x7f34f0759db0)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Python/import.c:3683
#32 PyImport_ImportModuleLevelObject (name=0x7f34f0759db0, 
    globals=<optimized out>, locals=<optimized out>, fromlist=0x0, level=0)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Python/import.c:3765
#33 0x00007f34f4566f27 in cfunction_vectorcall_FASTCALL_KEYWORDS (
    func=0x7f34f439dd50, args=0x7f34f0755978, nargsf=<optimized out>, 
    kwnames=<optimized out>)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Include/cpython/methodobject.h:50
#34 0x00007f34f455206a in PyObject_Call (callable=0x7f34f439dd50, 
    args=0x7f34f0755960, kwargs=0x7f34f0759f40)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Objects/call.c:373
#35 PyCFunction_Call (callable=0x7f34f439dd50, args=0x7f34f0755960, 
    kwargs=0x7f34f0759f40)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Objects/call.c:381
#36 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, 
    throwflag=<optimized out>)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Python/generated_cases.c.h:1355
#37 0x00007f34f456edd8 in _PyObject_VectorcallTstate (tstate=0x7f34b800e1f0, 
    callable=0x7f34f43d0400, args=0x7f34c71fdfe0, nargsf=<optimized out>, 
    kwnames=0x0)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Include/internal/pycore_call.h:168
#38 object_vacall (tstate=tstate@entry=0x7f34b800e1f0, base=<optimized out>, 
    callable=0x7f34f43d0400, vargs=0x7f34c71fe070)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Objects/call.c:819
#39 0x00007f34f45a59be in PyObject_CallMethodObjArgs (obj=<optimized out>, 
    name=<optimized out>)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Objects/call.c:880
#40 0x00007f34f45a4c10 in import_find_and_load (tstate=0x7f34b800e1f0, 
    abs_name=0x7f34f0759d70)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Python/import.c:3683
#41 PyImport_ImportModuleLevelObject (name=0x7f34f0759d70, 
    globals=<optimized out>, locals=<optimized out>, fromlist=0x7f34f43d4e80, 
    level=0)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Python/import.c:3765
#42 0x00007f34f4566f27 in cfunction_vectorcall_FASTCALL_KEYWORDS (
    func=0x7f34f439dd50, args=0x7f34c71fe2b0, nargsf=<optimized out>, 
    kwnames=<optimized out>)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Include/cpython/methodobject.h:50
#43 0x00007f34f4532bd4 in _PyObject_VectorcallTstate (tstate=0x7f34b800e1f0, 
    callable=0x7f34f439dd50, args=0x7f34c71fe2b0, nargsf=1, kwnames=0x0)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Include/internal/pycore_call.h:168
#44 _PyObject_CallFunctionVa (tstate=0x7f34b800e1f0, callable=0x7f34f439dd50, 
    format=<optimized out>, va=0x7f34c71fe290)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Objects/call.c:552
#45 PyObject_CallFunction (callable=0x7f34f439dd50, format=<optimized out>)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Objects/call.c:574
#46 0x00007f34f45b85a6 in PyImport_Import (module_name=0x7f34f0759d70)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Python/import.c:3950
#47 0x00007f34f463b5e0 in PyImport_ImportModule (
    name=name@entry=0x7f34f424dd16 "psycopg2.errors")
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Python/import.c:3390
#48 0x00007f34f42419f9 in basic_errors_init (module=0x7f34f0744090)
    at psycopg/psycopgmodule.c:673
#49 PyInit__psycopg () at psycopg/psycopgmodule.c:1026
#50 0x00007f34f462278c in _PyImport_RunModInitFunc (
    p0=p0@entry=0x7f34f4240d10 <PyInit__psycopg>, 
    info=info@entry=0x7f34c71fe6f0, p_res=p_res@entry=0x7f34c71fe640)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Python/importdl.c:423
#51 0x00007f34f4622373 in import_run_extension (
    tstate=tstate@entry=0x7f34f42cc850, 
    p0=p0@entry=0x7f34f4240d10 <PyInit__psycopg>, 
    info=info@entry=0x7f34c71fe6f0, spec=spec@entry=0x7f34f115e490, 
    modules=<optimized out>)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Python/import.c:1987
#52 0x00007f34f465513a in _imp_create_dynamic_impl (module=<optimized out>, 
    spec=0x7f34f115e490, file=<optimized out>)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Python/import.c:4715
#53 _imp_create_dynamic (module=<optimized out>, args=<optimized out>, 
    nargs=<optimized out>)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Python/clinic/import.c.h:485
#54 0x00007f34f457349b in cfunction_vectorcall_FASTCALL (func=<optimized out>, 
    args=0x7f34f1b89c78, nargsf=1, kwnames=<optimized out>)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Objects/methodobject.c:425
#55 0x00007f34f455206a in PyObject_Call (callable=0x7f34f2540c70, 
    args=0x7f34f1b89c60, kwargs=0x7f34f1168600)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Objects/call.c:373
#56 PyCFunction_Call (callable=0x7f34f2540c70, args=0x7f34f1b89c60, 
    kwargs=0x7f34f1168600)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Objects/call.c:381
#57 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, 
    throwflag=<optimized out>)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Python/generated_cases.c.h:1355
#58 0x00007f34f456edd8 in _PyObject_VectorcallTstate (tstate=0x7f34f42cc850, 
    callable=0x7f34f254db20, args=0x7f34c71fecc0, nargsf=<optimized out>, 
    kwnames=0x0)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Include/internal/pycore_call.h:168
#59 object_vacall (tstate=tstate@entry=0x7f34f42cc850, base=<optimized out>, 
    callable=0x7f34f254db20, vargs=0x7f34c71fed50)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Objects/call.c:819
#60 0x00007f34f45a59be in PyObject_CallMethodObjArgs (obj=<optimized out>, 
    name=<optimized out>)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Objects/call.c:880
#61 0x00007f34f45a4c10 in import_find_and_load (tstate=0x7f34f42cc850, 
    abs_name=0x7f34f116a2f0)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Python/import.c:3683
#62 PyImport_ImportModuleLevelObject (name=0x7f34f116a2f0, 
    globals=<optimized out>, locals=<optimized out>, fromlist=0x7f34f1bf17a0, 
    level=0)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Python/import.c:3765
#63 0x00007f34f4553f43 in import_name (tstate=<optimized out>, 
    frame=<optimized out>, name=0x7f34f116a2f0, fromlist=0x7f34f1bf17a0, 
    level=0x7f34f48f4db0 <_PyRuntime+14000>)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Python/ceval.c:2693
#64 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, 
    throwflag=<optimized out>)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Python/generated_cases.c.h:3201
#65 0x00007f34f461b58b in PyEval_EvalCode (co=0x7f34b80919c0, 
    globals=<optimized out>, locals=0x7f34f1ba1b00)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Python/ceval.c:596
#66 0x00007f34f4636104 in builtin_exec_impl (module=<optimized out>, 
    source=0x7f34b80919c0, globals=0x7f34f1ba1b00, locals=0x7f34f1ba1b00, 
    closure=<optimized out>)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Python/bltinmodule.c:1145
#67 builtin_exec (module=<optimized out>, args=<optimized out>, 
    nargs=<optimized out>, kwnames=<optimized out>)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Python/clinic/bltinmodule.c.h:556
#68 0x00007f34f4566f27 in cfunction_vectorcall_FASTCALL_KEYWORDS (
    func=0x7f34f25212b0, args=0x7f34f116a9d8, nargsf=<optimized out>, 
    kwnames=<optimized out>)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Include/cpython/methodobject.h:50
#69 0x00007f34f455206a in PyObject_Call (callable=0x7f34f25212b0, 
    args=0x7f34f116a9c0, kwargs=0x7f34f1bf7ec0)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Objects/call.c:373
#70 PyCFunction_Call (callable=0x7f34f25212b0, args=0x7f34f116a9c0, 
    kwargs=0x7f34f1bf7ec0)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Objects/call.c:381
#71 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, 
    throwflag=<optimized out>)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Python/generated_cases.c.h:1355
#72 0x00007f34f456edd8 in _PyObject_VectorcallTstate (tstate=0x7f34f42cc850, 
    callable=0x7f34f254db20, args=0x7f34c71ff450, nargsf=<optimized out>, 
    kwnames=0x0)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Include/internal/pycore_call.h:168
#73 object_vacall (tstate=tstate@entry=0x7f34f42cc850, base=<optimized out>, 
    callable=0x7f34f254db20, vargs=0x7f34c71ff4e0)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Objects/call.c:819
#74 0x00007f34f45a59be in PyObject_CallMethodObjArgs (obj=<optimized out>, 
    name=<optimized out>)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Objects/call.c:880
#75 0x00007f34f45a4c10 in import_find_and_load (tstate=0x7f34f42cc850, 
    abs_name=0x7f34f1168db0)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Python/import.c:3683
#76 PyImport_ImportModuleLevelObject (name=0x7f34f1168db0, 
    globals=<optimized out>, locals=<optimized out>, fromlist=0x7f34f1b883a0, 
    level=0)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Python/import.c:3765
#77 0x00007f34f4553f43 in import_name (tstate=<optimized out>, 
    frame=<optimized out>, name=0x7f34f1168db0, fromlist=0x7f34f1b883a0, 
    level=0x7f34f48f4db0 <_PyRuntime+14000>)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Python/ceval.c:2693
#78 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, 
    throwflag=<optimized out>)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Python/generated_cases.c.h:3201
#79 0x00007f34f461b58b in PyEval_EvalCode (co=0x7f34f1b3a3d0, 
    globals=<optimized out>, locals=0x7f34f1ba1c40)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Python/ceval.c:596
#80 0x00007f34f461b4a8 in exec_code_in_module (
    tstate=tstate@entry=0x7f34f42cc850, name=name@entry=0x7f34f11027f0, 
    module_dict=module_dict@entry=0x7f34f1ba1c40, 
    code_object=code_object@entry=0x7f34f1b3a3d0)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Python/import.c:2625
#81 0x00007f34f4505d56 in PyImport_ExecCodeModuleObject (name=0x7f34f11027f0, 
    co=0x7f34f1b3a3d0, pathname=<optimized out>, cpathname=<optimized out>)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Python/import.c:2668
#82 0x00007f34f4505e0a in PyImport_ExecCodeModuleWithPathnames (
    name=<optimized out>, co=0x7f34f1b3a3d0, pathname=<optimized out>, 
    cpathname=<optimized out>)
    at /usr/src/debug/python3.13-3.13.0-1.fc41.x86_64/Python/import.c:2585
#83 0x00007f34f505c867 in wsgi_load_source (pool=0x7f34b80030c8, 
    r=r@entry=0x7f34b8003140, 
    name=name@entry=0x7f34b800dbd0 "_mod_wsgi_d3946ea1d5222549ca44b2166f702494", exists=exists@entry=0, 
    filename=filename@entry=0x7f34b8004fe0 "/var/www/wsgi-bin/test.wsgi", 
    process_group=0x7f34f5078272 "", 
    application_group=0x7f34b800cf40 "fe80::dc83:db94:2e87:ecf5%enp1s0|/wsgitest/", ignore_system_exit=0) at src/server/mod_wsgi.c:3745
#84 0x00007f34f505ddb4 in wsgi_execute_script (r=0x7f34b8003140)
    at src/server/mod_wsgi.c:4174
#85 0x00005598ef9d13da in ap_run_handler (r=r@entry=0x7f34b8003140)
    at server/config.c:169
#86 0x00005598ef9db286 in ap_invoke_handler (r=0x7f34b8003140)
    at server/config.c:443
#87 0x00005598efa18e93 in ap_process_async_request (r=r@entry=0x7f34b8003140)
    at modules/http/http_request.c:452
#88 0x00005598efa194d3 in ap_process_http_async_connection (c=<optimized out>)
    at modules/http/http_core.c:155
#89 ap_process_http_connection (c=<optimized out>)
    at modules/http/http_core.c:246
#90 0x00005598ef9e21ca in ap_run_process_connection (c=c@entry=0x55991b956d00)
    at server/connection.c:42
#91 0x00007f34f5381c46 in process_socket (thd=thd@entry=0x55991b953af8, 
    p=<optimized out>, sock=<optimized out>, cs=<optimized out>, 
    my_child_num=my_child_num@entry=0, my_thread_num=my_thread_num@entry=0)
    at /usr/src/debug/httpd-2.4.62-2.fc41.x86_64/server/mpm/event/event.c:1086
#92 0x00007f34f5382676 in worker_thread (thd=0x55991b953af8, 
    dummy=<optimized out>)
    at /usr/src/debug/httpd-2.4.62-2.fc41.x86_64/server/mpm/event/event.c:2179
#93 0x00007f34f585c797 in start_thread (arg=<optimized out>)
    at pthread_create.c:447
#94 0x00007f34f58e0594 in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100

If you search for PyInit__psycopg function call in this stack trace, you'll find it twice: in frame 5 and in frame 49. And if you look into frame 48, you'll see that it imports psycopg2.errors module using PyImport_ImportModule("psycopg2.errors") in basic_errors_init module. There is some deep Python 3.13 machinery happening between frame 48 and frame 5, but I presume that for some reason to import psycopg.errors module, psycopg2 is imported again recursively, causing psycopg2._psycopg binary module to get initialised recursively again.

This happens only when run as a WSGI app, I found no other way to reproduce this behaviour outside of WSGI. I've tried to write a simple C wrapper that mimicks WSGI behaviour, but I think, I've oversimplified this behaviour, so it didn't help me to reproduce this error. Hence, I'm pretty sure that this is a problem either with mod_wsgi itself, or in the way it interacts with psycopg2.

@OlegGirko
Copy link

Updating both mod_wsgi and psycopg2 to newest versions (5.0.2 and 2.9.10) doesn't solve this problem. Only workaround with WSGIApplicationGroup works.

I've built psycopg2 RPM package from the same sources the previous version (2.9.9) uses, removing a patch for Python 3.13 support (already merged in 2.9.10) and just bumping the version. There is also another patch applied to build psycopg2 in Fedora, but it's not relevant to this issue: it just removes one test.

There is already mod_wsgi 5.0.2 build available for Fedora, it should reach updates repo in about a week if no serious problems found, so I installed a package from this build.

@GrahamDumpleton
Copy link
Owner

Key thing I mentioned above is that without the WSGIApplicationGroup %{GLOBAL} it would be using a sub interpreter context rather than the main interpreter. So there must be something different in Python 3.13 related to C extension module initialisation which is causing issues with psycopg2 or C extensions in general, in sub interpreters under mod_wsgi. Whether is mod_wsgi not having kept up properly with changes in module initialisation for sub interpreters, or psycopg2, I have no idea.

Am going to call in @vstinner and @tiran to see if they have any ideas.

Unfortunately I am about to go on trip and will have no ability to work on code when away, so if this needs an urgent resolution it needs to all be sorted out by Sunday.

@GrahamDumpleton
Copy link
Owner

BTW, don't know if it makes a difference or not, but in 5.0.1 there was a switch to using PyConfig API. I am not sure if one can build 5.0.0 on Python 3.13 due to other API removals.

Also note that cannot test with mod_wsgi-express, without fiddling the generated config file, as it forces use of the main interpreter context. So I will have to try a few tricks to try this in a Fedora 41 container to easily test it.

@OlegGirko
Copy link

OlegGirko commented Dec 5, 2024

BTW, don't know if it makes a difference or not, but in 5.0.1 there was a switch to using PyConfig API. I am not sure if one can build 5.0.0 on Python 3.13 due to other API removals.

5.0.0 was built for Fedora 41 using this patch:
https://src.fedoraproject.org/rpms/mod_wsgi/blob/071d898c9f8a06f5c6df47fcdf4b98da988ce795/f/883.patch
(this is essentially just a patch from #883).

This patch is not (obviously) applied for building 5.0.2, and installing 5.0.2 doesn't make any difference regarding this issue, as I mentioned above. Upgrading both mod_wsgi and psycopg2 to newest stable versions does not make this issue go away.

@GrahamDumpleton
Copy link
Owner

So the PyConfig changes were effectively back ported in that patch from mod_wsgi 5.0.1 (or more likely that patch was what was also submitted as PR against mod_wsgi). Either way, it is changing to the new APIs which I am wondering may be cause. That said, if it were the change to using those APIs, then there is a chance that should see issue with older Python versions as well unless internally the APIs changed between Python 3.12 and 3.13.

@vstinner
Copy link
Contributor

vstinner commented Dec 5, 2024

To debug the issue, you might try to import psycopg2 in a subinterpreter. Example to run print('Hello') in a subinterpreter:

$ ./python
>>> import _testcapi
>>> _testcapi.run_in_subinterp("print('Hello')")
Hello
0

It's likely that there are import machinery changes related to subinterpreters between Python 3.12 and Python 3.13.

Or maybe the PyConfig change is causing the regression. It's unclear to me how it can explain this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants