Support Python 3.11

[fabioz]

I'll start to work on this now that the first beta is out (as it seems Python 3.11 had many internal changes I think this will be a big task).

[juliangilbey]

Thanks for all your work so far on supporting Python 3.11!

This is a followup to my issue and PR on pydevd (fabioz/PyDev.Debugger#238 and fabioz/PyDev.Debugger#239). While the PR fixes the "TODO" noted in the pxd file, it does not address another major issue with Python 3.11: the PyFrameObject structure has completely changed, and so the ctypedef is no longer valid. Indeed, references to frame_obj.f_code in pydevd_frame_evaluator.template.pyx need to be replaced by frame_obj.f_frame.f_code, or so it would appear.

What I don't know how to do is make Cython conditionally include cdefs and the like depending on the Python version; it would appear that #if PY_VERSION_HEX only makes sense to Cython inside a docstring, so I'm somewhat stumped. It would be quite ugly (though perhaps feasible) to have two different pxd files for different Python versions, and I'm sure there is a better way to do it.

One could perhaps have conditional Cythonizing (using Cython's IF facility) and then define macros like F_CODE(frame_obj) to expand to frame_obj.f_code or frame_obj.f_frame.f_code as appropriate. That might be a plausible intermediate solution, but it would require running Cython at compilation time and not shipping compiled files.

Anyway, it is quite possible that this is what is causing a lot of the test failures with pydevd (fabioz/PyDev.Debugger#240).

Best wishes, Julian

[fabioz]

Hello @juliangilbey.

The issue here is that the frame eval mode is not currently available for Python 3.11 at all (as a note, the solution which is also used in other places is using code generation to generate code accordingly for each Python version, but this won't be done until the frame eval mode is actually targeted -- it has a requisite of having the bytecode library available, which has ongoing work to add that support: MatthieuDartiailh/bytecode#103).

So, until the bytecode library is ported, there's no work expected to be done on the frame eval mode.

What's kind of a mystery to me is why you're having compilation errors as the frame eval mode should not be compiled on Python 3.11 (i.e.: it's explicitly skipped in https://github.com/fabioz/PyDev.Debugger/blob/pydev_debugger_2_9_2/setup_pydevd_cython.py#L238).

[juliangilbey]

Hello @fabioz,

Thanks for that really clear explanation! I have been reading through the code this evening quite carefully (before I saw your message) and reached the same conclusion: there's something funny going on with the way I've set up the Debian build and I'll have to look into that. (I think it's because I modified setup.py to explicitly build the extension as setup_pydevd_cython.py wasn't easily working in our build setup. That's tomorrow's job to sort out!)

[juliangilbey]

Also, I was confused that setup.py explicitly lists one of the Cython modules _pydevd_bundle.pydevd_cython as an extension, but ignores _pydevd_frame_eval/pydevd_frame_evaluator.

[int19h]

Since frame-eval is disabled by default for the time being, so it's not a support blocker anymore.

[juliangilbey]

Amazing work - thank you so much all!

[davetapley]

I see this got closed with a few WIP commits, am I correct to assume it wasn't completed, beacuse ⬇️ ?

• Debugging under python 3.11 will skip the breakpoint #1284