Breakpoints don't work in notebooks

[rchiodo]

Environment data

• debugpy version: 1.5.1
• OS and version: Windows 10
• Python version (& distribution if applicable, e.g. Anaconda): Python 3.10.1
• Using VS Code or Visual Studio: VS Code

Actual behavior

See this issue: microsoft/vscode-jupyter#8803 (and this issue: ipython/ipykernel#841)

Debugging a cell in a notebook requires you set a breakpoint on the first line. But only with Python 3.10. With Python 3.9 it doesn't require a breakpoint on the first line.

Steps to reproduce:

1. Create a python 3.10 environment
2. Open the attached notebook in VS code
3. Stick a breakpoint like so:

4. Click on the 'Debug Cell' in the execute button:

The breakpoint should hit.

Looking at the logs, it looks like the breakpoints are sent and verified:

D+00000.672: Client[1] --> {
                 "seq": 5,
                 "type": "request",
                 "command": "setBreakpoints",
                 "arguments": {
                     "source": {
                         "name": "debug_failure.ipynb",
                         "path": "C:\\Users\\aku91\\AppData\\Local\\Temp\\ipykernel_19124\\1275961667.py"
                     },
                     "lines": [
                         4
                     ],
                     "breakpoints": [
                         {
                             "line": 4
                         }
                     ],
                     "sourceModified": false
                 }
             }

D+00000.672: /handling #5 request "setBreakpoints" from Client[1]/
             Server[1] <-- {
                 "seq": 13,
                 "type": "request",
                 "command": "setBreakpoints",
                 "arguments": {
                     "source": {
                         "name": "debug_failure.ipynb",
                         "path": "C:\\Users\\aku91\\AppData\\Local\\Temp\\ipykernel_19124\\1275961667.py"
                     },
                     "lines": [
                         4
                     ],
                     "breakpoints": [
                         {
                             "line": 4
                         }
                     ],
                     "sourceModified": false
                 }
             }

But for some reason the stop event never occurs.

Notebook to repro:
debug_failure.zip

Logs of failure:
failurelogs.zip

Logs of success if I stick a breakpoint on the first line of the cell:

successlogs.zip

[rchiodo]

If you run the same steps with a Python 3.9 environment, the stop event fires.

[fabioz]

I'll investigate.

[rchiodo]

I'll investigate.

Let me know if you need more information on setting it up. I was kinda brief on how to create the environment.

[fabioz]

@rchiodo I did the investigation. This is quite an elusive issue ;)

Apparently ipykernel with Python 3.10 does things differently. Instead of executing the frame contents directly, it just executes line by line, apparently by manipulating frames directly (I haven't really seen their code, so, this is mostly speculation).

They probably do a lot of magic under the hood, so, the frame that's received by the debugger tracing does have the proper frame.f_lineno set, but it's an imperfect frame because the frame.f_code just contains the current line being executed and one of the optimizations of the debugger is running with untraced frames, so, when the frame is called at the first line, it inspect the lines of frame.f_code and sees that it only contains line 1, so, the breakpoint added in line 2 for all purposes in the debugger won't ever be hit in this frame because it's not a part of it and thus the debugger disables tracing in that frame (so, the trace call at line 2 is never received by the debugger because tracing was disabled for that frame).

The best fix for this needs to be done in ipykernel by creating a better frame.f_code version (it could just do a bunch of no-ops for the lines that won't be executed -- the important thing for the debugger is that it does have all lines that may be hit in breakpoints in that context)...

Another fix could be done in the debugger (under a flag) to not do that optimization and always trace all frames from a file if there's any breakpoint in that file -- this means that the debugger will need to trace much more (and would thus be slower), which isn't ideal, but it's something to keep in mind if this can't/won't be fixed by ipykernel (note: we'd need to turn that flag on specifically for the jupyter case as there's no reason for this optimization to be removed on other cases).

A workaround users can use is always doing things within a method and then call it at the end or using try..except. In these cases the execution isn't done line-by line and the debugger works.

[fabioz]

Note: the related code seems to be: https://github.com/ipython/ipython/blob/0f4a73c91db71cde174275337fc5e226acc9dd7d/IPython/core/interactiveshell.py#L3144

[fabioz]

Actually, seeing their code it seems that it works the same in both versions of Python, so, I researched a bit more and the difference is not the frame manipulation per-se.

The issue is that the debugger caches the decision on whether to trace something or not based on a cache key which is: (frame.f_code.co_firstlineno, frame.f_code.co_name, frame.f_code.co_filename) and in Python 3.10 the frame.f_code.co_firstlineno is always the same for all invocations of lines from the interactive shell, whereas in Python 3.9 frame.f_code.co_firstlineno was always equal to frame.f_lineno, so, the cache is made to believe that both invocations are from the same frame when in reality they aren't (I'm still not sure why/how that happens in the IPython codebase... I'll need to research a bit more).

[rchiodo]

@fabioz did you find out anything else? Do you think this is a problem in ipykernel or is it something that Python 3.10 is causing?

[fabioz]

@rchiodo, I haven't finished investigating, so, I'm still not sure who is the culprit for that odd frame.f_code.co_firstlineno -- my plan is tackling this tomorrow.

[fabioz]

After investigating a bit more this really seems to be caused by Python 3.10.

I sent a message to python-dev as this seems like a regression for me (but maybe it isn't and it was done on purpose?).

https://mail.python.org/archives/list/python-dev@python.org/thread/VXW3TVHVYOMXDQIQBJNZ4BTLXFT4EPQZ/

I still don't have a good workaround for having multiple functions all with the same co_firstlineno in the same file...

[rchiodo]

Thanks for the follow up.

[fabioz]

Humm, I think that it may be possible to create a new code object based on that code object but with a new co_firstlineno as a patch for ipykernel... Let me experiment a bit with that.

[fabioz]

I provided a fix to ipython itself (ipython/ipython#13535).

[fabioz]

I ended up doing a fix in the debugger itself for this given that another library (hypothesis) got into the same situation (see comments in the related PR: #851).