Jupyter cell debugging does not support "step into" the third party library code with "justmycode:false"

[ckxckx]

Jupyter cell debugging does not support "step into" the third party library code with "justmycode:false" in launch.json.

Please fix it !

[Qiuzhuang]

can not step into lib code even setting:
"jupyter.debugJustMyCode": false

[rchiodo]

This is explicitly hardcoded here:

vscode-jupyter/src/client/debugger/jupyter/debuggingManager.ts

Line 299 in 350a17c

justMyCode: true,

Likely because stepping into 3rd party code isn't going to do what you want. It will end up stepping through the display hooks for ipython and not the 3rd party library.

@int19h is there a way to add libs to the 'justMyCode' list?

[int19h]

I don't think we have anything documented for this, but pydevd exposes some environment variables to control this. Presumably you'd want the library to debug to be listed in IDE_PROJECT_ROOTS, but I'm not sure how this will interact with the default exclusion filters - @fabioz should know.

I wonder if perhaps the Jupyter extension should be using the exclusion filters on a lower level to specifically exclude all those display hooks, while leaving "justMyCode" setting to the user?

[rchiodo]

I wonder if perhaps the Jupyter extension should be using the exclusion filters on a lower level to specifically exclude all those display hooks, while leaving "justMyCode" setting to the user?

That sounds interesting too. How do we set exclusion filters?

[int19h]

It's all in the same code linked above, so currently, also via env vars (or using pydevd APIs). But we can always plumb that API through debugpy as needed.

[roblourens]

Thanks, where does the justMyCode flag actually come in, is it implemented using those flags?

[speediedan]

In case it's helpful to others until this is addressed, I've been able to workaround the issue effectively using IDE_PROJECT_ROOTS as suggested but would note one needs to add the appropriate ipykernel temp files directory (/tmp by default on Linux) in addition to the third party library code in IDE_PROJECT_ROOTS, e.g.:

IDE_PROJECT_ROOTS=/tmp:/opt/miniconda/envs/<myenv>/lib/<python_ver>/site-packages/<target third party lib>

otherwise, the 3rd-party breakpoints will work but the other breakpoints you set in your notebook will stop working. Hope this helps!

[jhcsjcao]

Ok. After about 20 hours of trying, I got vscode to step into to dependent python libraries during debugging. I'm running Jupyter works too.

You have to use 'base' python interpreters - NOT python environments in vscode, and set 'justmycode' to false in vscode's launch.json.

My dependent library modules are in a subdirectory of the directory that contains my Jupyter notebook and is the first directory listed when I run 'os.sys.path'

Hope this helps!

[roblourens]

Oops, yep thanks. But does ipykernel import that debugger.py right away?

[fabioz]

Oops, yep thanks. But does ipykernel import that debugger.py right away?

Apparently yes: https://github.com/ipython/ipykernel/blob/main/ipykernel/ipkernel.py#L19

[fabioz]

Note: I also created an issue ipython/ipykernel#1004 for ipykernel to do the right thing here.

[roblourens]

Thanks. I suppose that we can also actually spawn the kernel with the proper environment. But it really seems like the proper thing is for ipykernel to set this flag. Any reason we haven't already brought it up with them? Does Jupyter Lab know about this or have a solution to this?

[fabioz]

Thanks. I suppose that we can also actually spawn the kernel with the proper environment. But it really seems like the proper thing is for ipykernel to set this flag. Any reason we haven't already brought it up with them? Does Jupyter Lab know about this or have a solution to this?

Well, I just brought it up in that ticket (ipython/ipykernel#1004) -- some ipython devs were on microsoft/debugpy#869, so I thought they were up to date on this, but maybe they didn't notice it.

[roblourens]

Don pushed a change in #11682 to spawn the kernel with the environment variable set, and I confirmed that with that change, justMyCode can work perfectly. So we have that for local kernels, and we just have to keep on top of ipython/ipykernel#1004 as well. Thanks @fabioz and @DonJayamanne!

[roblourens]

Hm, this is actually not working for the Interactive Window. When I start debugging, I always start at that same line in interactiveshell.py as above #8146 (comment) and can't hit a breakpoint in the cell code. I confirmed that the fix from yesterday is applying to the IW so I see Using PYDEVD_IPYTHON_COMPATIBLE_DEBUGGING: True. Can't tell why the IW would be different from the notebook here, any idea @DonJayamanne?

[roblourens]

I figured out that this happens with the breakpoint() statement, we inject this into the first line of the code run from the IW, and every time this runs, it breaks in interactiveshell.py. Happens if that statement is anywhere in the cell, when the above environment var is set.

@fabioz do you expect breakpoint() to work correctly here?

[roblourens]

We can probably get that working using a normal breakpoint. I'll open a new issue for that.

[fabioz]

I figured out that this happens with the breakpoint() statement, we inject this into the first line of the code run from the IW, and every time this runs, it breaks in interactiveshell.py. Happens if that statement is anywhere in the cell, when the above environment var is set.

@fabioz do you expect breakpoint() to work correctly here?

When you use breakpoint() what you're doing is asking the debugger to stop as soon as possible in the first opportunity, which isn't really what you want in this case (because the first opportunity is in the library code).

What you can do instead in this case is ask the debugger to stop just in user code.

The api for this is not as simple as just breakpoint() though.

If you know that you want to stop just at the main thread you can do something as:

import debugpy
debugpy.trace_this_thread()

from _pydevd_bundle.pydevd_api import PyDevdAPI
PyDevdAPI().stop_on_entry()

Note that this works just for the main thread (because stop_on_entry() makes the configuration for the main thread). We can create a better API for this use case though (maybe a keyword argument to breakpoint() like breakpoint(stop_on_user_code=True) would be good?)

[sachinruk]

To anyone else coming here looking for an answer

1. You need the pre-release version of Jupyter extensions (as of 12th Nov 2022)

2. Make sure you have the following in your local .vscode/settings.json file.

{
    "jupyter.debugJustMyCode": false,
}

This takes in the solution of the highest rated answer above and puts it into actual extension.

[nofreewill42]

holy jesus christ!
@sachinruk
You're my savior man!!

[nofreewill42]

This is lifechanging!

[nofreewill42]

Alright it's still not the best it could be, but still:
I don't have to create a new python script that I can debug as I wish just because I've started out with a jupyter notebook where doing so is not really possible.

[nofreewill42]

Okay, this is actually falling apart.... oh jeez I thought I'm saved but this is getting uglier and uglier : (

[nofreewill42]

OH YES!!!
This is actually working maaan!!
(just don't soft link your ~/venv to ./.venv in the project directory lol....)

I'm saaaaaaaaaaaaved !!!! :D
I feel like the cows that are released to freedom for the first time in their lives:
https://www.youtube.com/watch?v=jQQTmuOEPLU&ab_channel=TomPembertonFarmLife&t=296