Implement full notebook debugger

[tyoc213]

October milestone

• Implement %pdb support in notebooks #6374
• Implement edit and continue for jupyter cells while debugging #5621
• Native Notebooks - trying to escape %debug breaks notebook #5903
• Is the setting stopOnFirstLineWhileDebugging respected in the new run by line #7263
• Show source for changed or deleted cells #7532
• Can't hit a breakpoint on line following a function definition #7649
• Can't "restart" notebook debug session #7670
• No visual cue something is happening when debugging long running task #7708

Closed

• Notebook debugger crashes on step-over #6574
• Debug state and breakpoint gutter get out of sync #6570
• [Endgame testing] Breakpoints can't be set before debugging #6408
• [Endgame testing] Debugging doesn't feel "native" #6443
• Debugging requires manual ipkernel install (and creates additional file) #6836
• [Endgame testing] Debug console doesn't provide intellisense #6444
• [Endgame Testing] Cannot step into function from one cell above (Could not load source '/tmp/ipykernel_20889/3035911468.py': source unavailable.) #6483
• Starting a debug session on an untitled file opens the save as modal #6557
• Notebook debugger could not load source #7555
• [Endgame testing] Stepping into a built-in python function also leads to a random temp file being opened #6442
• [Endgame testing] stopping at a temp file, debugging the call to a function defined in a diff't cell fails #6413

Feature: Notebook Editor

Original issue description

Been able to debug ipynb cells

I know I can debug "cells" from within a .py file as

Bu there is no way to launch debug like this from a cell in a ipynb

It would love to launch the debugger "normally" with a button, if the breakpoints are not respected in the cells, at less, if they are in other .py that code has access at the moment because "in the working directory" would love that those breakpoints are respected as normal .py.

Microsoft Data Science for VS Code Engineering Team: @rchiodo, @IanMatthewHuff, @DavidKutu, @DonJayamanne, @greazer

[joyceerhl]

Thanks for the feature request! We will discuss this at our weekly triage.

In the meantime, we do have an experimental feature called Run By Line which provides debugger-like functionality for notebook cells, specifically by allowing you to step through a cell line-by-line and watch how variables in the kernel change via the variable panel. You can turn the feature on by opening your settings.json and adding the following value:

	"python.experiments.optInto": [
		"RunByLine - experiment"
	],

[tyoc213]

One interesting thing to see is how to be able to jump to file.py context (open tab if not open?) and still been debugin the cell on .ipynb`.

Or maybe an embedded tab as the variables tab.

[tyoc213]

Is this already implemented or public in latest version of vscode?

[joyceerhl]

@tyoc213 Hi! Full cell debugging is not yet available in the preview native notebooks editor, but Run By Line (a lite debugger that allows you to execute code cells and view variable state line by line) is available in the webview based notebook editor. Run By Line feature parity in the preview native notebooks editor is on our roadmap. This is what Run By Line looks like in the webview based notebook editor:

[nerkulec]

If I were to contribute to help resolve this particular issue, where would I start? I've already read the 'contributing guide'.

[rchiodo]

If I were to contribute to help resolve this particular issue, where would I start? I've already read the 'contributing guide'.

What were you thinking of contributing?

Our plans for debugging are:

• Interactive window will continue to support debug cell - this is working today
• Stable VS code will show the Run by Line support that @joyceerhl outlined above - this is working today
• Insiders VS code will eventually get Run by line as well. - this requires help from the VS code team at the moment.

[nerkulec]

If I were to contribute to help resolve this particular issue, where would I start? I've already read the 'contributing guide'.

What were you thinking of contributing?

Our plans for debugging are:

• Interactive window will continue to support debug cell - this is working today
• Stable VS code will show the Run by Line support that @joyceerhl outlined above - this is working today
• Insiders VS code will eventually get Run by line as well. - this requires help from the VS code team at the moment.

I would love to be able to debug my .ipynb files exactly the same way as you can do with .py files - set a breakpoint, step into functions etc.
I know convering to .py is possible, but i haven't seen an option to convert back to .ipynb. Anyways, even if converting back is possible I think everyone can agree that such workflow would be quite tedious.
Me, the person who raised this issue (and I believe many others) would love to be able to debug .ipynb just the same way as .py without any converting.
In my comment I'm offering help to implement functionality if anyone is working on it (I don't have much experience with OSS or VSCode extensions particularly)

[rchiodo]

Have you tried the run by line support? It's really single cell debugging (you can't step into other cells or into other modules) so not sure if you mean that it isn't sufficient for what you want.

[nerkulec]

Have you tried the run by line support? It's really single cell debugging (you can't step into other cells or into other modules) so not sure if you mean that it isn't sufficient for what you want.

It is not sufficient. The most important part for me is stepping into functions.

[rchiodo]

It is not sufficient. The most important part for me is stepping into functions.

Can you log a bug for this then? We argued back and forth when implementing Run By Line whether people wanted to step out of cells. If we get enough up votes (or if you want to implement stepping into functions) we might change how it works.

[nerkulec]

I don't understand 'log a bug'.
I can share my workflow if it brings something to the discussion:

• I write classes and functions in .py files
• I write experiments/long running code in .ipynb
• Experiment breaks
• (now the annoying part) - I copy a part of the code from .ipynb to a new .py file (could use the automatic export, but still doesn't resolve the issue)
• Debug it there
• Copy changes back to .ipynb

If someone has similar workflow you can leave an upvote to signal that full debugging would make your life easier.

[rchiodo]

I don't understand 'log a bug'.

Create a new issue with your reasons for why run by line is insufficient.

We have no plans to implement 'full' debugging of ipynb files like the xeus kernel supports in jupyter. However if enough people say that Run by line is not what they want, we'll have a reason to implement it.

[tyoc213]

It is not sufficient. The most important part for me is stepping into functions.

Can you log a bug for this then? We argued back and forth when implementing Run By Line whether people wanted to step out of cells. If we get enough up votes (or if you want to implement stepping into functions) we might change how it works.

So the original request was already discarded then?

[rchiodo]

So the original request was already discarded then?

The original request was intended to be handled by the Run by line functionality. If Run by line isn't sufficient, I think it would make more sense to log a different bug explaining why so other people can complain about that specifically.

[tyoc213]

I see, I took the first comment #1652 (comment) as current workaround and though people where working on the request because this issue was not closed.

I think if I make another issue, will be the same than this one "full debug support" because there is no debug from a cell, for example one important feature of full debug is that it will allow to put breakpoints in modules in the current search path that are used on the ipynb.

The workflow #1652 (comment) is an example of the go back and forth between ipynbs and py as a workaround but that takes time and I'm sure that is what ppl normally do when they really want to debug some ipynb cells.

[m-beau]

Hi,

I am another user very much interested by interactive debugging in the jupyter integration of VScode.

I am coming from two different workflows, that I currently cannot reproduce in VScode:

1 - Editing and testing backbone functions with toy datasets in spyder with %pdb ON. When an unhandled exception occurs, 1) spyder's built-in ipython console enters in Python debug mode, allowing to explore variables, 2) spyder's built-in editor opens the file and jumps to the line causing the error (denoting it with a blue arrow), allowing to at a glance see which code was executed just above and edit it on the fly, I wish VScode's editor and terminal emulation were working hand-in-hand as nicely. Currently, there is no dialogue between them - it is possible to run code in an ipython session, and enter debug mode there, but there is no feedback from the console to the editor.

2 - Running experiments on real datasets with jupyter notebooks. Again, with %pdb ON. When an unhandled exception occurs, a dialogue box opens up, allowing the user to explore variables.

This issue refers to the latter - but also having the former added to VScode would be absolutely fantastic.

[m-beau]

Any update on this?

[rchiodo]

This is not being actively worked on. It's under consideration but other things are currently in front of it right now.

We really need to have a roadmap somewhere.

[rchiodo]

I think there's 3 levels of things we can do here:

1. Implement run by line in the native notebook editor for VS code (this is the most likely to happen soon)
2. Implement support for %pdb
3. Implement full debugging support as asked for above.

[m-beau]

Thanks for your answer.

PS: the spyder jupyter notebook extension supports %pdb - no clue if there is some code that can be taken from there, but it just works! https://github.com/spyder-ide/spyder-notebook

[ahmednrana]

About the notebook debugger. Jupyter team has recently implemented it

Xues Python as per the below Jupyter blog can now do this.
https://blog.jupyter.org/a-visual-debugger-for-jupyter-914e61716559

PyCharm Professional provides this out of box.
https://www.jetbrains.com/help/pycharm/running-jupyter-notebook-cells.html#clear-output

[memeplex]

Sorry for asking here, but it's probably better than creating a potentially duplicated issue.

I find it very confusing that I can indeed set a breakpoint in a cell:

But when I run the cell it's not hit and it seems to be no specific option to explicitly debug the cell otherwise.

Is this supposed to work?

[memeplex]

Ok, forget about that, I realized there is an experimental option to enable what's called "Run by line", very nice.

The ability to set useless breakpoints is perplexing anyway, but if this experiment is going to be enabled soon by default maybe there's no point in polishing that rough edge.

[DavidKutu]

Hi @memeplex!

Run by line is now active by default on the latest Jupyter extension and VS Code. jupyter.experimental.debugging is used to enable Debug Cell, which will appear on a dropdown on the run button of each cell. That's when your breakpoints would be used.

The difference between Run by Line and Debug Cell is that Run by Line is very simplified. It only allows to keep running the cell or interrupt, and runs your cell line by line. Debug Cell uses the full debugging interface of VS Code.

Let me know if you have any more questions.

[m-beau]

Hi, there is now a 'full debugger' implemented in VScode jupyter notebooks, but pdb is still not supported - is this on the roadmap? Is there an experimental feature that can be enabled?

Thanks!

[rchiodo]

@m-beau %pdb is not on our backlog at the moment. Please feel free to add a separate issue.

[kaare-mikkelsen]

@m-beau , have you created a pdb issue yet? otherwise I might. I'm also trying to transition from spyder, but letting go of their debugging interface is not really feasible. Right now I have a suboptimal workflow where I write most things in VScode, and then switch to spyder when things break....