Jupyter notebooks after a while become very slow

[drkg4b]

Bug: Execution of code in Jupyter mode (when using '#%%' to start a cell) after a while becomes painfully slow and laggy and require vscode restart to get back to normal speed.

Steps to cause the bug to occur

1. Just use the editor for a while

Actual behavior

After running for a random amount of time, execution of commands is heavily lagged and require complete restart of the vscode editor

Expected behavior

Smooth operation of the editor without lag or slower than necessary execution of commands.

Your Jupyter and/or Python environment

Please provide as much info as you readily know

• Jupyter server running: Local
• Extension version: 2020.3.71659
• VS Code version: 1.43.2
• Setting python.jediEnabled: false
• Python version: 3.7.4
• OS: Windows 10
• Virtual environment: N/A/ sometimes venv.

Microsoft Data Science for VS Code Engineering Team: @rchiodo, @IanMatthewHuff, @DavidKutu, @DonJayamanne, @greazer

[IanMatthewHuff]

@drkg4b Thanks for reporting and sorry about the issue. We do have some performance issues that we are tracking, but if you have any more information about your scenario it might be helpful to know if this would be a similar issue that you are seeing or not. Can you expand a bit on:

Just use the editor for a while

Anything in particular that you are working with here? Like the type of code that you are working with? And is it particular cells that cause issues? Or just everything slows down over time? Knowing that would help us to diagnose things.

[drkg4b]

@IanMatthewHuff Hi Ian, thanks for getting back to me and for all the hard work! The issue actually started about a year ago, sorry can't remember the exact version, but I hoped it would resolve.

The issue happens during normal workflow. Normal python using data science libraries (numpy, sklearn...) even working with small datasets (2K, 3K rows). After a while, there doesn't appear to be a specific trigger, all just starts to lag and the only solution I found is to close and re-open vscode. Restarting the IPython kernel is not enough.

Please, do let me know how I can help provide more information. I know that all my colleagues also experience the same issue.

[IanMatthewHuff]

@drkg4b I asked around on the team and I do have one possible workaround to try. When you first start up a notebook could you try running the following in a cell?
%config Completer.use_jedi = False
This isn't a real fix (it just turns off jedi code analysis in jupyter) but we have a suspicion that this might be the cause of some of the long running perf problems. If you try this and you don't see the perf slowdowns that would hopefully unblock you for now and give us a good data point if this is the issue.

[drkg4b]

@IanMatthewHuff Sorry I took a couple of days to give the workaround a proper test. Unfortunately I can still see the issue.

Perhaps I should mention that I work with python files (with the .py and not the .ipynb) so the syntax checker complains a bit about the %config Completer.use_jedi = False bit and I'm not totally sure it's executed.

What information could I provide to help you? Any logs?

[IanMatthewHuff]

Yeah, if you could. When things are running slow for your can you go to "Help->Toggle Developer Tools" in VSCode. Then in that developer tools panel go the "Console" tab and save out the log here. I think that if you right click in the console tab you'll have an option to just save the whole thing to a file. Then back here you can just make a new comment and attach that file by dragging it below the comment box. That log might have some info that can help us with diagnosing.

[greazer]

Dupe of microsoft/vscode-python#10971

[Viynne]

Hello,
I recently started using Visual Studio code and I have enjoyed the experience until today.
I started to have the same issue with my vscode slowing down so much, I work with .ipynb files and after a while I barely get predictions, and outputs come out really slow.
I also realized that my vscode makes my files so large, compared to working with the conventional jupyter notebook on a browser.
The same file I had just run on jupyter notebook with a size of 696KB, when I run it on vscode it gets up to 53,881KB.
Please why's this happening?

[DonJayamanne]

@Viynne
Thanks for getting this issue, please could you file a new issue as you are experiencing a slightly different issue.

[rchiodo]

@Viynne, I would hazard a guess your issue is this one:
https://github.com/microsoft/vscode-python/issues/12658

We generate SVGs for all plots so that we have something to open in our plot viewer that scales well. You can disable this feature (and the SVG generation) but setting this value:

"python.dataScience.enablePlotViewer": false

[Viynne]

@Viynne
Thanks for getting this issue, please could you file a new issue as you are experiencing a slightly different issue.

Okay thank you very much for your response. I'll recreate this as a new issue.

[Viynne]

@Viynne, I would hazard a guess your issue is this one:
#459

We generate SVGs for all plots so that we have something to open in our plot viewer that scales well. You can disable this feature (and the SVG generation) but setting this value:

"python.dataScience.enablePlotViewer": false

Thank you very much!
I'll look into this and get back to you on it.
While I'm at it, I'll rewrite this as a new issue.
I'd really appreciate the support from your team, I do enjoy Visual Studio code, anything to keep using it.

[ZERONE00]

I had a same problem, and I solved it by 'Conda install Jupyter' in my virtual env. Hope it will be helpful ：)

[robertodawid]

Hi, I am running a basic example of plotnine in jupyter notebook in VSC. The code runs perfectly in jupyter-browser, however, VSC is slow and generates very very large file of 112MB.
The example is from
https://datacarpentry.org/python-ecology-lesson/07-visualization-ggplot-python/index.html
What is happening?

[IanMatthewHuff]

@robertodawid This is a known issue. With our plot viewer support we create both png and svg versions of plot outputs, and the svg plot outputs of plotnine in particular can be very dense and large. The best way to avoid this right now would be to turn off the VS Code setting Juypter: Enable Plot Viewer. The clear the outputs from the notebook that you are working with. This should bring the file back to a more reasonable size.

[waterchinap]

May be I can add some more info for this issue:
ipynb format,
I use pandas and read a file about 1M rows(about 4 m in size) into a df,

df.describe()
# this will take very long time, about 10 seconds..

it works fine in jupyter lab web app

[ShadyAbuKalam]

Loading a large UTF-8 text file (~160 MB), makes the next operations very slow (seems to block), even if it's as simple as function dentition.
But it works great inside the web app.

Version: 1.53.2 (user setup)
Commit: 622cb03f7e070a9670c94bae1a45d78d7181fbd4
Date: 2021-02-11T11:48:04.245Z
Electron: 11.2.1
Chrome: 87.0.4280.141
Node.js: 12.18.3
V8: 8.7.220.31-electron.0
OS: Windows_NT x64 10.0.18363
Python: 3.7.9

[salmanea]

I had the same problem. In the settings, I specifically assigned a 'local' server for Jupyter. It solved the problem. It used to be on 'default'.

[DonJayamanne]

@ShadyAbuKalam @salmanea Please could you install VS Code insiders and give this a test now.
We've made a lot of changes that would significantly improve the perf across the board (loading & execution).
Do let us know how this goes.
FYI - you'll need vscode insiders, as the changes have been shipped yet (once you install vscode insiders, please ensure you have the latest jupyter extension installed as well)

[salmanea]

@DonJayamanne
Just tested the insider version.
Seems like that Jupyter extension works much faster and lighter.
The variables are also shown much faster. The stable build struggles to update the variable list. Much slower than for example if you use Spyder IDE. So I think great job guys 👍

A quick side question: why are the variables are now shown down with the terminal and other stuff instead of the top of the interactive window? I think it's a much better idea to have them on the top.

[DonJayamanne]

y are the variables are now shown down with the terminal and other stuff instead of the top of the

This is great feedback, please could you file a separate bug to discuss this.
This was done as part of the overall UI changes to the notebook.

hink it's a much better idea to have them on the top.

Agreed, but lets discuss this separately in a separate issue.

& thanks a lot for testing this & providing your feedback.

[DonJayamanne]

Closing this issue based on the comments from @salmanea

If anyone is still experiencing perf issues, please file a new issue or please feel free to re-open this issu.e
FYI - The latest version with the perf fixes will be made availabel today.