Kernel won't start (heartbeat). May be related to use of --system-site-packages when defining virtual environment

[mldot]

Bug: Notebook Editor, Interactive Window, Editor cells

Steps to cause the bug to occur

1. Open vscode and select python interpreter from virtual environment
2. Activate virtual env with ipykernel and jupyter library installed from integrated terminal.
3. Command Pallete > Python > Create New Jupyter Notebook
4. In first cell type print('Hello World') and execute

Actual behavior

Notification shows - Connecting to IPython kernel: Connecting to kernel....
After couple of minutes
'Timed out waiting to get a heartbeat from kernel process'

Expected behavior

Cell in notebook should get executed correctly.

Your Jupyter and/or Python environment

Please provide as much info as you readily know

• Jupyter server running: Local
• Extension version: 2020.8.103604
• VS Code version: 1.48.2
• Setting python.jediEnabled: false
• Setting python.languageServer: None
• Python and/or Anaconda version: 3.8.5 (No Anaconda)
• OS: Windows | Mac | Linux (distro): OSX 10.14.6 (MacOS Mojave)
• Virtual environment: virtualenv

Python Output

mldot-python-debug.log

Microsoft Data Science for VS Code Engineering Team: @rchiodo, @IanMatthewHuff, @DavidKutu, @DonJayamanne, @greazer, @joyceerhl

[mldot]

Same issue when trying to open .py file in Python Interactive window. Log attached.
python-logging-debug.log

[IanMatthewHuff]

@mldot Thanks for filing this and sorry about the issue.

I'm digging through the log now. Can you tell me, where is the python that you expect to be using with this virtualenv located? In the log I'm seeing a couple different paths listed:
~/Dev/Projects/python/ml_master/bin/python
/usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/
/usr/local/opt/python@3.8/bin/python3.8

[mldot]

@IanMatthewHuff : /usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/ is the base python home where HomeBrew installed python3.8.5. The other paths point to symlinks to this location. ml_master is the virtualenv created that has all jupyter related packages including ipykernel. The path is activated and VS Code has it set in the status bar. I also saw the other paths in the log but couldn't figure out how it is resolving to those. Maybe the Jupyter setting is looking at base path ignoring the virtualenv paths.

Please see below:
(ml_master) [user@host:/Dev/Projects/python/ml_master/bin]$ ls -l python
lrwxr-xr-x 1 user group 39 Aug 20 21:39 python -> /usr/local/opt/python@3.8/bin/python3.8
(ml_master) [user@host:/Dev/Projects/python/ml_master/bin]$ cd /usr/local/opt/python@3.8/bin/
(ml_master) [user@host:/usr/local/opt/python@3.8/bin]$ ls -l python3.8
lrwxr-xr-x 1 user group 57 Jul 20 08:26 python3.8 -> ../Frameworks/Python.framework/Versions/3.8/bin/python3.8
(ml_master) [user@group:/usr/local/opt/python@3.8/bin]$

Jupyter and Ipykernel are installed under ml_master virtualenv:
(ml_master) [user@host:/usr/local/opt/python@3.8/bin]$ pip show ipykernel
Name: ipykernel
Version: 5.3.4
Summary: IPython Kernel for Jupyter
Home-page: https://ipython.org
Author: IPython Development Team
Author-email: ipython-dev@scipy.org
License: BSD
Location: ~/Dev/Projects/python/ml_master/lib/python3.8/site-packages
Requires: traitlets, jupyter-client, ipython, tornado, appnope
Required-by: qtconsole, notebook, jupyter, jupyter-console, ipywidgets

(ml_master) [user@host:/usr/local/opt/python@3.8/bin]$ pip show jupyter
Name: jupyter
Version: 1.0.0
Summary: Jupyter metapackage. Install all the Jupyter components in one go.
Home-page: http://jupyter.org
Author: Jupyter Development Team
Author-email: jupyter@googlegroups.org
License: BSD
Location: ~/Dev/Projects/python/ml_master/lib/python3.8/site-packages
Requires: ipykernel, nbconvert, notebook, ipywidgets, jupyter-console, qtconsole
Required-by:

[mldot]

Some more details: Jupyter and ipykernel seems to work when I installed them under base enviornment. But in the notebook when I switch the interpreter to the ml_master virtualenv it no longer now identifies the libraries installed in virtualenv. Looks like the notebook is ignoring virtualenv settings altogether. This is the reason it could not find the kernel installed in that environment.

import matplotlib.pyplot as plt
import numpy as np

ModuleNotFoundError Traceback (most recent call last)
in
----> 1 import matplotlib.pyplot as plt
2 import numpy as np

ModuleNotFoundError: No module named 'matplotlib

Jupyter_mat.log

[IanMatthewHuff]

@mldot thanks for the info. It does seem that something is up with that python env. Sorry about the questions, but issues like this can always be a bit hard to debug as they are very machine / environment specific and I don't have a repro to debug myself locally. I think that the next step here would be to determine if this is an issue with how VS Code is trying to activate your environment or an issue with just the environment in general.

Is this an environment that you were already using successfully outside of VS Code? If you have a terminal and you activate the ml_master virtualenv can you start up the python prompt and successfully import matplotlib or ipykernel? From your comment above I wasn't quite sure if the ModuleNotFoundErrors were coming from the VS Code scenario, but I assumed that they were.

[mldot]

@IanMatthewHuff Thanks for the follow-up.

Yes this environment works well with a stand alone jupyter notebook install. When I simply have a python .py file running in this environment in. VSCode it can successfully import the libraries. Also the python prompt works. It is only the vscode jupyter notebook itself that is somehow not able to resolve the environment settings.

Just to provide you with an update I was able to get vscode Jupyter notebook working with a new environment I created this morning. The only change was I didnot use the --system-site-packages setting when invoking virtualenv. The ml_master env was created using this parameter which allowed me access to global packages in the parent repository without having to again duplicate installs. But somehow this setting is not working with the vscode python jupyter notebook. If I do need to run Jupyter notebook with vscode I believe for the time being I can use the new environment I created.

[IanMatthewHuff]

@mldot. Glad that you have an environment that you can move forward in for now. The additional info on --system-site-packages is interesting we can try setting up a repro with that parameter to see if we can create this issue ourselves to debug.

[VishalDhayalan]

I'm facing a similar issue except that for me it's "Failed to start a session for the Kernel 'Python 3.7.4 64-bit ('tf-2.3': virtualenv)'.
View Jupyter log for further details." I've also tried all the steps described here: https://stackoverflow.com/questions/58119823/jupyter-notebooks-in-vscode-does-not-use-active-virtual-environment
but to no avail...

the Jupyter Log doesn't have anything useful either:

[rchiodo]

@VishalDhayalan can you create a new issue and fill out the issue template? It might not be related and your output log would be super useful in diagnosing. Thanks.

[VishalDhayalan]

@VishalDhayalan can you create a new issue and fill out the issue template? It might not be related and your output log would be super useful in diagnosing. Thanks.

Done. Created issue #551 Please check. Thank you

[DonJayamanne]

@mldot are you still running into this issue with the latest versions of VS Code and Jupyter extensions?

[mldot]

@DonJayamanne I have not had this issue since I upgraded in July. However, I have not been using this feature much because of scalability as well.

[DonJayamanne]

THanks for your feedback.

ave not been using this feature much because of scalability as well.

Please could you elaborate on that. I'd like to understand what you mean by that. Thanks

[mldot]

THanks for your feedback.

ave not been using this feature much because of scalability as well.

Please could you elaborate on that. I'd like to understand what you mean by that. Thanks

It crashes code if I have a memory intense processing. For example like if I am simply doing a gradient descent with regularization it will crash. Doesn't happen with native jupyter.

[DonJayamanne]

doing a gradient descent with regularization it will crash. Doesn't happen with native jupyter.

Would it be possible to provide some sample code or a notebook. Thanks.

[DonJayamanne]

ot had this issue since I upgraded in July. However, I have not been using this

Closing as this issue as its not longer a problem.
Please feel free to create a new issue or re-open this if it persists