Jupyter Kernel Spec + N times refreshing for N Kernels + Slow start up

[pkasinathan]

Hi Dev,

I found some strange scenario on Notebook. I'm running jupyter notebook 4.1.0 version on Python 3.5.2 :: Anaconda 4.1.1 (64-bit) platform.

Issue: If I enable N kernels, when I refresh Jupyter home page, then it's getting kernel spec N times and hence page refresh is too slow. Say for example: If I enable 10 custom kernels, when I refresh jupyter home page, it's taking 10+ seconds. When I see log, most of the time is taking in refreshing kernels again and again for 10 times as "Found kernel <> in /usr/local/share/jupyter/kernels". In logs, it show 100 times that kernel as found.

Version:

[22:48]:[root@myhost:~]# python -V
Python 3.5.2 :: Anaconda 4.1.1 (64-bit)
[22:48]:[root@myhost:~]# jupyter --version
4.1.0

Starting Jupyter with default Python Kernel:

[22:48]:[root@myhost:~]# jupyter notebook --no-browser --ip 0.0.0.0 --port 9000 --debug

Log: With only one Kernel by default:

[D 22:40:45.377 NotebookApp] 200 GET /api/sessions?_=1513320006484 (10.196.177.146) 0.85ms
[D 22:40:45.378 NotebookApp] 200 GET /api/terminals?_=1513320006485 (10.196.177.146) 0.42ms
[D 22:40:45.459 NotebookApp] 200 GET /api/contents?type=directory&_=1513320006486 (10.196.177.146) 1.18ms
[D 22:40:49.684 NotebookApp] Using contents: services/contents
[D 22:40:49.685 NotebookApp] 304 GET /tree (10.196.177.146) 1.46ms
[D 22:40:49.789 NotebookApp] 304 GET /custom/custom.css (10.196.177.146) 0.52ms
[D 22:40:50.111 NotebookApp] 304 GET /custom/custom.js?v=20171214222020 (10.196.177.146) 0.84ms
[D 22:40:50.112 NotebookApp] 200 GET /static/base/images/favicon.ico?v=30780f272ab4aac64aa073a841546240 (10.196.177.146) 0.64ms
[D 22:40:50.221 NotebookApp] 200 GET /api/config/common?_=1513320049961 (10.196.177.146) 2.28ms
[D 22:40:50.223 NotebookApp] 200 GET /api/config/tree?_=1513320049960 (10.196.177.146) 1.93ms
[D 22:40:50.225 NotebookApp] Native kernel (python3) available from /home/anaconda3/lib/python3.5/site-packages/ipykernel/resources
[D 22:40:50.745 NotebookApp] 304 GET /api/kernelspecs (10.196.177.146) 520.39ms
[D 22:40:50.746 NotebookApp] 200 GET /api/terminals?_=1513320049962 (10.196.177.146) 0.71ms
[D 22:40:50.747 NotebookApp] 200 GET /api/sessions?_=1513320049963 (10.196.177.146) 0.62ms
[D 22:40:50.748 NotebookApp] 200 GET /api/terminals?_=1513320049964 (10.196.177.146) 0.42ms
[D 22:40:50.993 NotebookApp] 200 GET /api/contents?type=directory&_=1513320049965 (10.196.177.146) 1.23ms

Enabling 1st custom kernel

[22:41]:[root@myhost:~]# jupyter-kernelspec install $SPARKMAGIC_PATH/sparkmagic/sparkmagic/kernels/sparkkernel
[InstallKernelSpec] Installed kernelspec sparkkernel in /usr/local/share/jupyter/kernels/sparkkernel
[22:41]:[root@myhost:~]# 

Log: After enabling 1st custom kernel, when I refresh home page

[D 22:43:48.936 NotebookApp] 200 GET /api/sessions?_=1513320206761 (10.196.177.146) 0.88ms
[D 22:43:48.938 NotebookApp] 200 GET /api/terminals?_=1513320206762 (10.196.177.146) 0.43ms
[D 22:43:49.295 NotebookApp] 200 GET /api/contents?type=directory&_=1513320206763 (10.196.177.146) 1.17ms
[D 22:44:05.654 NotebookApp] Using contents: services/contents
[D 22:44:05.655 NotebookApp] 304 GET /tree (10.196.177.146) 1.56ms
[D 22:44:05.769 NotebookApp] 304 GET /custom/custom.css (10.196.177.146) 0.55ms
[D 22:44:06.154 NotebookApp] 304 GET /custom/custom.js?v=20171214222020 (10.196.177.146) 0.50ms
[D 22:44:06.160 NotebookApp] 200 GET /static/base/images/favicon.ico?v=30780f272ab4aac64aa073a841546240 (10.196.177.146) 0.61ms
[D 22:44:06.252 NotebookApp] 200 GET /api/config/tree?_=1513320245977 (10.196.177.146) 2.08ms
[D 22:44:06.255 NotebookApp] 200 GET /api/config/common?_=1513320245978 (10.196.177.146) 2.57ms
[D 22:44:06.256 NotebookApp] Found kernel sparkkernel in /usr/local/share/jupyter/kernels
[D 22:44:06.256 NotebookApp] Native kernel (python3) available from /home/anaconda3/lib/python3.5/site-packages/ipykernel/resources
[D 22:44:06.768 NotebookApp] Found kernel sparkkernel in /usr/local/share/jupyter/kernels
[D 22:44:06.768 NotebookApp] Native kernel (python3) available from /home/anaconda3/lib/python3.5/site-packages/ipykernel/resources
[D 22:44:07.277 NotebookApp] 304 GET /api/kernelspecs (10.196.177.146) 1021.39ms
[D 22:44:07.279 NotebookApp] 200 GET /api/terminals?_=1513320245979 (10.196.177.146) 0.71ms
[D 22:44:07.280 NotebookApp] 200 GET /api/terminals?_=1513320245981 (10.196.177.146) 0.42ms
[D 22:44:07.281 NotebookApp] 200 GET /api/sessions?_=1513320245980 (10.196.177.146) 0.60ms
[D 22:44:07.602 NotebookApp] 200 GET /api/contents?type=directory&_=1513320245982 (10.196.177.146) 1.32ms

Enabling 2nd custom kernel

[22:41]:[root@myhost:~]# jupyter-kernelspec install $SPARKMAGIC_PATH/sparkmagic/sparkmagic/kernels/pysparkkernel
[InstallKernelSpec] Installed kernelspec pysparkkernel in /usr/local/share/jupyter/kernels/pysparkkernel
[22:45]:[root@myhost:~]# 

Log: After enabling 2nd custom kernel, when I refresh home page

[D 22:45:37.584 NotebookApp] 200 GET /api/sessions?_=1513320245983 (10.196.177.146) 0.90ms
[D 22:45:37.585 NotebookApp] 200 GET /api/terminals?_=1513320245984 (10.196.177.146) 0.42ms
[D 22:45:37.668 NotebookApp] 200 GET /api/contents?type=directory&_=1513320245985 (10.196.177.146) 1.18ms
[D 22:45:38.691 NotebookApp] Using contents: services/contents
[D 22:45:38.692 NotebookApp] 304 GET /tree (10.196.177.146) 1.51ms
[D 22:45:38.812 NotebookApp] 304 GET /custom/custom.css (10.196.177.146) 0.55ms
[D 22:45:39.134 NotebookApp] 304 GET /custom/custom.js?v=20171214222020 (10.196.177.146) 0.50ms
[D 22:45:39.150 NotebookApp] 200 GET /static/base/images/favicon.ico?v=30780f272ab4aac64aa073a841546240 (10.196.177.146) 0.61ms
[D 22:45:39.231 NotebookApp] 200 GET /api/config/tree?_=1513320338963 (10.196.177.146) 2.07ms
[D 22:45:39.233 NotebookApp] 200 GET /api/config/common?_=1513320338964 (10.196.177.146) 1.73ms
[D 22:45:39.235 NotebookApp] Found kernel pysparkkernel in /usr/local/share/jupyter/kernels
[D 22:45:39.235 NotebookApp] Found kernel sparkkernel in /usr/local/share/jupyter/kernels
[D 22:45:39.235 NotebookApp] Native kernel (python3) available from /home/anaconda3/lib/python3.5/site-packages/ipykernel/resources
[D 22:45:39.752 NotebookApp] Found kernel pysparkkernel in /usr/local/share/jupyter/kernels
[D 22:45:39.752 NotebookApp] Found kernel sparkkernel in /usr/local/share/jupyter/kernels
[D 22:45:39.752 NotebookApp] Native kernel (python3) available from /home/anaconda3/lib/python3.5/site-packages/ipykernel/resources
[D 22:45:40.271 NotebookApp] Found kernel pysparkkernel in /usr/local/share/jupyter/kernels
[D 22:45:40.271 NotebookApp] Found kernel sparkkernel in /usr/local/share/jupyter/kernels
[D 22:45:40.271 NotebookApp] Native kernel (python3) available from /home/anaconda3/lib/python3.5/site-packages/ipykernel/resources
[D 22:45:40.786 NotebookApp] 200 GET /api/kernelspecs (10.196.177.146) 1551.01ms
[D 22:45:40.787 NotebookApp] 200 GET /api/terminals?_=1513320338965 (10.196.177.146) 0.73ms
[D 22:45:40.788 NotebookApp] 200 GET /api/terminals?_=1513320338967 (10.196.177.146) 0.53ms
[D 22:45:40.789 NotebookApp] 200 GET /api/sessions?_=1513320338966 (10.196.177.146) 0.65ms
[D 22:45:41.125 NotebookApp] 200 GET /api/contents?type=directory&_=1513320338968 (10.196.177.146) 1.21ms

In above logs, you can see I'm seeing Found Kernel 9 times, if I have 3 kernels enabled.

[D 22:45:39.235 NotebookApp] Found kernel pysparkkernel in /usr/local/share/jupyter/kernels
[D 22:45:39.235 NotebookApp] Found kernel sparkkernel in /usr/local/share/jupyter/kernels
[D 22:45:39.235 NotebookApp] Native kernel (python3) available from /home/anaconda3/lib/python3.5/site-packages/ipykernel/resources
[D 22:45:39.752 NotebookApp] Found kernel pysparkkernel in /usr/local/share/jupyter/kernels
[D 22:45:39.752 NotebookApp] Found kernel sparkkernel in /usr/local/share/jupyter/kernels
[D 22:45:39.752 NotebookApp] Native kernel (python3) available from /home/anaconda3/lib/python3.5/site-packages/ipykernel/resources
[D 22:45:40.271 NotebookApp] Found kernel pysparkkernel in /usr/local/share/jupyter/kernels
[D 22:45:40.271 NotebookApp] Found kernel sparkkernel in /usr/local/share/jupyter/kernels
[D 22:45:40.271 NotebookApp] Native kernel (python3) available from /home/anaconda3/lib/python3.5/site-packages/ipykernel/resources

Please let me know your comments to fix this issue. It's really slowing down our Production instances performance when we enable 10+ kernels.

Thanks!

[takluyver]

I'm investigating - I can reproduce this.

[takluyver]

I think I've tracked this down, and #3136 should fix it.

[pkasinathan]

Hi @takluyver, Thanks for your quick response!
Here, in PayPal, around 200+ users are using Jupyter with 15+ kernels (Python, Hive, Spark, Presto, Teradata with multiple systems) enabled. And, every time when user clicks home page or logo, it’s taking minimum 10 to 15 seconds and affecting user experience a lot.
Please let us know, if you have any updates on patch. Thanks in advance!
We will start tracking #3136.

[takluyver]

Thanks @prabhu1984 . It turned out to need some changes in jupyter_client, one of the packages the notebook uses. I've made a couple of fixes there, and I'll try to make a release in the next couple of days. Once jupyter_client 5.2 is out, upgrading should make it a bit quicker, and then #3136 should be a further improvement.