the whole lab gets stuck when open a big notebook that contains lots of output

[lumberbot-app]

Question

the whole lab gets stuck when open a big notebook that contains lots of output.

I know showing lots of data in a notebook is not appropriate, but I think opening notebook should not cause the whole service to get stuck. Is there some optimization measures to avoid it?

---

Originally opened as jupyter/notebook#6077 by @icankeep, migration requested by @kevin-bates

[lumberbot-app]

@kevin-bates commented: Hi @icankeep - this does sound like an issue and I'm wondering if you can try something...

If you're already using Lab 3.0+ the underlying server comes from the jupyter_server repo rather than this one. In jupyter_server 1.1 (current is 1.8) we released the ability to use asynchronous content management - which is not enabled by default. I'm wondering if you could try this using Lab 3.0+ and start the server with the following additional option and see how that behaves:

--ServerApp.contents_manager_class=jupyter_server.services.contents.largefilemanager.AsyncLargeFileManager

If this resolves your issue, I think we might want to consider a switch of the default in jupyter_server (at the appropriate release boundary of course).

cc: @mwakaba2

[lumberbot-app]

@icankeep commented: @kevin-bates
thank you for your reply, I try to set this in my lab, but it doesn't work right.

notebook=6.3.0
jupyterlab=3.0.14
jupyter-server=1.6.4

This issue is more obvious when accessing remote directories that require network interaction, it looks like the main thread is blocking

the code that generates large output can reproduce the issue.

import time
a = ''
for i in range(0, 3000):
    a = str(i) + a
for i in range(0, 10000):
    time.sleep(0.1)
    print(a)

[lumberbot-app]

@kevin-bates commented: Hi @icankeep - I'd like to verify which server is in use here as we should be focusing on using Jupyter Server. Once established and the problem still persists, I will migrate this issue to the jupyter_server repository.

To verify which server is in use, there should be an informational message logged to the console of the form:

[I 10:23:09.581 NotebookApp] Jupyter Notebook 6.4.0 is running at:

or

[I 10:23:09.581 ServerApp] Jupyter Server 1.8.0 is running at:

The version information isn't as important as the application name that proceeds it.

Thanks.

[lumberbot-app]

@icankeep commented: @kevin-bates
we use jupyterhub(version: 1.3.0), spawn a singleuser like this jupyter-labhub

[lumberbot-app]

@icankeep commented: I found a problem that might be related to the issue.

Reproduce step

1. pip install py-spy (dump python thread)
2. just start a simple jupyter program jupyter lab
3. execute py-spy dump -p {jupyter-pid}

the dump result:

Process 98632: /Users/passer/conda/envs/python3.6.7/bin/python3.6 /Users/passer/conda/envs/python3.6.7/bin/jupyter-lab
Python v3.6.7 (/Users/passer/conda/envs/python3.6.7/bin/python3.6)

Thread 0x20524FE00 (idle): "MainThread"
    select (selectors.py:577)
    _run_once (asyncio/base_events.py:1404)
    run_forever (asyncio/base_events.py:427)
    start (tornado/platform/asyncio.py:199)
    start_ioloop (jupyter_server/serverapp.py:2148)
    start (jupyter_server/serverapp.py:2160)
    launch_instance (jupyter_server/extension/application.py:528)
    <module> (jupyter-lab:8)
Thread 0x30B41A000 (idle): "ThreadPoolExecutor-0_0"
    wait (threading.py:295)
    get (queue.py:164)
    _worker (concurrent/futures/thread.py:67)
    run (threading.py:864)
    _bootstrap_inner (threading.py:916)
    _bootstrap (threading.py:884)
Thread 0x30B91D000 (idle): "ThreadPoolExecutor-2_0"
    wait (threading.py:295)
    get (queue.py:164)
    _worker (concurrent/futures/thread.py:67)
    run (threading.py:864)
    _bootstrap_inner (threading.py:916)
    _bootstrap (threading.py:884)
Thread 0x30BE20000 (idle): "ThreadPoolExecutor-2_1"
    wait (threading.py:295)
    get (queue.py:164)
    _worker (concurrent/futures/thread.py:67)
    run (threading.py:864)
    _bootstrap_inner (threading.py:916)
    _bootstrap (threading.py:884)
Thread 0x30C323000 (idle): "ThreadPoolExecutor-2_2"
    wait (threading.py:295)
    get (queue.py:164)
    _worker (concurrent/futures/thread.py:67)
    run (threading.py:864)
    _bootstrap_inner (threading.py:916)
    _bootstrap (threading.py:884)

4. spawning some kernels, and then dump process stack again.

dump result:

Process 98632: /Users/passer/conda/envs/python3.6.7/bin/python3.6 /Users/passer/conda/envs/python3.6.7/bin/jupyter-lab
Python v3.6.7 (/Users/passer/conda/envs/python3.6.7/bin/python3.6)

Thread 0x20524FE00 (idle): "MainThread"
    select (selectors.py:577)
    _run_once (asyncio/base_events.py:1404)
    run_forever (asyncio/base_events.py:427)
    start (tornado/platform/asyncio.py:199)
    start_ioloop (jupyter_server/serverapp.py:2148)
    start (jupyter_server/serverapp.py:2160)
    launch_instance (jupyter_server/extension/application.py:528)
    <module> (jupyter-lab:8)
Thread 0x30B41A000 (idle): "ThreadPoolExecutor-0_0"
    wait (threading.py:295)
    get (queue.py:164)
    _worker (concurrent/futures/thread.py:67)
    run (threading.py:864)
    _bootstrap_inner (threading.py:916)
    _bootstrap (threading.py:884)
Thread 0x30B91D000 (idle): "ThreadPoolExecutor-2_0"
    wait (threading.py:295)
    get (queue.py:164)
    _worker (concurrent/futures/thread.py:67)
    run (threading.py:864)
    _bootstrap_inner (threading.py:916)
    _bootstrap (threading.py:884)
Thread 0x30BE20000 (idle): "ThreadPoolExecutor-2_1"
    wait (threading.py:295)
    get (queue.py:164)
    _worker (concurrent/futures/thread.py:67)
    run (threading.py:864)
    _bootstrap_inner (threading.py:916)
    _bootstrap (threading.py:884)
Thread 0x30C323000 (idle): "ThreadPoolExecutor-2_2"
    wait (threading.py:295)
    get (queue.py:164)
    _worker (concurrent/futures/thread.py:67)
    run (threading.py:864)
    _bootstrap_inner (threading.py:916)
    _bootstrap (threading.py:884)
Thread 0x30C92C000 (idle): "ThreadPoolExecutor-2_3"
    wait (threading.py:295)
    get (queue.py:164)
    _worker (concurrent/futures/thread.py:67)
    run (threading.py:864)
    _bootstrap_inner (threading.py:916)
    _bootstrap (threading.py:884)
Thread 0x30CE2F000 (idle): "ThreadPoolExecutor-2_4"
    wait (threading.py:295)
    get (queue.py:164)
    _worker (concurrent/futures/thread.py:67)
    run (threading.py:864)
    _bootstrap_inner (threading.py:916)
    _bootstrap (threading.py:884)
Thread 0x30D332000 (idle): "ThreadPoolExecutor-2_5"
    wait (threading.py:295)
    get (queue.py:164)
    _worker (concurrent/futures/thread.py:67)
    run (threading.py:864)
    _bootstrap_inner (threading.py:916)
    _bootstrap (threading.py:884)

... omitting a lot

a lot of ThreadPoolExecutors were created, and then I shutdown all of the kernels, these ThreadPoolExecutors are not destroyed.

I don't know where these thread pools were created.

[lumberbot-app]

@kevin-bates commented: This issue belongs in jupyter_server (I will migrate it to jupyter_server after posting this comment).

If you did not "warm-up" the server by starting 1 kernel, I would suggest doing that. I suspect at least one of these is to monitor for culling - which won't start until the first kernel starts. So perhaps you could dump the threads following the start of the first kernel as well.

Also, is this last comment conflating the other issue (in jupyter_server) you opened (#530)? This issue only talks about opening a notebook that contains a large output area yet you talk about threads related to "all of the kernels" - so I'm a little confused.