Make debugging of Jupyter Panel Preview easier

[MarcSkovMadsen]

I've experienced it many times and also seen others struggle with the Jupyter Panel Preview.

If the jupyter panel preview just displays a blank screen you have no way of finding the cause as you cannot find the server logs any where.

Here is an example of a user struggling https://discourse.holoviz.org/t/how-to-customize-the-display-url-from-panel-serve-for-use-behind-jupyterhub-with-jupyter-server-proxy/3571/5?u=marc. I'm trying to help but have no chance to find the root cause.

I don't know what the solution is but

• If the log exists somewhere then document how to find it.
• If no log exist then create one by writing to a file. Document where to find that file.
• If no log exist then make a general feature request to jupyter and maybe help them implement

[tlvu]

I launched the docker image in https://discourse.holoviz.org/t/how-to-customize-the-display-url-from-panel-serve-for-use-behind-jupyterhub-with-jupyter-server-proxy/3571/ directly on my machine and I got this error when trying to launch the Panel preview:

ServerApp - ERROR - Uncaught exception GET /panel-preview/render/workdir/PAVICS-e2e-workflow-tests/binder/test-notebooks/TEST-panel-servable.ipynb (10.10.10.7)
HTTPServerRequest(protocol='http', host='lvu.ouranos.ca:8890', method='GET', uri='/panel-preview/render/workdir/PAVICS-e2e-workflow-tests/binder/test-notebooks/TEST-panel-servable.ipynb', version='HTTP/1.1', remote_ip='10.10.10.7')
Traceback (most recent call last):
  File "/opt/conda/envs/birdy/lib/python3.8/site-packages/tornado/web.py", line 1704, in _execute
    result = await result
  File "/opt/conda/envs/birdy/lib/python3.8/site-packages/panel/io/jupyter_server_extension.py", line 116, in get
    page = server_html_page_for_session(
  File "/opt/conda/envs/birdy/lib/python3.8/site-packages/bokeh/embed/server.py", line 255, in server_html_page_for_session
    bundle = bundle_for_objs_and_resources(None, resources)
  File "/opt/conda/envs/birdy/lib/python3.8/site-packages/bokeh/embed/bundle.py", line 211, in bundle_for_objs_and_resources
    js_files.extend(js_resources.js_files)
  File "/opt/conda/envs/birdy/lib/python3.8/site-packages/panel/io/resources.py", line 299, in js_files
    dist_dir = self.dist_dir
  File "/opt/conda/envs/birdy/lib/python3.8/site-packages/panel/io/resources.py", line 259, in dist_dir
    if self.absolute:
AttributeError: 'Resources' object has no attribute 'absolute'

[hoxbro]

@tlvu This should be fixed in the next patch release. If you can't wait, you can install it with pip install panel --pre or conda install -c pyviz/label/dev -c conda-forge "panel=0.13.1a2" .

[tlvu]

@hoxbro Tried panel=0.13.1a2, Panel preview green icon gives blank page.

  ! conda env export -n birdy | grep -i -e panel -e proxy -e bokeh -e cpython

  - bokeh=2.4.2=py38h578d9bd_1
  - configurable-http-proxy=4.5.0=node16_h3b247e2_2
  - jupyter-panel-proxy=0.2.0a2=py_0
  - jupyter-server-proxy=3.2.1=pyhd8ed1ab_0
  - jupyter_bokeh=3.0.4=pyhd8ed1ab_0
  - lazy-object-proxy=1.7.1=py38h0a891b7_1
  - panel=0.13.1a2=py_0
  - python=3.8.13=h582c2e5_0_cpython

Clicking on the Panel preview green icon gives a blank page with this error on STDOUT:

[W 2022-04-27 14:50:55.477 ServerApp] Notebook home/lvu/repos/PAVICS-e2e-workflow-tests/binder/test-notebooks/TEST-panel-servable.ipynb is not trusted
[I 2022-04-27 14:50:55.968 ServerApp] Kernel started: c119bf68-0b9b-4ef7-9bf9-31a5f53ad8b3                                                            
[IPKernelApp] ERROR | No such comm target registered: jupyter.widget.control                                                                          
[IPKernelApp] WARNING | No such comm: 536c392f-ad19-4e6b-aa0a-2c68853ecd2e
[I 2022-04-27 14:50:59.131 ServerApp] Saving TEST-panel-servable.ipynb 

To reproduce:

• Check out repo at this specific commit https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/tree/52d82220024659218f70b61dd43a7d0a4d055808
• Execute script launchnotebook from checkout to run pre-built docker image of JupyterLab with pre-installed plugins
• In JupyterLab, navigate to the file binder/test-notebooks/TEST-panel-servable.ipynb open it
• Click on "Panel preview" green icon on the file

[hoxbro]

I can recreate your problem - but I have no idea why it doesn't work. I would suggest you try to remove some of the complexity and see if that helps.

I tried to set up a simple docker container which did work.

FROM mambaorg/micromamba

RUN micromamba install -y -n base -c pyviz/label/dev -c conda-forge python=3.9 jupyterlab=3 "panel=0.13.1a2"
RUN micromamba clean --all --yes

EXPOSE 8888
CMD ["/bin/bash", "-c", "jupyter lab --ip '*'"]

1. docker build . -t "test"
2. docker run -p 8888:8888 test

[tlvu]

I can recreate your problem - but I have no idea why it doesn't work. I would suggest you try to remove some of the complexity and see if that helps.

@hoxbro Yes that JupyterLab docker image is huge since it is the image that powers this site https://pavics.ouranos.ca/.

Is there any logging/tracing we can enable to debug this problem? Sorry if it's a beginner question, I am new at debugging JupyterLab plugin issues. Any generic JupyterLab plugins debugging technique and any specific to Panel is appreciated.

[MarcSkovMadsen]

I can confirm that upgrading to panel=0.13.1a2 makes the Jupyter Panel Preview work for me.

[hoxbro]

I would simply run a Dockerfile like the following and remove dependencies from environment.yml to see what causes the problem.

FROM mambaorg/micromamba

COPY docker/environment.yml ./
RUN micromamba install -y -n base -f environment.yml
RUN micromamba clean --all --yes

EXPOSE 8888
CMD ["/bin/bash", "-c", "jupyter lab --ip '*'"]

[hoxbro]

The problem seems to be related to handcalcs.

This works:

FROM mambaorg/micromamba

RUN micromamba install -y -n base -c pyviz/label/dev -c conda-forge jupyterlab=3 "panel=0.13.1a2"
RUN micromamba clean --all --yes

EXPOSE 8888
CMD ["/bin/bash", "-c", "jupyter lab --ip '*'"]

This does not:

FROM mambaorg/micromamba

RUN micromamba install -y -n base -c pyviz/label/dev -c conda-forge jupyterlab=3 "panel=0.13.1a2" handcalcs
RUN micromamba clean --all --yes

EXPOSE 8888
CMD ["/bin/bash", "-c", "jupyter lab --ip '*'"]

[tlvu]

The problem seems to be related to handcalcs.

@hoxbro Wow spot on. Tested with a new build on my side with handcalcs removed and the Preview panel green button worked. How did you suspect handcalcs in the numerous jupyter plugins we installed in our JupyterLab image?!

By the way the same error [IPKernelApp] ERROR | No such comm target registered: jupyter.widget.control appeared again on STDOUT even when it works so basically there are nothing useful on STDOUT.

So it works when I launch the docker image standalone. It gives 500: Internal Server Error when launched by JupyterHub using DockerSpawner. Here is my jupyterhub config https://github.com/bird-house/birdhouse-deploy/blob/d5d2332580a1682b43c697fcc232aacf347a2a92/birdhouse/config/jupyterhub/jupyterhub_config.py.template. Do you happen to see anything fishy in that config?

When launched by DockerSpawner I also do not have the STDOUT, very annoying since if there are traceback I can not see it ! Will try to get STDOUT from the container launched by DockerSpawner.

[tlvu]

So it works when I launch the docker image standalone. It gives 500: Internal Server Error when launched by JupyterHub using DockerSpawner.

@hoxbro

I think I found the cause of my problem and a new bug with the Panel preview green button: when looking up for the file on disk, it forgot to prepend the value of --notebook-dir:

[D 2022-04-28 19:40:03.697 ServerApp] Saving /home/lvu/repos/PAVICS-e2e-workflow-tests/binder/test-notebooks/TEST-panel-servable.ipynb
ServerApp - ERROR - Uncaught exception GET /panel-preview/render/test-notebooks/TEST-panel-servable.ipynb (10.10.10.12)
HTTPServerRequest(protocol='http', host='[lvu.ouranos.ca:8890](http://lvu.ouranos.ca:8890/)', method='GET', uri='/panel-preview/render/test-notebooks/TEST-panel-servable.ipynb', ver
sion='HTTP/1.1', remote_ip='10.10.10.12')
Traceback (most recent call last):
File "/opt/conda/envs/birdy/lib/python3.8/site-packages/tornado/[web.py](http://web.py/)", line 1704, in _execute
result = await result
File "/opt/conda/envs/birdy/lib/python3.8/site-packages/panel/io/jupyter_server_extension.py", line 107, in get
app = build_single_handler_application(path)
File "/opt/conda/envs/birdy/lib/python3.8/site-packages/bokeh/command/[util.py](http://util.py/)", line 135, in build_single_handler_application
raise ValueError("Path for Bokeh server application does not exist: %s" % path)
ValueError: Path for Bokeh server application does not exist: /test-notebooks/TEST-panel-servable.ipynb

It complains path /test-notebooks/TEST-panel-servable.ipynb not found and that's because the real path on disk is /home/lvu/repos/PAVICS-e2e-workflow-tests/binder/test-notebooks/TEST-panel-servable.ipynb (arg --notebook-dir=/home/lvu/repos/PAVICS-e2e-workflow-tests/binder given to jupyter lab) !

That error is reproducible when I make this change to my launchnotebook script

$ git diff
diff --git a/launchnotebook b/launchnotebook
index a3dcc0f..828afe2 100755
--- a/launchnotebook
+++ b/launchnotebook
@@ -26,4 +26,4 @@ docker run --rm --name $CONTAINER_NAME \
              usermod -u $UID jenkins; \
              echo 'ENV_PATH PATH=/usr/local/envs/birdy/bin:/opt/conda/bin:/usr/local/bin:/usr/bin:/bin' >> /etc/login.defs; \
              su jenkins -s /bin/bash -c \
-                 'jupyter lab --ip=0.0.0.0 --port=$PORT --no-browser'"
+                 'jupyter lab --ip=0.0.0.0 --port=$PORT --no-browser --notebook-dir=`pwd`/binder'"

Should I open a new bug?

[tlvu]

I think I found the cause of my problem and a new bug with the Panel preview green button: when looking up for the file on disk, it forgot to prepend the value of --notebook-dir:

Looks like similar to #3170

[tlvu]

I attempted a fix in PR #3469 but something weird happens when I test on the same test env I've been using to debug this issue.

Maybe it's late and I am tired but it seems the JupyterLab server restart each time I save the notebook to test the "Render on Save" checkbox. Is this a known issue?

[hoxbro]

@hoxbro Wow spot on. Tested with a new build on my side with handcalcs removed and the Preview panel green button worked. How did you suspect handcalcs in the numerous jupyter plugins we installed in our JupyterLab image?!

Just like I said, by removing dependencies from the environment file. There was no magic fix just manual work.

[tlvu]

Just a recap that a way to debug the Jupyter Panel Preview would be immensely useful. Right now there is absolutely no logs anywhere.

I had 2 problems breaking Jupyter Panel Preview on my JupyterLab environment. I had package handcalcs that somehow interfere with it. Thanks @hoxbro for finding out by removing dependencies from the environment.yml file. I found out --notebook-dir had a regression and fixed it with my PR #3469. This was pure luck because I wanted to set --notebook-dir to save me time to navigate to the test notebook, it did not expect it to expose the regression.

Basically all the debugging technique currently are either guess work (handcalcs) or pure luck (--notebook-dir regression). A more "formal" approach would be much more helpful.

I also currently have trouble capturing the STDOUT from JupyterLab image (when launched by JupyterHub via DockerSpawner). If launch by Binder, same issue, no access to STDOUT.

I would suggest, if possible, the logging goes to the browser console would be best for accessibility.

[kuatroka]

Just a recap that a way to debug the Jupyter Panel Preview would be immensely useful. Right now there is absolutely no logs anywhere.

+1. I'm on version '0.14.1', no mentioned extensions and still no panel preview