v2 breaks jupyterlab with jupytext extension

[st-bender]

Description

Hi,

My search did not turn up anything useful, nor could I find something in the changelog.

After updating jupyter-server to v2.0.1, I observed several issues with jupyterlab.
One issue stops the notebooks from being saved with an error message that a "coroutine is not subscriptable".
The second issue breaks starting new notebooks with whatever kernel with a very uninformative message "unhandled error" (nothing else).

Reproduce

1. Start jupyter lab
2. Open any notebook
3. Click on 'save' -> Error 1

2b. Click on 'New' -> 'Notebook'
3b. Second error occurs.

Downgrading jupyter-server to 1.23.3 fixes both issues.

Expected behavior

1. Notebooks are saved successfully.
2. A new notoebook appears in a tab in jupyterlab.

Context

• Operating System and version: Ubunutu 18.04
• Browser and version: Firefox
• Jupyter Server version: v2.0.1

Troubleshoot Output

(Grepped "jupyter" to shorten the list a little, let me know if you need more package details.)

        jupyter                           1.0.0                                                                 
        jupyter_client                    7.4.8                                                                 
        jupyter-console                   6.4.4                                                                 
        jupyter-contrib-core              0.4.0
        jupyter-contrib-nbextensions      0.7.0    
        jupyter_core                      5.1.0
        jupyter-events                    0.5.0
        jupyter-highlight-selected-word   0.2.0    
        jupyter-latex-envs                1.4.6                                                                 
        jupyter-nbextensions-configurator 0.4.1
        jupyter_server                    2.0.1
        jupyter-server-proxy              3.2.2                                                                 
        jupyter_server_terminals          0.4.2     
        jupyterlab                        3.5.1                                                                 
        jupyterlab-code-formatter         1.5.3
        jupyterlab-launcher               0.13.1       
        jupyterlab-pygments               0.2.2
        jupyterlab_server                 2.16.5                                                                
        jupyterlab-slurm                  3.0.1
        jupyterlab-vim                    0.15.1  
        jupyterlab-widgets                3.0.4  
        jupytext                          1.14.2
        jupyter                   1.0.0            py39hf3d152e_8    conda-forge
        jupyter-server-proxy      3.2.2              pyhd8ed1ab_0    conda-forge
        jupyter_client            7.4.8              pyhd8ed1ab_0    conda-forge
        jupyter_console           6.4.4              pyhd8ed1ab_0    conda-forge
        jupyter_contrib_core      0.4.0              pyhd8ed1ab_0    conda-forge
        jupyter_contrib_nbextensions 0.7.0              pyhd8ed1ab_0    conda-forge                            
        jupyter_core              5.1.0            py39hf3d152e_0    conda-forge                               
        jupyter_events            0.5.0              pyhd8ed1ab_0    conda-forge
        jupyter_highlight_selected_word 0.2.0           py39hf3d152e_1005    conda-forge
        jupyter_latex_envs        1.4.6           pyhd8ed1ab_1002    conda-forge
        jupyter_nbextensions_configurator 0.4.1              pyhd8ed1ab_2    conda-forge
        jupyter_server            2.0.1              pyhd8ed1ab_0    conda-forge
        jupyter_server_terminals  0.4.2              pyhd8ed1ab_0    conda-forge
        jupyterlab                3.5.1              pyhd8ed1ab_0    conda-forge
        jupyterlab-slurm          3.0.1                    pypi_0    pypi
        jupyterlab_code_formatter 1.5.3              pyhd8ed1ab_0    conda-forge
        jupyterlab_launcher       0.13.1                     py_2    conda-forge
        jupyterlab_pygments       0.2.2              pyhd8ed1ab_0    conda-forge
        jupyterlab_server         2.16.5             pyhd8ed1ab_0    conda-forge
        jupyterlab_vim            0.15.1             pyhd8ed1ab_0    conda-forge
        jupyterlab_widgets        3.0.4              pyhd8ed1ab_0    conda-forge
        jupytext                  1.14.2             pyh5da7574_0    conda-forge
          - jupyter=1.0.0=py39hf3d152e_8
          - jupyter-server-proxy=3.2.2=pyhd8ed1ab_0
          - jupyter_client=7.4.8=pyhd8ed1ab_0
          - jupyter_console=6.4.4=pyhd8ed1ab_0
          - jupyter_contrib_core=0.4.0=pyhd8ed1ab_0
          - jupyter_contrib_nbextensions=0.7.0=pyhd8ed1ab_0
          - jupyter_core=5.1.0=py39hf3d152e_0
          - jupyter_events=0.5.0=pyhd8ed1ab_0
          - jupyter_highlight_selected_word=0.2.0=py39hf3d152e_1005
          - jupyter_latex_envs=1.4.6=pyhd8ed1ab_1002
          - jupyter_nbextensions_configurator=0.4.1=pyhd8ed1ab_2
          - jupyter_server=2.0.1=pyhd8ed1ab_0
          - jupyter_server_terminals=0.4.2=pyhd8ed1ab_0
          - jupyterlab=3.5.1=pyhd8ed1ab_0
          - jupyterlab_code_formatter=1.5.3=pyhd8ed1ab_0 
          - jupyterlab_launcher=0.13.1=py_2
          - jupyterlab_pygments=0.2.2=pyhd8ed1ab_0
          - jupyterlab_server=2.16.5=pyhd8ed1ab_0
          - jupyterlab_vim=0.15.1=pyhd8ed1ab_0
          - jupyterlab_widgets=3.0.4=pyhd8ed1ab_0
          - jupytext=1.14.2=pyh5da7574_0
              - jupyterlab-slurm==3.0.1

[welcome]

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! 🤗

If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively.

You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! 👋

Welcome to the Jupyter community! 🎉

[kevin-bates]

Hi @st-bender - thank you for bringing this to our attention. Could you please include any error stacks produced on your server's console window (the window in which you started jupyter lab)? (Preferably relative to each issue you see.)

I'm unable to reproduce the first issue (open/save any existing notebook). However, if I create a new notebook and am prompted to rename the untitled notebook on first save, and change the name, I see the following in my console window:

[E 2022-12-09 10:34:41.938 ServerApp] Uncaught exception POST /api/contents/alice/Untitled6.ipynb/checkpoints?1670610881929 (::1)
    HTTPServerRequest(protocol='http', host='localhost:8888', method='POST', uri='/api/contents/alice/Untitled6.ipynb/checkpoints?1670610881929', version='HTTP/1.1', remote_ip='::1')
    Traceback (most recent call last):
      File "/opt/miniconda3/envs/server-dev/lib/python3.9/site-packages/tornado/web.py", line 1713, in _execute
        result = await result
      File "/opt/miniconda3/envs/server-dev/lib/python3.9/site-packages/jupyter_server/services/contents/handlers.py", line 297, in post
        checkpoint = await ensure_async(cm.create_checkpoint(path))
      File "/opt/miniconda3/envs/server-dev/lib/python3.9/site-packages/jupyter_core/utils/__init__.py", line 172, in ensure_async
        result = await obj
      File "/opt/miniconda3/envs/server-dev/lib/python3.9/site-packages/jupyter_server/services/contents/manager.py", line 1022, in create_checkpoint
        return await self.checkpoints.create_checkpoint(self, path)
      File "/opt/miniconda3/envs/server-dev/lib/python3.9/site-packages/jupyter_server/services/contents/filecheckpoints.py", line 144, in create_checkpoint
        await self._copy(src_path, dest_path)
      File "/opt/miniconda3/envs/server-dev/lib/python3.9/site-packages/jupyter_server/services/contents/fileio.py", line 360, in _copy
        await async_copy2_safe(src, dest, log=self.log)  # type:ignore
      File "/opt/miniconda3/envs/server-dev/lib/python3.9/site-packages/jupyter_server/services/contents/fileio.py", line 50, in async_copy2_safe
        await run_sync(shutil.copyfile, src, dst)
      File "/opt/miniconda3/envs/server-dev/lib/python3.9/site-packages/anyio/to_thread.py", line 31, in run_sync
        return await get_asynclib().run_sync_in_worker_thread(
      File "/opt/miniconda3/envs/server-dev/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
        return await future
      File "/opt/miniconda3/envs/server-dev/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 867, in run
        result = context.run(func, *args)
      File "/opt/miniconda3/envs/server-dev/lib/python3.9/shutil.py", line 264, in copyfile
        with open(src, 'rb') as fsrc:
    FileNotFoundError: [Errno 2] No such file or directory: '/Users/kbates/notebooks/alice/Untitled6.ipynb'
[W 2022-12-09 10:34:41.939 ServerApp] wrote error: 'Unhandled error'
[E 2022-12-09 10:34:41.939 ServerApp] {
      "Host": "localhost:8888",
      "Accept": "*/*",
      "Referer": "http://localhost:8888/lab/tree/alice/Untitled6.ipynb",
      "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36"
    }
[E 2022-12-09 10:34:41.939 ServerApp] 500 POST /api/contents/alice/Untitled6.ipynb/checkpoints?1670610881929 (4edd29f587224235b14958e401a04cd7@::1) 5.03ms referer=http://localhost:8888/lab/tree/alice/Untitled6.ipynb
[D 2022-12-09 10:34:41.940 ServerApp] Renaming checkpoint /Users/kbates/notebooks/alice/.ipynb_checkpoints/Untitled6-checkpoint.ipynb -> /Users/kbates/notebooks/alice/.ipynb_checkpoints/yyyy-checkpoint.ipynb
[D 2022-12-09 10:34:41.941 ServerApp] 200 PATCH /api/contents/alice/Untitled6.ipynb?1670610881930 (4edd29f587224235b14958e401a04cd7@::1) 7.15ms

Note that no error is displayed on the UI and all appears to work (from an end-user perspective). Also note that if I do not change the name and just keep the "untitledN" entry, then select rename to dismiss the dialog (which I find very confusing since it appears to be the only way to actually save the file at this point), no errors are logged and "untitledN" is saved. If I update my settings to NOT prompt for "rename on first save", all is well. So, only if I change the name in that "first save" dialog, does the stack trace occur, but, in my case, nothing is visible in the browser.

[st-bender]

Hi @kevin-bates
Checking the error messages and some more googling turned up the following: switching to async breaks jupytext, discussed here: mwouts/jupytext#1020.

I observe the same error output when saving a notebook as in: mwouts/jupytext#1020 (comment)

And a very similar error output when creating a new notoebook:

[I 2022-12-09 21:28:42.211 ServerApp] Creating new notebook in /notebooks                             
/home/user/miniconda3/envs/pangeo/lib/python3.8/site-packages/jupytext/contentsmanager.py:400: RuntimeWarning: coroutine AsyncFileContentsManager.dir_exists' was never awaited
  if not self.dir_exists(path):                                                           
/home/user/miniconda3/envs/pangeo/lib/python3.8/site-packages/jupytext/contentsmanager.py:504: RuntimeWarning: coroutine 'AsyncFileContentsManager.file_exists' was never awaited                                        
  if self.file_exists(path):
[E 2022-12-09 21:28:42.212 ServerApp] Uncaught exception POST /api/contents/notebooks?1670617722204 (127.0.0.1)
    HTTPServerRequest(protocol='https', host='localhost:8888', method='POST', uri='/api/contents/notebooks?1670617722204', version='HTTP/1.1', remote_ip='127.0.0.1')
    Traceback (most recent call last):
      File "/home/user/miniconda3/envs/pangeo/lib/python3.8/site-packages/tornado/web.py", line 1713, in _execute
        result = await result
      File "/home/user/miniconda3/envs/pangeo/lib/python3.8/site-packages/jupyter_server/services/contents/handlers.py", line 227, in post
        await self._new_untitled(path, type=type, ext=ext)
      File "/home/user/miniconda3/envs/pangeo/lib/python3.8/site-packages/jupyter_server/services/contents/handlers.py", line 175, in _new_untitled
        self.contents_manager.new_untitled(path=path, type=type, ext=ext)
      File "/home/user/miniconda3/envs/pangeo/lib/python3.8/site-packages/jupytext/contentsmanager.py", line 404, in new_untitled
        config = self.get_config(path)
      File "/home/user/miniconda3/envs/pangeo/lib/python3.8/site-packages/jupytext/contentsmanager.py", line 572, in get_config
        self.cached_config.config = self.load_config_file(
      File "/home/user/miniconda3/envs/pangeo/lib/python3.8/site-packages/jupytext/contentsmanager.py", line 541, in load_config_file
        config_content = model["content"]
    TypeError: 'coroutine' object is not subscriptable
[W 2022-12-09 21:28:42.241 ServerApp] wrote error: 'Unhandled error'
[E 2022-12-09 21:28:42.241 ServerApp] {                                   
      "Host": "localhost:8888",                                                                                 
      "Accept": "*/*",                                                  
      "Referer": "https://localhost:8888/lab/tree/notebooks/Untitled1.ipynb",                     
      "User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:107.0) Gecko/20100101 Firefox/107.0"
    }                                                                                     
[E 2022-12-09 21:28:42.241 ServerApp] 500 POST /api/contents//notebooks?1670617722204 (aed08db9c9664c0eb7475959769f76c5@127.0.0.1) 30.79ms referer=https://localhost:8888/lab/tree/notebooks/Untitled1.ipynb                                                                                             
/home/user/miniconda3/envs/pangeo/lib/python3.8/site-packages/tornado/web.py:1728: RuntimeWarning: coroutine 'AsyncFileContentsManager.get' was never awaited                                         
  self._prepared_future.set_result(None)                               

[kevin-bates]

@st-bender - thanks for the update. Since it looks like jupytext isn't compatible with the async contents manager you should be able to use server 2.0 by configuring the synchronous contents manager either on the command line via:

jupyter lab --ServerApp.contents_manager_class=jupyter_server.services.contents.largefilemanager.LargeFileManager

or configuration file via:

c.ServerApp.contents_manager_class = jupyter_server.services.contents.largefilemanager.LargeFileManager

Could you please try either of those to see if that gets you further? (Although please note that I only confirmed the CLI option prior to writing this response and do not have jupytext installed.)

Looking at the jupytext code, it looks like its definitely deriving its contents manager from LargeFileManager and, I would assume, users need to configure the server to use jupytext's contents manager (similar to what I list above), so it's strange how AsyncLargeFileManager is getting into the mix at all.

(Hmm, on a side note, I suspect we might have a general "rename on first save" issue here (per my previous response).)

[st-bender]

Hi @kevin-bates
Thanks for the update, I will update the description to include jupytext since it seems that this might be the underlying cause.

I can confirm that it works with setting the config option you suggested, it also works with the suggestion from the linked issue:

c.NotebookApp.contents_manager_class = jupytext.TextFileContentsManager

or

c.ServerApp.contents_manager_class = jupytext.TextFileContentsManager

(Although in practice I used the .json configuration file, not sure if that matters.)

I guess a proper fix would be for jupytext to update the config file in <env>/etc/jupyter/jupyter_server_config.d/jupytext.json to include that setting by default.

[st-bender]

PS
Seems like I jumped to conclusions, setting that option in <env>/etc/... does not work, it only works when in one of the main configuration files.

The code you linked to seems to fail or is ignored for some reason, I observed the following in the console output.

Not working (i.e. without config option):

[I 2022-12-10 16:39:40.402 ServerApp] [Jupytext Server Extension] Deriving a JupytextContentsManager from AsyncLargeFileManager

Working:

[I 2022-12-10 16:28:55.311 ServerApp] [Jupytext Server Extension] NotebookApp.contents_manager_class is (a subclass of) jupytext.TextFileContentsManager already - OK

Cheers.

[mwouts]

Indeed the JupytextContentsManager(s) build from the new async classes don't work properly. See here for an analysis of the issue and two possible workarounds.

I am willing to fix that. I see that the documentation mentions example tests on the Jupyter contents managers, but I can't seem to find them at jupyter_server.services.contents.tests. They would be very helpful as I don't know much of async at the moment... Could you point me at the right location?

Also the problem was not spotted in the Jupytext CI as the Jupytext CM used in the test is derived from a LargeFileManager. Is there a way to obtain programmatically the Jupyter-prefered CM so that I run the tests using that one?

Thanks!

[blink1073]

Hi @mwouts, you can use the following to check:

>>> from jupyter_server.serverapp import ServerApp
>>> ServerApp.contents_manager_class.default()
jupyter_server.services.contents.largefilemanager.AsyncLargeFileManager

As for tests, they have moved to https://github.com/jupyter-server/jupyter_server/tree/main/tests/services/contents

[mwouts]

Thank you @blink1073 , your two comments above are super helpful.

I am undecided on what is the best approach, may I ask what solution you think is best?

1. Stay with the LargeFileManager by default
2. Make Jupytext's contents manager always async
3. Do both (1 for the next minor version update, then 2 for the next intermediate version update)

I started studying 2 at mwouts/jupytext#1021 and I got a (small) fraction of my own tests passing. As I am not very much used to async / await, I would find it very helpful if you could review the changes. Do you think it is possible to make them simpler? Do you see anycase where we can do something smarter than await?

Also - sorry that's probably a beginner question, and probably not a great idea neither? - would it make sense (e.g. for avoiding impacting the test suite too much) to let the Jupytext CM not async, and use asyncio.run rather than await?

[blink1073]

We've moving toward a state of all the managers being async, so I'd say now is a good time to do a full transition. The async class will work in all versions of Jupyter Server. The pattern you're using of using async def and await ensure_async() looks good to me, most likely you'll need to keep going to cover all of the async methods in AsyncLargeFileManager. You may want to parameterize your tests to cover both forms of managers to be sure everything is working, as we do here.

[kevin-bates]

Thanks for the update @st-bender - glad there's a workaround using 2.0. What I'm still confused about, since jupytext's contents manager (JupytextContentsManager) derives from the synchronous form, is how an instance of AsyncLargeFileContentsManger is getting into the execution path of JupytextContentsManager. @blink1073 or @mwouts, do you have any insight into what is going on?

(This will likely be moot once jupytext is updated, but I'd like to understand how the expectation of async methods is creeping into this scenario. I must be missing something obvious.)

[mwouts]

Hi @kevin-bates , well actually in Jupytext there's not a single contents manager, but instead a function build_jupytext_contents_manager_class which, given a base contents manager class, returns another CM class that can handle text notebooks.

That might sound a bit complex, but I decided to do so when the LargeFileManager was moved from notebook to jupyter_server as I could not find an other way to make sure the CM would derived from the expected base class. And in some (rare) cases it also has its utility like when the users want to build a CM based on e.g. jupyterfs.metamanager.MetaManager.

Now, unless the user has a specific config, that function build_jupytext_contents_manager_class is called with app.contents_manager_class as its argument in the Jupytext server extension, and this app.contents_manager_class just became AsyncLargeFileContentsManger in jupyter_server==2.0.0 (see more details here).

As a quick fix jupytext==1.14.3 will use LargeFileManager instead:

https://github.com/mwouts/jupytext/blob/b7a8dac60a388329491847fb2d22150dec1322e0/jupytext/__init__.py#L36-L50

Then I plan to make Jupytext's CM async in jupytext==1.15.0 (WIP at mwouts/jupytext#1021)

[st-bender]

Hi @kevin-bates
Since a temporary fix has been published downstream, and the issue is only indirectly caused by jupyter_server, feel free to close this issue.
Cheers.

[blink1073]

Thanks @st-bender for the report, and @mwouts for the fix!