Use with Quarto / NBClient / nbconvert?

[allefeld]

I'm currently developing a Matlab toolbox, and I use Quarto to create documentation. Quarto can execute code in documents by interacting with a Jupyter kernel. For a preliminary version of my documentation see e.g. here.

So far I have been using the unofficial matlab_kernel which works generally okay, and with which the document linked to above has been created. However, it does have a few annoying bugs, which are unlikely to get fixed anytime soon.

So I was happy to see that now there is an official Jupyter kernel for Matlab. I installed the package, and after a bit of fiddling around, mostly related to convincing Matlab that I am a licensed user, I got it to work in Jupyter Lab.

Unfortunately, the same does not hold for a Quarto document. Processing the document

---
title: Test jupyter_matlab_kernel
jupyter: jupyter_matlab_kernel
---

```{matlab}
ver
```

leads to

Debian GNU/Linux 11.7, kernel 5.10.0-23-amd64
conda 23.1.0
Python 3.10.5
jupyter_core 5.3.1
jupyter-matlab-proxy 0.6.0
MATLAB 9.13.0.2166757 (R2022b)

[allefeld]

My suspicion is that this is related to the licensing?

Is there a way to settle the question of being licensed in a specific step once and for all, so that subsequently tools like Quarto can simply use the kernel?

To be honest, I don't understand why you require users of the kernel to explicitly confirm the license – my locally installed copy of Matlab already has an activated license. Why can't the kernel simply ask the installed copy of Matlab whether it is licensed (in my case through a lic file in /opt/MATLAB/R2022b/licenses/), and that's it?

[rashedmyt]

Hi @allefeld

Thank you for trying out MATLAB Integration for Jupyter.

1. The current implementation of MATLAB Kernel uses matlab-proxy for communication with MATLAB. matlab-proxy requires a user to provide licensing information at least once. Hence, further MATLAB Kernel usage after completing the licensing once should re-use the cached licensing information. As of v0.7.0, matlab-proxy provides the capability to use a MATLAB which is already activated.

2. From the error displayed in the attached screenshot, I am able to ascertain that the issue is not related to licensing. Currently, MATLAB Kernel depends on the Jupyter server which launches it. It seems like the tool you are using is starting MATLAB Kernel directly instead of using a Jupyter server because of which you are seeing the error.

Can you provide an example Jupyter notebook and the command used to generate the documentation so that we are able to reproduce the issue?

[cderv]

It seems like the tool you are using is starting MATLAB Kernel directly instead of using a Jupyter server because of which you are seeing the error.

Just to try to help identify the limitation. In Quarto we are leveraging nbclient and we are executing the notebook that way
https://nbclient.readthedocs.io/en/latest/client.html

Is there any limitation for this matlab kernel in that case ?

[allefeld]

@rashedmyt, thanks for the reply. Good to know it is not a licensing issue.

In included an example document in the issue. You can reproduce the error if you copy the text in the first codeblock into a file with the extension qmd, e.g. quarto-matlab.qmd, and then run the command

quarto render quarto-matlab.qmd

(for which of course Quarto needs to be installed). Quarto creates a file quarto-matlab.html, which if opened in a browser looks like the image I included.

My understanding of what happens internally (@cderv correct me if I'm wrong):

• Quarto converts the qmd file (Markdown with code blocks) into a notebook file quarto-matlab.ipynb which contains text and code input cells, but no output cells.
• Quarto uses NBClient to execute the notebook, filling in output cells.
• Quarto converts the notebook file into a Markdown file quarto-matlab.md where the output cells are represented by Markdown code.
• Quarto processes this Markdown file using a combination of Pandoc with its own logic. Based on Pandoc functionality, the result can be in a large variety of formats, but the default is HTML.

If this description is correct, then as @cderv pointed out, the part that fails is most likely the notebook execution via NBClient. As far as I can tell, NBClient is an official part of Juypter.

I.e. instead of testing with Quarto, you could directly use NBClient to reproduce the problem. From the command line it is called as

jupyter execute quarto-matlab.ipynb

I just tried that, and with the same result. The respective notebook does not have to be created by Quarto, it can be any notebook using your kernel.

notebook code

{
 "cells": [
  {
   "cell_type": "raw",
   "id": "bfcbc676",
   "metadata": {},
   "source": [
    "---\n",
    "title: Test jupyter_matlab_kernel\n",
    "keep-ipynb: true\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "7d84d83e",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "            Error: MATLAB Kernel for Jupyter was unable to find the notebook server from which it was spawned!\n",
      "\n",
      "            Resolution: Please relaunch kernel from JupyterLab or Classic Jupyter Notebook.\n",
      "            "
     ]
    }
   ],
   "source": [
    "ver"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "MATLAB Kernel",
   "language": "matlab",
   "name": "jupyter_matlab_kernel"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}

[allefeld]

Maybe a note why supporting Quarto is a good idea:

Quarto is the successor to RMarkdown, and in contrast to it it is not focused on R but supports arbitrary languages for which a Jupyter kernel exists. It also incorporates a lot of functionality (e.g. complete website or complete book creation from multiple documents) for which RMarkdown needed extra software. RMarkdown is the standard for publishing computational documents based on R, and Quarto is well on its way to take on the same role more generally. It is similar to nbconvert, but much more feature-rich and polished, with high-quality templates and seamless integrations into a number of IDEs.

[allefeld]

Just out of curiosity, I checked nbconvert, and it seems it has the same problem:

jupyter nbconvert --execute --to html quarto-matlab.ipynb

produces

So not just Quarto via NBClient, but also another standard Jupyter tool is affected.

[cderv]

@allefeld I think you got it all right. FYI quarto can be used on .ipynb file directly so quarto render test.ipynb. Not need to start from .qmd.

Anyhow, as you detailed after, this is probably something with nbclient execution yes. It needs to be supported for Quarto be able to work.

[allefeld]

This might be relevant: jupyter/nbclient#213

[rashedmyt]

Thank you for the response.

I was able to successfully reproduce the issue. As I have mentioned, the current implementation of MATLAB Kernel expects a Jupyter server to launch the kernel.

I have forwarded the use case to our development team, who will consider supporting this in a future version of the product.

I'll reach out to you once this support is available.

[BoPeng]

Just add that this also prevents the use of matlab in a multi-kernel notebook (https://github.com/vatlab/sos-matlab)

[allefeld]

For others encountering this problem: I have created my own Matlab kernel https://github.com/allefeld/mkernel.

It does not have this problem, and therefore can be easily used with Quarto.

In comparison with Calysto's matlab_kernel, it fixes a number of that project's bugs, is better documented and configurable, and partially bypasses a bug in the Matlab Engine for Python.