custom directives / ipynb-filters during quarto convert

[githubpsyche]

Bug description

We can convert between .ipynb and .qmd representations of a notebook using the quarto convert command.

However, some notebooks have custom directives. These are normally supported during project compilation using the ipynb-filters option in _quarto.yml and are essential to the nbdev project where directives such as #|export select code cells for compilation into a python library.

quarto convert doesn't work on notebooks designed to make use of this advanced feature.

If an ipynb file had the code cell:

#| export
import numpy as np

converting it into QMD will produce the cell:

#| '0': e
#| '1': x
#| '2': p
#| '3': o
#| '4': r
#| '5': t
import numpy as np

and converting back will discard the directive completely:

import numpy as np

Starting out with a correct #|export directive in the QMD version of the file and attempting to convert the file into ipynb produces the following error:

> ERROR: TypeError: Cannot delete property '0' of [object String]

Ideally what would happen during a round-trip conversion is that the directives included at the top of code cells are preserved, just as natively supported directives are presented.

If quarto convert or some other tool could handle the round trip conversion between ipynb and qmd more cleanly, then nbdev projects and other tools utilizing the advanced nb-filter feature could use qmd source files for their work instead of jupyter notebooks! And the advantages of QMD over IPYNB as a format for work mixing text and code probably don't need to be outlined here.

Simple example file that quarto convert will struggle with:
https://gist.githubusercontent.com/githubpsyche/32c6391c30699bfd103b4992743b0af3/raw/a17f651cc28a9bb4f3afeab14bb9259ebc0bd0f6/example.ipynb

Using Windows 11. I believe I've tried this with the current release 1.1.251 and 1.2.253.

Checklist

• Please include a minimal, fully reproducible example in a single .qmd file? Please provide the whole file rather than the snippet you believe is causing the issue.
• Please format your issue so it is easier for us to read the bug report.
• Please document the RStudio IDE version you're running (if applicable), by providing the value displayed in the "About RStudio" main menu dialog?
• Please document the operating system you're running. If on Linux, please provide the specific distribution.

[cscheid]

I think what's happening here, code-wise, is that export is a valid YAML string. And then something on our side is iterating over entries in that yaml object (expecting it to be a mapping, but seeing an export string instead) and then ends up iterating character-by-character instead of over key-value pairs.

[githubpsyche]

A temporary workaround I've found somewhat effective is to start a code cell with a line that contains a single space character - - before adding export and other directives. But this only really works for a one-way conversion.

The original file can still be rendered with nbdev and quarto without issue, and a format-converted file has the whitespace removed and works properly.

[sswam]

The main problem is that Quarto assumes that the directive lines are valid YAML, but nbdev directives aren't.

I've come up with a somewhat hacky workaround, which converts any nbdev directive foo to foo: null before attempting to parse it as YAML, e.g.:

#| default_exp foo
#| export

is changed to:

#| default_exp foo: null
#| export: null

When writing a code cell, it strips off the null value and just shows the plain nbdev directive again. I'm assuming that Quarto directives don't ever use a value of "null".

With these changes, it seems to be stripping the nbdev directives out of my blog posts when rendering from ipynb, and that's okay with me.

It can't currently render from qmd format if these nbdev directives are used. We would need to adjust something in how it's reading the YAML there too.

I'll submit a PR, although it's pretty hacky and I don't expect it would be accepted as it is.

[cscheid]

(@hamelsmu: When this PR comes up, do you mind spending some time with me so we can figure out how to make this do the proper thing for everyone? Thanks!)

[hamelsmu]

@cscheid Thanks for pinging me on this! I encounter this issue as well and have to remove the nbdev directives via a notebook filter like this, which probably isn't ideal for everyone (I'm only removing one directive there).

The script: https://github.com/hamelsmu/hamel/blob/master/quarto_filter.py
The yaml: https://github.com/hamelsmu/hamel/blob/master/_quarto.yml#L5

Please ping me on the PR and I will take a close look. What is the idea on how to fix this? I'm not sure I see a clean way.

P.S. IIRC @jph00 and @jjallaire discussed this quite some time ago and tried to agree on a common directive format that would play nicer with each other. I'm not sure if the thinking has changed or not since then so this is why I'm mentioning that conversation

cc: @seeM

[cscheid]

I'm not sure I see a clean way either, but before I come up with a solution (or be convinced that there isn't one) I would like to understand as much of the landscape as I can.

[sswam]

Rather than doing something hacky like this PR: #4807

It might be better to change nbdev to use YAML-format directives also,
e.g.
#|export: is valid YAML equivalent to #|export: null
and
#|default_exp: foo would be a good alternative to #|default_exp foo.
Could possibly use some sort of hook to add the ":" characters if a user forgot them.