MyST directives CodeBlock nodes normalized to Code nodes

[kdeldycke]

I'm trying to use Pandoc to lint and normalized my documentation that is written in MyST. So far so good, but I stumbled upon some edge-cases with MyST directives specified as back-ticked code-blocks. Here are my use-cases:

$ cat ./myst-code-blocks.md

```{include} ../readme.md
```

```{toctree}
:maxdepth: 2
:hidden:
install
usage
```

```{sidebar} Default layout
![](plugin.png)
```

```{admonition} Danger: **Misleading package names**
:class: danger
[*`mpm`*, the Python module](https://pypi.python.org/pypi/mpm), is not the same
as **`meta-package-manager`**. Only the later provides the {command}`mpm` CLI
*per-se*. The former has nothing to do with the current project.
```

This is the way pandoc normalize these blocks:

$ pandoc ./myst-code-blocks.md --to=gfm
`{include} ../readme.md`

``` {toctree}
:maxdepth: 2
:hidden:
install
usage
```

`{sidebar} Default layout ![](plugin.png)`

`` {admonition} Danger: **Misleading package names** :class: danger [*`mpm`*, the Python module](https://pypi.python.org/pypi/mpm), is not the same as **`meta-package-manager`**. Only the later provides the {command}`mpm` CLI *per-se*. The former has nothing to do with the current project. ``

What's funny is that the {toctree} directive seems to be recognized as a CodeBlock node and normalized as such (see the extra space introduced, as per #7616). All others are normalized back as Code nodes.

I'm not sure how to interpret these discrepancies.

• Should the {admonition} be interpreted as CodeBlock too because of the :class: attribute?
• Why is the {sidebar} not a CodeBlock if it has a non-empty body?

For reference I use the latest pandoc on macOS Big Sur:

$ pandoc --version
pandoc 2.14.2
Compiled with pandoc-types 1.22, texmath 0.12.3.1, skylighting 0.11,
citeproc 0.5, ipynb 0.1.0.1

[kdeldycke]

For context, the style of attribute-based MyST directives have been proposed there: executablebooks/MyST-Parser#63 (comment)

[kdeldycke]

I just solved the conversion from CodeBlock to Code by forcing in the input type to gfm dialect:

$ pandoc ./myst-code-blocks.md --from=gfm --to=gfm
``` {include}
```

``` {toctree}
:maxdepth: 2
:hidden:
install
usage
```

``` {sidebar}
![](plugin.png)
```

``` {admonition}
:class: danger
[*`mpm`*, the Python module](https://pypi.python.org/pypi/mpm), is not the same
as **`meta-package-manager`**. Only the later provides the {command}`mpm` CLI
*per-se*. The former has nothing to do with the current project.
```

This is much better:

• All CodeBlock are recognized as such.
• Their curly-braced directives are left untouched. Better, they're properly normalized with an extra-space.

The only issue is that all parameters after the curly braces are stripped out.

[jgm]

This isn't really a bug, because we don't claim to support MyST. I wish they had used syntax constructions that would be understood by standard parsers, rather than inventing their own syntax, and so I advised them. But alas. For example, they could have easily made the syntax:

```{include="../readme.md"}
```

which would have built on an existing, established syntax, and would be handled without problems by pandoc (indeed, the include could even be implemented as a simple pandoc filter).

The behavior of pandoc (with -f markdown) is perfectly expected, because {include} ../readme.md isn't a valid code block attribute. Hence pandoc interprets this as a code span.

[kdeldycke]

I just solved my MyST formatting issue by switching from Pandoc to mdformat (see executablebooks/MyST-Parser#440).

I guess my expectations of Pandoc being able to recognize MyST edge-cases are unfounded, so I'll close this issue.

[kdeldycke]

Thanks @jgm for the answer. Everything makes sense now. And sorry for the noise. 🙇

Indeed, the all-encompassing curly-braces syntax seems like a strong proposition.

Anyway, I should have checked with the MyST community first about the current state of the ecosystem instead of going with Pandoc (which I'll keep as my go-to tool for anything Markdown related! 😉).

[jgm]

Maybe you could put in a plug for a more compatible syntax!
(They could always add the more compatible syntax while still keeping the old one available for backwards compatibility.)