[myst] Implement Markdown output parsing for mystmd

[agoose77]

Jupyter Book 2 beta requires Markdown output parsing to restore feature-parity with Jupyter Book 1 (see jupyter-book/mystmd#1026). To facilitate this, we have launched #5395 which sets up the pre-requisite AST changes.

Context

Whilst #5395 adjusts the AST to support a one-to-one association between IOutput messages and the AST, we still need a policy on what the children of these (per-IOutput) Output nodes should contain. IOutput bundles can contain multiple MIME types, and there are several possible ways to materialise these into the AST.

In order to decide upon the most appropriate course of action, we need to consider the intention of the AST. Although the MyST engine operates on different ASTs for different build backends, it treats the web-build as the canonical sharing format (xrefs). This means that we want to avoid making web-first decisions in the shared AST if that makes non-web builds of the shared content avoidably sub-optimal (e.g. embedding a low-res small image into the xref AST which is then embedded into a PDF).¹

Suggested Actions

1. Implement a generic "preferred MIME type" hierarchy.
   Most of the time, an IOutput bundle contains only a few MIME types, and there's usually a clear preference.
   The complicated scenario is if there are different kinds of images. We can probably have mystmd just prefer a single output and assume it's fine across all backends (e.g. take the SVG over PNG, text).²
2. Implement transform for lifting preferred outputs from IOutput bundles, including Markdown, into the AST.
3. Add support to myst-theme to render parsed AST and widgets.

As part of our "tactical upstream work", this issue tracks the implementation of this feature.

Definition of Done

• Markdown output is parsed as MyST
• Markdown sub-AST is visible in published site AST
• Markdown output is enumerable e.g. figures alongside other MyST content

Footnotes

1. This is a wider conversation, though. ↩

2. Or, perhaps we choose a preferred class of output (e.g. images), and de-dupe the image paths via an "image resolver" that can be nudged with a preferred image type. ↩