feat: upgrade to MDX v2

[slorber]

Motivation

Docusaurus v3 will upgrade MDX to v2

It is much better to work with than v1 for many reasons explained here: https://mdxjs.com/blog/v2/

MDX v2 also exposes a better infrastructure for Docusaurus to build new useful features, and optimize performances.

Fix #4029

Also makes it possible to support a CommonMark mode (opt-in), requested in #3018, but by default we still use MDX (even on .md files, see also #9097)

Need support to upgrade to MDX v2? See MDX v2 Upgrade Support

Other notable changes:

• Better Webpack error messages
• Preprocessor
• remark-directive included by default and makes it easier to invent your own Markdown directives (admonitions use it, but you can create your own ::myDirective if you want.
• Ability to opt-in for CommonMark format: feat(mdx): add siteConfig.markdown.format to configure the default content parser (MDX / CommonMark) #9097

---

Breaking changes

Breaking changes coming from MDX v2 itself, notably:

• Need to update your custom Remark/rehype plugins for v2
• Lowercase JSX tags are not substituted by MDXProvider components anymore: tags like <pre> or <code> will render as is
• Html comments are not supported (➡️ compat option provided)
• Syntax for explicit heading ids is invalid (➡️ compat option provided)
• Syntax admonition title is different (➡️ compat option provided)
• Everything else documented by MDX v2 itself (v2 migration guide)
• Need to upgrade Remark-math + Rehype-katex?

Breaking changes coming from Docusaurus public APIs:

• .md and .mdx can be handled differently if you opt-in for CommonMark (
• blog truncate marker should now use {/* truncate */} for .mdx files (➡️ compat option provided)
• docs.admonitions.tag option is removed, we can now only use ::: (markdown directive syntax)
• docs.admonitions.extendDefaults option is now true by default (merge default+user-provided admonition keywords instead of replacing them)
• the mdx-code-block workaround for Crowdin is implemented in a very different way. It still works in most cases but we recommend to not use this anymore and put complex React components outside of MDX files.

Notable private API changes:

• mdx-loader requires the options.markdownConfig to be provided (see also docs: document siteConfig.markdown + better mdx-loader retrocompat #8419)

---

Compatibility options

Site config

We have a few options to keep Docusaurus v3 more retro-compatible with v2 features, that are for now turned on by default.

The goal is to give you the opportunity to progressively adopt a stricter MDX syntax.

This is important to aim for MDX compatibility, so that other tools understand your files: GitHub, VSCode, Prettier, ESLint, Crowdin... You will have to progressively turn these flags off to do so.

const config = {
  markdown: {
    // By default we compile all files as MDX (allow JSX), but you can opt-in for CommonMark if you want
    // See https://github.com/facebook/docusaurus/pull/9097
    format: 'mdx', 

    mdx1Compat: {
      // Allow html comments in MDX files like blog `<!-- truncate -->` markers
      comments: true,
      // Allow former admonition title syntax :::note Title instead of new syntax :::note[Title]
      admonitions: true,
      // Allow usage of unescaped {#headingId} syntax instead of \{#headingId}
      headingIds: true
    },

    // Gives you opportunity to apply your own string processing before the MDX compilation
    // Usage is NOT recommended, use this as a last resort escape hatch or temporarily)
    preprocessor: ({ filePath, fileContent }) => {
      return fileContent;
    }
  }
};

MDX vs CommonMark

By default, Docusaurus v3 will still compile all files as MDX (like Docusaurus v2 did), but it is possible to opt-in for CommonMark parser (does not allow usage of JSX).

You can do so:

• with site settings: siteConfig.markdown.format: 'detect' (will use CommonMark for .md files and MDX for .mdx files)
• with front matter: format: 'md'

Note: some Docusaurus features currently don't work yet in CommonMark, make sure to test this issue before adopting this new mode: #9092

---

Follow-up

Things to handle later, after this PR is merged:

• Fix relaxed types due to TypeScript not making easy to import ESM types in CJS modules (Allow ES Module Type-Only Imports from CJS Modules microsoft/TypeScript#49721) ❌
• Implement alternative syntax for heading ids, natively compatible with MDX (likely {/* headingId */}) ❌
• Document better the differences between .md and .mdx format? ✅ feat(mdx): add siteConfig.markdown.format to configure the default content parser (MDX / CommonMark) #9097
• Remove useless JS/Babel loader uselessly retranspiling already valid React components? ✅ fix: remove useless js-loader in front of mdx-loader #8972
• Expose global format setting? ✅ feat(mdx): add siteConfig.markdown.format to configure the default content parser (MDX / CommonMark) #9097
• Update starter templates to use .mdx? 🤷‍♂️
• Add components={{theme}} and use <theme.Tabs/> ? ➡️ Make most features work in CommonMark mode: head, live code blocks, tabs... #9092

---

Tests

We have many unit tests to ensure the behavior is similar to what it used to be.

We also started using visual regressions tests, comparing visually each page from our sitemap to detect potential regressions. The setup is quite hacky (using an external repo for now) but you can take a look here:

• https://github.com/slorber/docusaurus-visual-tests
• https://argos-ci.com/

Some review links:

• https://deploy-preview-8288--docusaurus-2.netlify.app/
• https://deploy-preview-8288--docusaurus-2.netlify.app/docs/
• https://deploy-preview-8288--docusaurus-2.netlify.app/blog/
• https://deploy-preview-8288--docusaurus-2.netlify.app/docs/markdown-features/admonitions/
• https://deploy-preview-8288--docusaurus-2.netlify.app/tests/pages/markdown-tests-mdx/
• https://deploy-preview-8288--docusaurus-2.netlify.app/tests/pages/markdown-tests-md/

---

Other Relevant links:

• Migrate to MDX 2.0 #4029
• https://github.com/pomber/docusaurus-mdx-2
• https://mdxjs.com/blog/v2/
• https://mdxjs.com/migrating/v2/
• https://github.com/pomber/mdx-debugger
• https://jestjs.io/docs/ecmascript-modules

[netlify]

✅ [V2]

Name	Link
🔨 Latest commit	5ab8e86
🔍 Latest deploy log	https://app.netlify.com/sites/docusaurus-2/deploys/6442c74c8c23ef0008b98b54
😎 Deploy Preview	https://deploy-preview-8288--docusaurus-2.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site settings.

[github-actions]

⚡️ Lighthouse report for the deploy preview of this PR

URL	Performance	Accessibility	Best Practices	SEO	PWA	Report
/	🟠 82	🟢 97	🟢 92	🟢 100	🟢 90	Report
/docs/installation	🟠 76	🟢 100	🟢 92	🟢 100	🟢 90	Report

[github-actions]

Size Change: +148 B (0%)

Total Size: 1.01 MB

Filename	Size	Change
website/build/assets/css/styles.********.css	113 kB	+36 B (0%)
website/build/assets/js/main.********.js	752 kB	+112 B (0%)

ℹ️ View Unchanged

Filename	Size
website/.docusaurus/globalData.json	101 kB
website/build/index.html	41.3 kB

_{compressed-size-action}

[slorber]

Note the docs says:

Unescaped left angle bracket / less than (<) and left curly brace ({) have to be escaped: < or { (or use expressions: {'<'}, {'{'})

https://mdxjs.com/docs/what-is-mdx/

Problem - Heading anchor syntax {#anchor-id}

MDX 2 interprets { as the start of an expression and it should be escaped to \{ in our case.

https://mdxjs.com/docs/troubleshooting-mdx/#could-not-parse-importexports-with-acorn-error

Solution: escape the md files with regexp for retrocompatibility and convenience (83064b2)

It's not possible to implement this as a remark plugin (too late in the process)

We probably don't want to ask users to do such escaping, nor change our existing syntax.

Note the React beta site have the same feature with a different syntax:
https://github.com/reactjs/reactjs.org/blob/main/beta/src/content/index.md

## What is this site? {/*what-is-this-site*/}

---

Problem - <!-- HTML comments

The < lead to a similar acorn error and need to be escaped.

Note: it only happens with format: "mdx", not 'md', and we may allow users to use the md format (through frontmatter) for improved compatibility and easy migration to Docusaurus.

Solution? A quick prepressing that escapes all HTML comments

---

Problem - // js comments in JS

This code in MDX doesn't work anymore due to the JS comment:

import LiteYouTubeEmbed from 'react-lite-youtube-embed';

<div className="video-container">
  <LiteYouTubeEmbed
    // cSpell:ignore Yhyx Sksg
    id="Yhyx7otSksg"
    params="autoplay=1&autohide=1&showinfo=0&rel=0"
    title="Docusaurus: Documentation Made Easy"
    poster="maxresdefault"
    webp
  />
</div>

Unexpected character / (U+002F) after self-closing slash, expected > to end the tag (note: JS comments in JSX tags are not supported in MDX

Solution? Remove the comment?

---

Problem - Math equations

$$
I = \int_0^{2\pi} \sin(x)\,dx
$$

Could not parse expression with acorn: Expecting Unicode escape sequence \uXXXX

Solution: again, escape \{ ?

$$
I = \int_0^\{2\pi} \sin(x)\,dx
$$

[slorber]

Struggling to upgrade your site to MDX v2?

Here's a discussion to ask for support: #9053

[johnnyreilly]

Just linking in a related issue here: #9081 (comment)

I'd love to see a guide on how to migrate remark / rehype plugins to Docusaurus v3 / MDX v2

I documented this a bit better already for v3, but due to Crowdin issues our website refuse to deploy for a long time... 😓

It sounds like the docs may exist - but not yet be visible!