Use JSON Schema to define and document MermaidConfig

[aloisklink]

📑 Summary

Currently, the MermaidConfig definition is a bit broken.

• The default values for the configuration are stored in packages/mermaid/src/defaultConfig.ts.
• The documentation for the configuration is stored in packages/mermaid/src/defaultConfig.ts.
  However, it uses documentationjs, which is broken with our current workflow, and has been removed from the Mermaid website since we migrated to Vitepress (see 9468bff).
• The TypeScript definition is located in packages/mermaid/src/config.type.ts, but the types often conflict with the types described by the documentation.

This PR proposes instead moving as much as possible into a JSON Schema definition.

From a JSON Schema file, we can:

• Automatically generate documentation in markdown, using @adobe/jsonschema2md.
  This has been added in f9d3d53.
  • See a preview of this documentation at https://aloisklink.com/mermaid/config/schema-docs/config.html
• Automatically generate TypeScript definitions using json-schema-to-typescript.
  • The script to do this has been added in a156446, and is backwards compatible with the old TypeScript definition.
    • For a future version of Mermaid, we probably want to make the MermaidConfig stricter, but that will be a breaking change, so it might need to wait for Mermaid v11.
  • The changed TypeScript file can be found in 1e8d266.
• Automatically generate default JSON values using ajv.
  • This is done by a custom Vite plugin I wrote in 81b7d74.
  • Non-JSON values like undefined and function are still located in
    defaultConfig.ts.
  • As a temporary measure, I've added a test in 2c7b50f that checks that the old defaultConfig is the same as the new defaultConfig. This test can be git revert-ed once this PR is merged, and there is no risk of merge conflicts.
• Validate user submitted configuration and print human-readable warnings/errors
  (i.e. like what eslint uses for plugins)
  (not in this PR, since it will be a breaking change).

Before (broken)

Two files to edit (defaultConfig.ts and config.types.ts) with broken documentation.

flowchart TD
    default(defaultConfig.ts) -->|documentation.js **BROKEN**| docs[Documentation]
    configTypes(config.types.ts)

Loading

After

One file to edit (schemas/config.schema.yaml JSON Schema) that auto-generates defaultConfig.ts, config.types.ts, and documentation.

flowchart TD
    schema(schemas/config.schema.yaml JSON Schema)
    schema -->|.vite/jsonSchemaPlugin.ts| default[defaultConfig.ts]
    schema -->|docs.mts using jsonschema2md| docs[Documentation]
    schema --> |create-types-from-json-schema.mjs\nusing json-schema-to-typescript| configTypes(config.types.ts)

Loading

📏 Design Decisions

• The documentation is generated in the same packages/mermaid/src/docs.mts script as the rest of the markdown documentation. However, I've decided to only put it in Vitepress, and not in the docs/ folder, mainly because it creates 100s of files, which slows down git, and would make this PR massive.
  • I've uploaded a preview of the documentation at https://aloisklink.com/mermaid/config/schema-docs/config.html
• I've made sure that the new packages/mermaid/src/config.type.ts definition is backwards compatible with the old one. This however, has some issues:
  • A bunch of types for variables like securityLevel are changed from a string/number to an enum. (e.g. securityLevel type is "strict" | "loose" | "antiscript" | "sandbox"). To ensure backwards compatibility, I've added a | string to them, but in the future, they should probably just be an enum.
  • All the TypeScript properties are optional. Ideally, anything that is documented as required, should be a required type. In the future, we should mark these as required in MermaidConfig, and change functions like setConfig() to only accept a Partial<MermaidConfig>.
  • In order to make the git diff simpler, the script that generates packages/mermaid/src/defaultConfig.ts has some custom code to make sure everything is output in the same order as it was before. We can delete this bit later, once we no longer need to worry about git merge conflicts.
  • The packages/mermaid/scripts/create-types-from-json-schema.mjs script I wrote is a bit hacky.
    • Ideally, we should extract some of the common functions from packages/mermaid/src/docs.mts, so that both of these scripts can re-use the same code and functions.
    • It's also not written in TypeScript, mainly because this script creates the MermaidConfig TypeScript file, but also uses the MermaidConfig TypeScript file. We may want to fix this in the future.
• packages/mermaid/src/defaultConfig.ts:
  • The JSON default values are automatically generated when running pnpm run build or pnpm run test from the default: keys in the JSON Schema, by the custom .vite/jsonSchemaPlugin.ts file I made, that uses ajv to get the default values.
    • There is a tiny difference in size (~400 bytes), mainly because my custom Vite plugin quotes everything (e.g. it does {"key": "bar"} instead of {key: "bar"}). It's probably not worth changing.
  • Currently, gitGraph.useMaxWidth is undefined. I've changed this to false in my implementation.
    However, every other diagram uses true, so should we also set gitGraph.useMaxWidth to true?
  • A bunch of options have explicit default values of undefined, which isn't a valid JSON type,
    so we can't put their default values into the JSON Schema.
    Can we instead make their default values null?

Generated Documentation Preview

See https://aloisklink.com/mermaid/config/schema-docs/config.html if you want to explore the docs yourself.

📋 Tasks

Make sure you

• 📖 have read the contribution guidelines
• 💻 have added unit/e2e tests (if appropriate)
  • E2E tests show that no snapshots have changed.
• 📓 have added documentation (if appropriate)
• 🔖 targeted develop branch
  • I've targeted commit fec193e, which is on both develop and release/10.0.0, so that this PR should also merge cleanly into the release/10.0.0 branch.
    However, since this PR is pretty big, it can wait until after v10.0.0, especially since this PR doesn't yet have any breaking changes.

It's probably worth asking most of the core contributors if they'd be comfortable writing in JSON Schema, instead of modifying defaultConfig.ts/config.types.ts.

[sidharthv96]

This is wonderful. Even the PR description is top-notch!

Nits:
Can this be clickable link? Saw this in all files.

Currently, gitGraph.useMaxWidth is undefined. I've changed this to false in my implementation.
However, every other diagram uses true, so should we also set gitGraph.useMaxWidth to true?

Yes.

---

A bunch of options have explicit default values of undefined, which isn't a valid JSON type,
so we can't put their default values into the JSON Schema.
Can we instead make their default values null?

We'll have to check the impact during runtime. There were checks like typeof x === 'undefined' and x === undefined, which would fail if we use null.

[aloisklink]

Git force push changes

I've git rebase-d onto develop to fix merge conflicts (and some PR comments).

You can use the following command to view the changes (GitHub's UI isn't great for viewing the difference between two patch series):

git range-diff fec193ebaf85403e3bc8031d79b4f593a5db1c31..74257756fc7e1fc773aca1c882a47c17c2b80cae  8b5cb75ef70ee1ccc66373cadffae7e5e7425213..430830ada64cc3d8e79fb5c15213f8ca964bbf7b

Main changes:

• Removed lazyLoadedDiagrams and loadExternalDiagramsAtStartup from the MermaidConfig schema, since 543e4de removed them (these were previously marked as @deprecated anyway.
• Switched the pnpm run pre-commit script to automatically fix and git add the changes in packages/mermaid/src/config.types.ts (see Use JSON Schema to define and document MermaidConfig #4112 (comment)). Previously, it just threw an error if this file wasn't in sync.
• Fixed me adding @adobe/jsonschema2md as a devDependency in the wrong commit (doesn't change this PR, but it does make the git history slightly cleaner).
• Fixed links to schema.json not working. Vitepress apparently doesn't handle .json files properly, so as I work around, I've stuck it into Vitepress's public/ folder, see 430830a

Rely to #4112 (comment)

This is wonderful. Even the PR description is top-notch!

Perhaps it's a bit too wordy though :) Maybe I need to ask ChatGPT to summarize it for me, hahaha. Unfortunately, I couldn't really figure out a way to split this up into smaller PRs.

Nits:
Can this be clickable link? Saw this in all files.

The markdown that is generated creates it as a code-block:

```txt
https://mermaid-js.github.io/schemas/config.schema.json
```

To be honest, we could make it into a clickable URL, but I'm not sure we should, since @adobe/jsonschema2md might have a good reason to make it into a code block (feel free to view example @adobe/jsonschema2md output here: https://github.com/adobe/jsonschema2md/blob/main/examples/docs/complex.md).

I think it might be because you're never expected to click on it (it's just an ugly JSON file), it's mainly just used as a unique ID, or for programs to download the schema.

See https://aloisklink.com/mermaid/schemas/config.schema.json for an example of what the file looks like (I've pretty-printed it to make it look nicer). Apparently, Vitepress wasn't bundling .json files properly, so I had to change the script slightly to output it to public/ to get it to appear!

Currently, gitGraph.useMaxWidth is undefined. I've changed this to false in my implementation.
However, every other diagram uses true, so should we also set gitGraph.useMaxWidth to true?

Yes.

Awesome, that will be something to discuss/fix in the next PR (it should be a separate discussion, since previous changes to useMaxWidth have caused issues to the mermaid-cli project, see mermaid-js/mermaid-cli@2c4fd1a)

[aloisklink]

Git force push changes

I've git rebase-d onto develop (aka commit 7647ae3) to fix merge conflicts.

You can use the following command to view the changes (GitHub's UI isn't great for viewing the difference between two patch series):

git range-diff 8b5cb75ef70ee1ccc66373cadffae7e5e7425213..430830ada64cc3d8e79fb5c15213f8ca964bbf7b 7647ae317a7b2130e32777248d25a9c4d24b8f9f..049d1c13d3d91614490bac2d1c023998cc411227

Changes (found by running git diff 8b5cb75ef70ee1ccc66373cadffae7e5e7425213..develop ./packages/mermaid/src/config.type.ts ./packages/mermaid/src/defaultConfig.ts):

• Added the textPosition option to piechart config (from 82f5b4c)
• Added nodeSpacing/rankSpacing/diagramPadding/htmlLabels to class diagram config (from 46f2aeb)
  Minor change, I've made htmlLabels default to false instead of undefined
  (this is different from flowchart diagrams, which currently defaults to true).
  Currently undocumented.
• Added padding option to timeline diagrams (added in 2ab1e15).
  Currently undocumented.

Edit: Fixed broken links and wrong formatting that was making CI fail.

[knsv]

This looks great! Good job @aloisklink!

[aloisklink]

I've fixed merge conflicts in the pnpm-lock.yaml file.

It looks like we have a use-inline-specifiers-lockfile-format=true config setting in our .npmrc file that changes the lockfile setting:

mermaid/.npmrc

Line 3 in e4d2118

use-inline-specifiers-lockfile-format=true

However, renovate seems to ignore this setting, so it's constantly converting our pnpm-lock.yaml file, which causes a bunch of merge conflicts.

Edit: I've made a PR to fix this, see #4249

---

Is it worth getting a review from @ashishjain0512 too (maybe not a full review, but at least making sure they're okay with using a new method to declare the default values, types, and documentation for MermaidConfig)? They seem to be the main other @mermaid-js member that has edited these files.

[jgreywolf]

What is the status on this?

[codecov]

Codecov Report

Merging #4112 (29291c8) into develop (1a4e5d9) will increase coverage by 7.43%.
The diff coverage is 100.00%.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #4112      +/-   ##
===========================================
+ Coverage    69.82%   77.26%   +7.43%     
===========================================
  Files          143      143              
  Lines        16489    14455    -2034     
  Branches       518      516       -2     
===========================================
- Hits         11513    11168     -345     
+ Misses        4848     3182    -1666     
+ Partials       128      105      -23     

Flag	Coverage Δ
e2e	84.32% <ø> (+16.00%)	⬆️
unit	45.28% <100.00%> (-12.84%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
packages/mermaid/src/defaultConfig.ts	46.81% <100.00%> (-47.02%)	⬇️

... and 41 files with indirect coverage changes

[aloisklink]

What is the status on this?

Sorry for the delay! Since this is a pretty big PR the changes how the MermaidConfig is defined, fixing merge conflicts is a bit of a pain. IMO, since this PR has been open for a while, I guess anybody who wanted to review it already has. Anybody mind if I go ahead and merge this after PR #4580 gets merged?

---

I've git rebase-d onto 9c011ab, aka the latest develop branch commit.

git range-diff 7647ae317a7b2130e32777248d25a9c4d24b8f9f..049d1c13d3d91614490bac2d1c023998cc411227 9c011abbd424641172b696c038490a36ec99c518..85423656fe46edc1461893c62d9c01677be4688e

Changes:

• Improve wording of security level values (from a991c32)
• Added wrappingWidth to flowchart options (from 4caf7d7)
• Added quadrant chart options (from Added support for quadrant chart #4383)
  • I've made all options Required, even though the documentation says Optional, since the TypeScript types made them Required.
• Refactor: Moved Gantt Chart displayMode definition to align with TypeScript position change made in ba1c5dc
• Added sankey diagram options (from Sankey diagrams #4502)
  • I'll included some of the changes from Remove all TypeScript enums and forbid them in ESLint #4580 to avoid merge conflicts.
  • I've also added some documentation for these options.
  • The default value of useMaxWidth is false for sankey diagrams. However, every other diagram has a useMaxWidth: true default value. Is this correct?

[sidharthv96]

The default value of useMaxWidth is false for sankey diagrams. However, every other diagram has a useMaxWidth: true default value. Is this correct?

@nirname, feels like it should be true, any reason for setting false as the default?

[Yokozuna59]

I just want to point out something: these changes would work fine with the current approach, but they're most likely to break or change when applying the stuff discussed in #4499, i.g., moving types from config.types.ts into each diagram types files. #4499 (comment)

[aloisklink]

I've git rebase-d onto c742ac7, aka the latest develop branch commit + the 38013de commit from #4602.

git range-diff 9c011abbd424641172b696c038490a36ec99c518..0477a9f68a8d6f474dc792d430ca892d1aa41b66 c742ac71a4f59455ce3dc0b6ac56d5f9b2baa4d2..c29088af019a8180ac2f85d79b890cab4cd5fee5

Changes:

• Make quadrant chart options TypeScript types optional (from 38013de in PR Make quadrant chart options TypeScript types optional #4602)
• Change default sankey width (from 9f5f0a6)
• .vite/jsonSchemaPlugin.ts: Minor refactoring to fix comments from @sidharthv96
• packages/mermaid/scripts/create-types-from-json-schema.mjs: convert to .mts and fix TypeScript issues .
  Because this file imports some types it creates, type checking for this file will be run with pnpm run --filter mermaid types:verify-config, but not with pnpm run --filter mermaid types:build-config.

---

I just want to point out something: these changes would work fine with the current approach, but they're most likely to break or change when applying the stuff discussed in #4499, i.g., moving types from config.types.ts into each diagram types files. #4499 (comment)

We could have a separate config.schema.yaml for each diagram, then have a root config.schema.yaml that includes them all, something like:

packages/mermaid/src/
├── schemas/
│   └── config.schema.yaml # imports all the diagrams/*/config.schema.yaml to create the master MermaidConfig
└── diagrams/
    ├── baseDiagram/
    │   └── config.schema.yaml
    ├── my-cool-diagram/
    │   └── config.schema.yaml # imports and extends ../baseDiagram/config.schema.yaml

Honestly, it would be even better to split things up. The current config.schema.yaml is almost 2000 lines of code, so it's a big. And the generated Vitepress page (at http://localhost:3333/config/schema-docs/config.html) is a 1.2 MiB download. The main reason I haven't done it in this PR is to keep things as similar as possible to the existing code.

[aloisklink]

Sorry for accidentally closing this PR (it's my fault, not @sidharthv96!) I wrote Resolves https://github.com/mermaid-js/mermaid/pull/4112#discussion_r1250397326 in PR #4602, so GitHub auto-closed this PR when that one was merged.

[Yokozuna59]

We could have a separate config.schema.yaml for each diagram, then have a root config.schema.yaml that includes them all, something like:

packages/mermaid/src/
├── schemas/
│   └── config.schema.yaml # imports all the diagrams/*/config.schema.yaml to create the master MermaidConfig
└── diagrams/
    ├── baseDiagram/
    │   └── config.schema.yaml
    ├── my-cool-diagram/
    │   └── config.schema.yaml # imports and extends ../baseDiagram/config.schema.yaml

Honestly, it would be even better to split things up. The current config.schema.yaml is almost 2000 lines of code, so it's a big. And the generated Vitepress page (at http://localhost:3333/config/schema-docs/config.html) is a 1.2 MiB download. The main reason I haven't done it in this PR is to keep things as similar as possible to the existing code.

Yeah, splitting might be the way to go.
I think we should also add the configuration of each diagram to its own page. If it were me, if I opened the docs, I wouldn't want to open multiple pages to know the syntax, possible themes, and configuration variables. I saw quadrant-chart 's page and really like this approach; the page contains almost everything.

I'm not saying to drop Theming and Config pages; it would help and allow users to see through other possible variables in other diagrams and, of course, to make existing links work :). But I'm a bit weird about the flowchart page; it's already large (my phone hangs when opening the page :) ).

---

I likes the idea of baseDiagram directory, I think it would help to organize things, like moving base types, i.e.,BaseDiagramConfig.

[sidharthv96]

I love the split config idea. Let's do it in another PR.