mdox

mdox (spelled as md docs) is a CLI for maintaining automated, high-quality project documentation and website leveraging GitHub Flavored Markdown and git.

This project can be used both as CLI as well as a library.

Goals

Allow projects to have self-updating up-to-date documentation available in both markdown (e.g readable from GitHub) and static HTML. Hosted in the same repository as code and integrated with Pull Requests CI, hosting CD, and code generation.

Features

• Enhanced and consistent formatting for markdown files in GFM format, focused on readability.
• Auto generation of code block content based on mdox-exec directives (see #code-generation). Useful for:
  • Generating help output from CLI --help
  • Generating example YAML from Go configuration struct (+comments)
• Robust and fast relative and remote link checking. (see #link-validation-configuration)
• Website integration:
  • "Localizing" links to relative docs if specified (useful for multi-domain websites or multi-version doc). (see #link-localization)
    • This allows smooth integration with static document websites like Docusaurus or hugo based themes!
  • Flexible pre-processing allowing easy to use GitHub experience as well as website. (see #transform-usage)
• Allows profiling(using fgprof) and exports metrics(saves to file in OpenMetrics format) for easy debugging

Usage

Formatting and Link Checking

Just run mdox fmt and pass markdown files (or glob matching those).

For example, this README is formatted by the CI on every PR using mdox fmt -l *.md command.

usage: mdox fmt [<flags>] <files>...

Formats in-place given markdown files uniformly following GFM (GitHub Flavored
Markdown: https://github.github.com/gfm/). Example: mdox fmt *.md


Flags:
  -h, --[no-]help              Show context-sensitive help (also try --help-long
                               and --help-man).
      --[no-]version           Show application version.
      --log.level=info         Log filtering level.
      --log.format=clilog      Log format to use.
      --profiles.path=PROFILES.PATH  
                               Path to directory where CPU and heap profiles
                               will be saved; If empty, no profiling will be
                               enabled.
      --metrics.path=METRICS.PATH  
                               Path to directory where metrics are saved in
                               OpenMetrics format; If empty, no metrics will be
                               saved.
      --[no-]check             If true, fmt will not modify the given files,
                               instead it will fail if files needs formatting
      --[no-]soft-wraps        If true, fmt will preserve soft line breaks for
                               given files
      --[no-]code-fmt          Reformat code snippets
      --[no-]code.disable-directives  
                               If false, fmt will parse custom fenced
                               code directives prefixed with 'mdox-gen' to
                               autogenerate code snippets. For example:
                               
                                 ```<lang> mdox-exec="<executable + arguments>"
                               
                               This directive runs executable with arguments
                               and put its stderr and stdout output inside code
                               block content, replacing existing one.
      --anchor-dir=ANCHOR-DIR  Anchor directory for all transformers. PWD is
                               used if flag is not specified.
      --links.localize.address-regex=LINKS.LOCALIZE.ADDRESS-REGEX  
                               If specified, all HTTP(s) links that target a
                               domain and path matching given regexp will be
                               transformed to relative to anchor dir path (if
                               exists). Absolute path links will be converted to
                               relative links to anchor dir as well.
  -l, --[no-]links.validate    If true, all links will be validated
      --links.validate.config-file=<file-path>  
                               Path to YAML file for skipping
                               link check, with spec defined in
                               github.com/bwplotka/mdox/pkg/linktransformer.ValidatorConfig
      --links.validate.config=<content>  
                               Alternative to 'links.validate.config-file'
                               flag (mutually exclusive). Content of YAML file
                               for skipping link check, with spec defined in
                               github.com/bwplotka/mdox/pkg/linktransformer.ValidatorConfig
      --[no-]cache.clear       If true, entire cache database will be dropped
                               and rebuilt when mdox is run. Useful in case
                               cache needs to be cleared immediately from GitHub
                               Actions or other CI runner cache.

Args:
  <files>  Markdown file(s) to process.

Code Generation

It's not uncommon that documentation is explaining code or configuration snippets. One of the challenges of such documentation is keeping it up to date. This is where mdox code block directives comes handy! To ensure mdox will auto update code snippet add mdox-exec="<whatever command you want take output from>" after language directive on code block.

For example this Readme contains mdox --help which is has to be auto generated on every PR:

```bash mdox-exec="mdox fmt --help"
...

This also enables auto updating snippets of code in code blocks using tools like sed. For example, below code block directive will auto update and insert lines 3 to 6 from main.go into code block.

```go mdox-exec="sed -n '3,6p' main.go"
...

Some commands might have non-zero exit codes. mdox will fail commands in such cases(otherwise errors might get formatted into markdown) but the expected exit code can also be passed as a code block directive! For example, below code block executes go --help which has 2 as its exit code,

```text mdox-exec="go --help" mdox-expect-exit-code=2
...

You can disable this feature by specifying --code.disable-directives

Link Validation Configuration

By default, mdox checks all links (both relative and remote) in passed markdown files! However, in some cases, link checks might fail even when links are working, such as with rate limiting, Cloudflare protections, or something else(like localhost links in docs).

This might lead to failed CI checks which aren't desirable. So, you can use the links.validate.config-file flag to pass in YAML configuration file for selective link checking using special validators and link regexes.

For example,

version: 1
timeout: '1m'
parallelism: 100
host_max_conns: 2
random_delay: '1s'

validators:
  - regex: '(^http[s]?:\/\/)(www\.)?(github\.com\/)bwplotka\/mdox(\/pull\/|\/issues\/)'
    type: 'githubPullsIssues'

  - regex: 'localhost'
    type: 'ignore'

  - regex: 'thanos\.io'
    type: 'roundtrip'

As seen above, mdox supports validate configuration supports a few parameters and passing an array of link validators with types and regexes. The supported configuration parameters are:

• timeout: The HTTP client's timeout. Defaults to "10s".
• parallelism: The maximum amount of concurrent HTTP requests. Defaults to 100.
• host_max_conns: The maximum amount of HTTP connections open per host. Defaults to 2.
• random_delay: A random delay between 0 and this value is added between requests. It takes values like "500ms", "1s", "1m", or "1m30s". Defaults to no delay.

There are three types of validators:

• ignore: This type of validator makes sure that mdox does not check links with provided regex. This is the most common use case.
• githubPullsIssues: This is a smart validator which only accepts a specific type of regex of the form (^http[s]?:\/\/)(www\.)?(github\.com\/){ORG}\/{REPO}(\/pull\/|\/issues\/). It performs smart validation on GitHub PR and issues links, by fetching GitHub API to get the latest pull/issue number and matching regex. This makes sure that mdox doesn't get rate limited by GitHub, even when checking a large number of GitHub links(which is pretty common in documentation)!
• roundtrip: All links are checked with the roundtrip validator by default(no need for including into config explicitly) which means that each link is visited and fails if http status code is not 200(even after retries).

Relative link checking is not affected by this configuration, as it is expected that such links will work.

YAML can be passed in directly as well using links.validate.config flag! For more details go.dev reference or Go struct.

Link localization

It is expected for documentation to contain remote links to the project website. However, in such cases, it creates problems for multi-version docs or multi-domain websites (links would need to be updated for each version which is cumbersome). Also, it would not be navigatable locally or through GitHub (would always redirect to the website) and requires additional link checking.

This is where the links.localize.address-regex flag comes in handy!

It ensures that all HTTP(s) links that target a domain and path matching given regex will be transformed by mdox to relative links which are relative to anchor dir path (if exists). Also, all absolute path links will be converted to relative links to anchor dir as well.

So passing in regex such as --links.localize.address-regex="https:\/\/example\.\/.* will allow mdox to transform links like https://example.com/getting-started.md/ to simply getting-started.md.

Transformation

mdox allows various types of markdown file transformation which are useful for website pre-processing and is often required when using static site generators like Hugo. It helps in generating front/backmatter, renaming, and moving files, and converts links to work on websites.

Just run mdox transform --config-file=.mdox.yaml and pass in YAML configuration.

usage: mdox transform [<flags>]

Transform markdown files in various ways. For example pre-process markdown files
to allow it for use for popular static HTML websites based on markdown source
code and front matter options.


Flags:
  -h, --[no-]help                Show context-sensitive help (also try
                                 --help-long and --help-man).
      --[no-]version             Show application version.
      --log.level=info           Log filtering level.
      --log.format=clilog        Log format to use.
      --profiles.path=PROFILES.PATH  
                                 Path to directory where CPU and heap profiles
                                 will be saved; If empty, no profiling will be
                                 enabled.
      --metrics.path=METRICS.PATH  
                                 Path to directory where metrics are saved in
                                 OpenMetrics format; If empty, no metrics will
                                 be saved.
      --config-file=<file-path>  Path to Path to the YAML
                                 file with spec defined in
                                 github.com/bwplotka/mdox/pkg/transform.Config
      --config=<content>         Alternative to 'config-file' flag
                                 (mutually exclusive). Content of Path
                                 to the YAML file with spec defined in
                                 github.com/bwplotka/mdox/pkg/transform.Config

For example,

version: 1

inputDir: "docs"
outputDir: "website/docs-pre-processed/tip"
extraInputGlobs:
  - "CHANGELOG.md"
  - "static"

gitIgnored: true
localLinksStyle:
  hugo:
    indexFileName: "_index.md"

transformations:

  - glob: "../CHANGELOG.md"
    path: /thanos/CHANGELOG.md
    popHeader: true
    frontMatter:
      template: |
        title: "{{ .Origin.FirstHeader }}"
        type: docs
        lastmod: "{{ .Origin.LastMod }}"
    backMatter:
      template: |
        Found a typo, inconsistency or missing information in our docs?
        Help us to improve [Thanos](https://thanos.io) documentation by proposing a fix [on GitHub here](https://github.com/thanos-io/thanos/edit/main/{{ .Origin.Filename }}) :heart:

  - glob: "getting-started.md"
    path: /thanos/getting-started.md
    frontMatter:
      template: |
        type: docs
        title: "{{ .Origin.FirstHeader }}"
        lastmod: "{{ .Origin.LastMod }}"
        slug: "{{ .Target.FileName }}"

  - glob: "../static/**"
    path: /favicons/**

As seen above,

• inputDir: It's a relative (to PWD) path that assumes input directory for markdown files and assets.

• outputDir: It's a relative (to PWD) output directory where you can expect all files to land in. Typically that can be a content dir which Hugo uses as an input.

• extraInputGlobs: It allows you to bring files from outside of inputDir.

• linkPrefixForNonMarkdownResources: It specifies the link to be glued onto relative links which don't point to markdown or image files.

• gitIgnored: It specifies whether outputDir should be git-ignored.

• localLinksStyle: It sets the linking style to be applied. If empty, mdox assumes default style.

• transformation: Array of transformations to apply for files in inputDir and extraInputGlobs. This consists of,

  • glob: It is matched against the relative path of the file in the inputDir using https://github.com/gobwas/glob.
  • path: It is an optional different path for the file to be moved into. If not specified, the file will be moved to the exact same position as it is in inputDir.
  • popHeader: If set to true, it pops the first header of md file. True by default for files in the root of inputDir
  • frontMatter: Optional template for constructing frontmatter of markdown file.
  • backMatter: Optional template for constructing backmatter of markdown file(content appended to end like edit links)

YAML can be passed in directly as well using --config flag! For more details go.dev reference or Go struct.

Installing

Requirements to build this tool:

• Go 1.19+
• Linux or macOS

go install github.com/bwplotka/mdox@latest

or via bingo if want to pin it:

bingo get -u github.com/bwplotka/mdox

Production Usage

• Thanos
• Prometheus Operator
• kube-prometheus
• Observatorium
• RedHat Observability Group Handbook
• Bingo
• effiecientgo/tools
• effiecientgo/e2e

Contributing

Any contributions are welcome! Just use GitHub Issues and Pull Requests as usual. We follow Thanos Go coding style guide.

Have questions or feedback? Join our slack channel!

Initial Author

@bwplotka

Note: This project was a part of GSoC'21 (mentor: @bwplotka, mentee: @saswatamcode).