✨ NEW: Add toclist extension

[chrisjsewell]

By adding "toclist" to myst_enable_extensions (in the sphinx conf.py configuration file),
you will be able to specify sphinx toctree in a Markdown native manner.

toclist are identified by bullet lists that use the + character as the marker,
and each item should be a link to another source file or an external hyperlink:

+ [Link text](subsection.md)
+ [Link text](https://example.com "Example external link")

is equivalent to:

```{toctree}

subsection.md
Example external link <https://example.com>
```

Note that the link text is omitted from the output toctree, and the title of the link is taken from either the link title, if present, or the title of the source file.

You can also specify the maxdepth and numbered options for all toclist in your conf.py:

myst_toclist_maxdepth = 2
myst_toclist_numbered = True

---

Design notes:

• I think the extension could be really useful for offering the possibility to write documentation using only CommonMark syntax; toctree is essentially the only directive/role strictly necessary to create multi-page sphinx sites.
  • At present the extension requires quite minimal code to implement, so should not incur much additional code maintenance burden
• This extension is similar to https://recommonmark.readthedocs.io/en/latest/auto_structify.html#auto-toc-tree
  • Also, to some extent, https://nbsphinx.readthedocs.io/en/0.8.8/subdir/toctree.html
• I chose the + marker since I felt it is the least used of the possible CommonMark bullet markers (-, *, +), so should generally not conflict with "standard" bullet lists
• For specifying entry titles, I use [](page.md "title") instead of [title](page.md). A key reason is because the latter is parsed to markdown tokens, and so you would need to convert it back to plain text, which would be tricky.
• toctree options can only be set globally, using myst_toclist_maxdepth, myst_toclist_numbered. Specifying them per toclist would mean some kind of bespoke "decorator" syntax, which would complicate the implementation , and I feel is not necessary for most use cases.
  • Although caption would be nice to specify, see below...

---

Some way to specify the caption, per toclist might be nice, preferably with minimal overhead on syntax/implementation code overhead.

Possibly the simplest would be allowing an initial item with the caption:

+ `toctree caption`
+ [Link text](subsection.md)
+ [Link text](https://example.com "Example external link")

Note, similar to entry titles, we would want to specify them as inline code, so they are not parsed as markdown.

A slightly more complex implementation would be to use nested lists, also allowing multiple toctrees in a single list:

+ `toctree 1 caption`
   + [Link text](subsection1.md)
   + [Link text](https://example.com "Example external link")
+ `toctree 2 caption`
   + [Link text](subsection2.md "a title")

would be equivalent to:

```{toctree}
:caption: toctree 1 caption

subsection1.md
Example external link <https://example.com>
```

```{toctree}
:caption: toctree 2 caption

a title <subsection2.md>
```

thoughts @choldgraf, @mmcky, etc?

[codecov]

Codecov Report

Merging #485 (32194f2) into master (885651f) will decrease coverage by 0.29%.
The diff coverage is 75.00%.

@@            Coverage Diff             @@
##           master     #485      +/-   ##
==========================================
- Coverage   90.03%   89.74%   -0.30%     
==========================================
  Files          16       16              
  Lines        2058     2097      +39     
==========================================
+ Hits         1853     1882      +29     
- Misses        205      215      +10     

Flag	Coverage Δ
pytests	89.74% <75.00%> (-0.30%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
myst_parser/docutils_.py	79.62% <ø> (ø)
myst_parser/sphinx_renderer.py	90.00% <73.68%> (-3.65%)	⬇️
myst_parser/main.py	88.97% <100.00%> (+0.17%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 885651f...32194f2. Read the comment docs.

[chrisjsewell]

See also documentation: https://myst-parser--485.org.readthedocs.build/en/485/syntax/optional.html#table-of-contents-lists

[choldgraf]

Thanks for the prototype - my main question is: what is the problem that this addition is trying to solve? You mention this would let people write documentation purely with CommonMark - is that something people have been requesting?

I think our main alternative would be to define a directive that behaved similarly, like:

```{toclist}
[Displayed site name](https://example.com)
[Displayed page name](page1.md)
```

You'd lose the "pure commonmark" aspect though.

From a design standpoint:

• I agree that the + is probably the least-used CommonMark token for lists.
• I think it will confuse people that they need to write something like [Link text](https://example.com "Example external link"). It seems like [Example external link](https://example.com) is what they'd expect to be able to do. I suggest removing Link text entirely from examples as it will make it seem like that text does something, when it does not.
• To that extent, if people have to write not-quite-markdown syntax like [Link text](https://example.com "Example external link"), then don't they lose the benefits of "Writing a TOCtree in pure CommonMark" (since the links themselves are not commonmark)

I don't have super strong opinions in general - as long as this is an optional syntax I think it's fine to evolve it over time as people use it.

[chrisjsewell]

Heya,

To that extent, if people have to write not-quite-markdown syntax like Link text, then don't they lose the benefits of "Writing a TOCtree in pure CommonMark" (since the links themselves are not commonmark)

This is CommonMark syntax though. Perhaps you didn't realise the links can have titles? [link text](https://example.com "title") -> link text (if you hover over you'll see the title)

[chrisjsewell]

It seems like Example external link is what they'd expect to be able to do. I suggest removing Link text entirely from examples as it will make it seem like that text does something, when it does not.

As explained in the initial comment, this would parse Example external link as markdown, which you would then have to work back to get the original raw text. this is not the case with titles.
I would note also, that the documentation does already clarify this, unless you think this is not clear:

[chrisjsewell]

ou mention this would let people write documentation purely with CommonMark - is that something people have been requesting?

yes, particularly in the notebook context

(also, as I note in the PR comment, it is similar to functionality recommonmark and nbsphinx already have)

what is the problem that this addition is trying to solve?

Copied from Slack:

To clarify, its not primarily meant to improve "user-friendliness" (although I believe it does), it is meant to improve "render-friendliness":

if you use:

```{toclist}
...
```

any non-myst renderer will just show literal text, whereas

- [text](page.md)

will "correctly" show a link to the page.md document.

this is the "problem" being solved, not that users find it hard to use the toctree directive

As an example, go to https://github.com/executablebooks/MyST-Parser/blob/32194f2d45f351419b9e4dc7975c9a9d0aba6946/docs/syntax/optional.md#table-of-contents-lists and if you click on the link, it will correctly take you to https://github.com/executablebooks/MyST-Parser/blob/32194f2d45f351419b9e4dc7975c9a9d0aba6946/docs/syntax/subsection.md.

I think our main alternative would be to define a directive that behaved similarly, like:

this is why your suggestion does not solve the issue: you're still using it within a code fence, so it's still just going to be rendered as literal text

[mmcky]

Sorry a bit late to this party.

I'm usually in favour of running these sort of extensions through a directive -- I typically think of core markdown as simple markup syntax without any large state transformations (other than visual or direct translations). For example, going from [text](link) is a simple state transformation but a context specific interpretation of a list could be confusing.

Not suggesting it would ever be, but I think it is important this type of extension would not be enabled by default (so it would be purely opt in syntax)

[chrisjsewell]

it is important this type of extension would not be enabled by default

just to clarify, no extensions are enabled by default, I will even be removing the current dollarmath on by default behaviour

[chrisjsewell]

cheers for the feedback, its always welcome 👍

[arwedus]

@chrisjsewell : I like the concept. How could we include :caption: in some way?

Maybe

Caption:

+ [Link text](subsection.md)
+ [Link text](https://example.com "Example external link")

[choldgraf]

A few more thoughts below:

This is CommonMark syntax though

Ah gotcha - I did not realize that, you can disregard my comments about this not being CM compliant.

I use instead of title. A key reason is because the latter is parsed to markdown tokens, and so you would need to convert it back to plain text, which would be tricky.

I agree that we can clarify things with documentation, though I worry this will still be unintuitive to folks so we'd pay a UX penalty even if it were documented.

Since the main reason for using the "title" syntax is implementation complexity, is there a low-hanging fruit we could shoot for to compromise? E.g., what if you enforced the use of a literal as the title? [`my section title`](page.md), would that simplify the parsing?

captions

The syntax you proposed here seems reasonable to me:

+ `toctree 1 caption`
   + [Link text](subsection1.md)
   + [Link text](https://example.com "Example external link")
+ `toctree 2 caption`
   + [Link text](subsection2.md "a title")

[arwedus]

@chrisjsewell: I just came across a use case for this extension again.
In this instance, we want to serve both the casual reader (VS code preview or bitbucket preview) as well as the sphinx documentation generator, and we ended up using "hidden toctrees" and writing all links twice. I'd be happy to see this extension in a future release of myst-parser 👍🏼

Just one point: Some markdown linters complain if you use the same bullet element on different levels, so maybe do something like:

+ `toctree 1 caption`
   - [Link text](subsection1.md)
   - [Link text](https://example.com "Example external link")
+ `toctree 2 caption`
   - [Link text](subsection2.md "a title")

But it's also okay to keep everything "+"