ENH: markdown: Table of Contents (with_toc_data)

[westurner]

• add a syntax directive for including a Table of Contents (TOC) in markdown documents

redcarpet now supports a with_toc_data (--render-with-toc-data) option for automatically generating a Table of Contents <ul> from markdown setext (=,_ underline) and ATX (# prefix) headings.

• with_toc_data: GitHub style anchors vmg/redcarpet#186 (comment) ("with_toc_data: GitHub style anchors ", Changelog entry)
• TOC links / permalink generation vmg/redcarpet#391 (comment) ("TOC links / permalink generation")
• Duplicate anchor names vmg/redcarpet#307 (appending incrementing numeric values for duplicate headers)
• Ampersand (&) not rendered correctly in Table Of Contents vmg/redcarpet#529 (ampersand & not rendering correctly)

[westurner]

Coming from ReStructuredText (docutils (setext)), auto-generated Table of Contents directives are a really useful feature.

.. ```restructuredtext

.. contents::

.. contents:: Table of Contents 2
   :depth: 10
   :local:

• http://docutils.sourceforge.net/docs/ref/rst/directives.html#table-of-contents
• http://docutils.sourceforge.net/COPYING.html (Public Domain (:+1:))

[tukkek]

I strongly second this: every major markup language for wikis has a table-of-contents feature. The sheer number of people who have created scripts, userscripts and online tools in this thread to help us do it manually goes to show how important such a feature is! Not to mention the almost 300 comments there - mostly "+1" comments as one is quick to notice.

There are several questions on Stack Overflow about this as well, not to mention other sites:

• http://stackoverflow.com/questions/9721944/automatic-toc-in-github-flavoured-markdown?s=1|0.0000
• http://stackoverflow.com/questions/9239588/toc-or-sidebar-in-github-wiki?s=3|0.0000

Sure GitHub offers other markup languages that have this feature but markdown is their default one by choice - there is no excuse for them not to go ahead and deal with this issue once and for all.

[tukkek]

I've just received a reply from GH on this. Spoiler alert: its pretty depressing, despite this issue already having over 40 thumbs up when I wrote to them.

If a GH staff member is reading this please forgive me but doesn't look like they care at all since they couldn't be bothered to even make a comment here or in the linked thread. As you can see from the other thread that is pretty much their default reply every time they're contacted so I'm really having a hard time believing something will be done about this in the next few years.

Thanks GH, up to now I had nothing but respect for this website.

Thanks for reaching out! We don't comment publicly on our roadmap, so I'm not able to say if or when this will happen, but I will definitely add your +1 to the feature on our internal feature request list.

Thanks again for reaching out and for using GitHub!

Regards,
GitHub Support

[tukkek]

EDIT: Now that I think of it there is zero evidence they even bothered to open up the link and see this issue. I've also cleaned up my last comment since it was badly written.

[abalter]

Make sure it's known that there are a zillion people here who want it.

isaacs/github#215

[franslundberg]

+1. Please, implement this crucial feature.

[brabenetz]

+1

[joihelgi]

+1

[steel]

+1

[jlengstorf]

Jekyll has support for a table of contents via Kramdown:

- 
{:toc}

NOTE: There's a space after that hyphen that's required for it to work properly.

It's not particularly flexible, but given that GitHub Pages is running Jekyll, it could be worthwhile to use this style of ToC generation — that way if I'm looking at the source for a GH Pages site, the ToC works on the live site and the Jekyll-generated site without issues.

Personally, I'd favor consistency over flexibility in this case. The lower learning curve for new users and decreased complexity to implement/maintain/support is a plus, too.

[i2000s]

I personally prefer to have a toggle content (toggle content under a title/header line) and/or table of contents switch on the top right corner of the Github markdown preview page. Currently, we have already had blame, history and source buttons on the top menu. Why not add another as/under a default preview options button defined by who created the doc and can be changed by readers?

The technical difficulty might be a burden on the server to generate previous pages case by case for different repos and/or for various readers. But I don't see why this cannot be done by GitHub staff.

@jlengstorf @westurner The problem of having an explicit {:toc} code line on the markdown doc is that the doc might be rendered using another markdown markup engine to generate a Github page or something else, and adding lines like {:toc} might bring in some conflicts or unexpected results on the resulted Github pages or other products.

I have also thought about enabling a global switch on one's github setting page, but the level of granular control of rendering seems too rough if one only wants to generate a table of contents or toggle all headers/titles on a few github pages. In the meantime, a reader can definitely have some level of control over what effect they want to see on markdown preview pages, but should the writer/owner of a doc have the control over the preview effect at the first place?

Just my 2cent. Of course, my 👍 to those who opens this issue and who contribute.

[jlengstorf]

I personally prefer to have a toggle content (toggle content under a title/header line) and/or table of contents switch on the top right corner of the Github markdown preview page.

@i2000s I'm not sure I follow what you're suggesting, but it seems it might be an entirely separate feature request.

The problem of having an explicit {:toc} code line on the markdown doc is that the doc might be rendered using another markdown markup engine

This is fair, but GitHub has already thrown in with Jekyll (and, by proxy, Kramdown) for GitHub Pages. It's the default there. What I'm suggesting is adding cross-channel support so that the GitHub default and the GitHub Pages default versions of Markdown files are consistent.

Anyone using alternative Markdown syntax is already generating their GH Pages files before sending up to GitHub, so this wouldn't break anything, and the lack of support for non-standard Markdown wouldn't change. Unless I'm missing something, this would be a non-breaking change that most people wouldn't notice — people using Jekyll would see improvements in the Markdown rendering on github.com, and that's all.

[kivikakk]

We're now using CommonMark, so feature requests of this sort should go to CommonMark Discussion or our support team if we were going to consider adding it to our own cmark implementation. Thanks!

[westurner]

@kivikakk I wouldn't close this because the feature is not implemented? wontfix? pending CommanMark standardization?

Information that would be helpful in satisfying the clear demand for this feature (158 👍, 9 ❤️):

• A link to the implementation of CommonMark which will need the changes
  • https://github.com/github/cmark/
• An issue in github/cmark for tracking the implementation

And then a post here when it's implemented would be really cool.

[freelance-tech-writer]

Utilizing CommonMark doesn't necessarily mean GitHub can't provide a supplemental tool for this. I'm not sure closing this issue will help resolve it one way or the other.

In the meantime, I created a supplemental JavaScript-based solution for this: table-of-contents-generator. Hope it's helpful to others who find this thread.

[kivikakk]

@westurner @freelance-tech-writer This repository is the markup gem, and the issues are for issues relating to the gem specifically (e.g. it's incorrectly interacting with one of its dependency gems, or a change to pod2html, or a version bump for a dependency).

Seeing as this is a feature request for github.com's Markdown support, it should go to our support team so that it gets to the right avenues, is tracked and prioritised accordingly, etc. I've closed the issue because it just doesn't relate to the repository at all (and indeed, there is no repository for site feature requests; that's why we have a support team to collate them and respond to them).

[westurner]

@kivikakk So, you sent an email to support@github.com? Or one or more of these 159+ people need to send an email to support@github.com requesting a "Markdown Table of Contents" as discussed in #904 ?

[kivikakk]

@westurner It's better to use the form. We are aware of the feature request already (it's been made a few times), but if you want to add your voice, you're welcome to.

[tukkek]

As you'd know if you'd actually read this issue's log, I have contacted GH support already concerning this and nothing came out of it. Telling people to try that avenue is inefficient at best and dubious at worst.

[tukkek]

Issue has almost 200 likes (or over 300 if you look at the post I linked to last year). GitHub's reaction: close it due to technicalities without as much as a follow-up ticket on the appropriate repository. Cool story @kivikakk, outstanding work!

[AndreasHassing]

@tukkek please stop poisoning this issue, it's closed and you have been given a way to formally request this feature.

I just sent a feature request for TOC via the form mentioned in that comment and got a quick response:

Hi Andreas,

Thanks for reaching out to GitHub Support!

I agree, a Table of Content would be great to have. We'll be sure to pass this feature request onto the team. I can't make any promises on this, but your request has definitely been noted!

Have a great day further!

Cheers,
John
GitHub Support

Obviously, this is no indication that Team GitHub will implement it, but here's to hoping 🍻.

[tukkek]

I'm not deliberately trying to poison anything, I'm just amazed that a GH employee can close an issue with hundreds of upvotes without as much as opening a new one in the appropriate forum. If that's not subpar service I don't know what is.

Nice of you to contact customer support, like several others before you. You can see though there's no follow up or accountability there as well, they've just sent you the cookie cutter response they do to everyone who tries and your message goes to the trash folder as fast as it came.

Does GH have an obligation to follow up on any of that or actually provide us with more info? Not at all. Doesn't change the fact they're treating the hundreds of users who have liked, subscribed and participated on this an related issues with something akin to contempt. Really, how much time does it take for GH to officially follow up on this in the Common Mark tracker? Literally a couple of minutes, they just don't care.

[abalter]

I totally agree. What's really rude is that the fact of transitioning to Common Mark as absolutely nothing to do with having a TOC feature. Fine, it's Common Mark now. But we still want a TOC, just in Common Mark.

[abalter]

#1036

[westurner]

TOC Implementations

• (pseudocode)
• Parse the markdown headings: (id=slugify(name), name=name)
  • Find ATX headings: # ## ### #### (# prefix (level n))
  • Find setext headings: =====, ----- (underline (level 1, 2))
• Generate a unique id= for each h1, h2, h3, h4 (So that #deep-links-work)
  • slugify(): Thing 123 and 1234 -> thing-and OR thing-123-and-1234
• Generate a unique id= for each TOC a tag (So that headings can link back to the TOC)
• Replace each instance of the TOC "directive"? with a table of contents:
  • If it's a local TOC, only include the headings nested below the most recent heading
    • :local: (docutils (public domain))
      • http://docutils.sourceforge.net/docs/ref/rst/directives.html#table-of-contents
• Generate markdown and/or HTML:
  • Markdown: before_render(generate_tocs)
  • HTML: ~ Renderer().add_directive(TocDirective); renderer.render(document)

test case

#### Contents
- [h1-0](#h1-0)
  - [h2-0-0](#h2-0-0)
    - [h3-0-0-0](#h3-0-0-0)
  - [Thing 123 and 1234](#thing-123-and-1234)
- [h1-1](#h1-1)
        - [h4-1-0-0-0](#h4-1-0-0-0)

Contents

• h1-0
  • h2-0-0
    • h3-0-0-0
  • Thing 123 and 1234
• h1-1
  - h4-1-0-0-0

jgm/CommonMark

• | Src: https://github.com/jgm/CommonMark/blob/master/tools/make_spec.lua#L57 make_toc()
• | Docs: https://github.com/jgm/CommonMark/wiki/Generic-Directive-Extension-List

# Title
TOC: Contents
# h1-0
## h2-0-0
### h3-0-0-0
## Thing 123 and 1234
# h1-1
#### h4-1-0-0-0

Python-Markdown

• | Src: https://github.com/waylan/Python-Markdown/blob/master/markdown/extensions/toc.py
• | Pypi: https://pypi.python.org/pypi/Markdown
• | Docs: - https://pythonhosted.org/Markdown/extensions/toc.html#syntax

# Title
# Contents
|TOC|
# h1-0
## h2-0-0
### h3-0-0-0
## Thing 123 and 1234
# h1-1
#### h4-1-0-0-0

jonschlinkert/markdown-toc

• Src: https://github.com/jonschlinkert/markdown-toc

# Title
# Contents
<!-- toc -->
# h1-0
## h2-0-0
### h3-0-0-0
## Thing 123 and 1234
# h1-1
#### h4-1-0-0-0

jgm/cmark

• Src: https://github.com/jgm/cmark/

github/cmark

• Src: https://github.com/github/cmark/
• Docs: https://github.github.com/gfm/

commonmark spec

• | Spec: http://spec.commonmark.org/0.27/

Table of Contents syntax

• Brainstorm requirements
• See: the "TOC Implementations" heading at the top of {this} post
  • @github: toc-implementations-288622154 (slug-futurecommentid)
• Discussion: (TODO: create github issue linking back to here; with lots of helpers)
  • "toc" OR "contents"
  • depth
  • local
  • backlinks: whether to link headings back to the TOC
  • class: an extra CSS class to apply to the TOC block(s)

docutils

docutils ``contents`` directive
================================
.. contents:: Contentz
   :depth: 3
   :local:

• https://github.com/docutils-mirror/docutils/blob/master/COPYING.txt (Public Domain ❤️ )
• | Src: http://svn.code.sf.net/p/docutils/code/trunk/docutils/docutils/nodes.py
  • class pending

    • But the "contents" directive can't do its work until the entire document has been parsed and possibly transformed to some extent. So the directive code leaves a placeholder behind that will trigger the second phase of its processing [...]

• | Src: https://github.com/docutils-mirror/docutils/blob/master/docutils/parsers/rst/directives/parts.py
  • class Contents (docutils.parsers.rst.directive.parts:Contents)
• | Src: https://github.com/docutils-mirror/docutils/blob/master/docutils/transforms/parts.py
  • class Contents (docutils.transforms.parts:Contents)

[abalter]

#1037

[westurner]

CommonMark:

• "Feature request: automatically generated ids for headers" https://talk.commonmark.org/t/feature-request-automatically-generated-ids-for-headers/115

• "HTML table-of-contents renderer"
  HTML table-of-contents renderer commonmark/cmark#78

  GitHub generates unique IDs and anchors for CommonMark headings. How do they do it?

[westurner]

Build-less tables of contents are possible with .mediawiki (by default) and .rst/.rest documents (with a .. contents:: directive); since launch.

(I've emailed a feature request for TOC on Markdown documents)

Are there other great tools for handling this need with e.g. an entry in a Makefile and an offline build step?