🐛 Rewrite heading processing

[Lemmingh]

Summary

• Fix "Create Table of Contents" fails when the file contains tab #853, Fix Table of contents creation fails when some lines begin with tab #603: Perform leading tab-space conversion required by the CommonMark Spec in getAllRootHeading(). (Rewrite the whole function.)
• Fix Markdown Intellisense with multiple identical labels shows non-unique anchors #831: Expose proper slug in IHeading.
• Fix Table of contents not recognized if indented. #830: Rewrite detectTocRanges() on top of markdown-it to eliminate various mysterious bugs.
• Fix GitLab TOC support #469: Align with GitLab 13.8.
• Correct REGEX_FENCED_CODE_BLOCK, isInFencedCodeBlock().
• Correct/Adapt related functions.
• Correct slugify().
• Correct GitHub slugify function.
• Correct Gitea (blackfriday) slugify function.
• Rewrite many fsPath/object-property-based dictionaries as Uri/Map-based, to mitigate various mysterious bugs.

[yzhang-gh]

in progess

[yzhang-gh]

Thanks for the great effort! ❤

One last question: should we force all the users to adopt the [heading](<link>)-style TOC? It will affect hundreds of thousands of people (creating a change to their git history). I would suggest not adding this when/where it is not necessary.

[yzhang-gh]

The "root" in getAllRootHeading refers to the position in the document tree.

Many block structures accept block structures as their content.

You can create some deeply nested things like:

# Heading A

> + # Heading B ###

But we never care about those non-root headings, right 😳? It is true that they are "headings" tags in terms of the syntax tree, but we won't say they are "headings" in terms of a document/article.

Also, it looks like we didn't make use of the syntax tree at all in this function. So I guess it would be better to move this "root" thing from the name to a piece of comment.

[Lemmingh]

But we never care about those non-root headings

It's just our current architecture cannot handle them. (the same reason as stated above)

We indeed have to consider them!

They play an important role in generating slugs. And implementations differ.

---

To solve all the problems, I'm privately prototyping version 4, where almost all features are powered by two language servers (one for symbol info and theming, the other for actions), and a plugin system is first class member.

I now get stuck constructing AST from markdown-it Tokens.

[yzhang-gh]

But we never care about those non-root headings

It's just our current architecture cannot handle them. (the same reason as stated above)

We indeed have to consider them!

They play an important role in generating slugs. And implementations differ.

Probably I don't have enough context (about your plan). I was just saying it (assuming heading is rootHeading) is enough for the current feature set (TOC, section numbering). A so-called "heading" in a fenced code block is not a valid heading in our features (not in the TOC, not involved in the section numbering, and not present in the reference links).

Do you mean there are other cases where a heading nested in some blocks should be considered in the current feature set?
Or you plan to use the AST in the getAllRootHeading function soon/later?

[yzhang-gh]

Or you plan to use the AST in the getAllRootHeading function soon/later?

Not sure whether we should do it in this PR, but #880 is an example supporting this.

[Lemmingh]

I don't have enough context (about your plan).

I'll email you more details next month.

export type ProviderResult<T> = T | Thenable<T>;

export interface IHeadingInfoProvider<T extends IHeadingInfo = IHeadingInfo> {

    readonly provideHeadingInfo: (document: vscode.TextDocument, options: IHeadingInfoOption, token: vscode.CancellationToken) => ProviderResult<T[]>;

    readonly resolveHeadingInfo: (items: T[], token: vscode.CancellationToken) => ProviderResult<Required<T>[]>;
}

export interface IHeadingInfoOption {
    documentMode: SlugifyMode;
    projectLevelOmittedHeading: [markdownSpec.MarkdownHeadingLevel, string][] | undefined;
    respectMagicCommentOmit: boolean;
}

export interface IHeadingInfo {

    location: vscode.Location;
    level: markdownSpec.MarkdownHeadingLevel;
    rawContent: string;

    canInToc?: boolean;
    slug?: string;
    visibleText?: string;
}

---

Do you mean there are other cases where a heading nested in some blocks should be considered in the current feature set?

Yes. They play an important role in generating slugs. And I tend to regard the existence of getAllRootHeading() as a compromise. It's best to merge getAllTocEntry() and getAllRootHeading() as one method getAllHeading(), but the current architecture makes it too hard.

Render this example:

# Heading A

> + # Heading B ###

# Heading B

It's evident that GitHub, GitLab, VS Code, Azure DevOps ... all assign anchor ID to all headings no matter where they are.

Then, you have to get all headings to calculate slugs correctly.

---

Or you plan to use the AST in the getAllRootHeading function soon/later?

Probably in 2022. NOT now. I cannot make it within this month.

I tried to work directly on token stream. But as you can see in detectTocRanges(), it's frustrating. Then I decided to create something similar to Markdig, however, markdown-it appears to erase lots of information, which is useless for rendering but critical for understanding a document. So, stuck.

[yzhang-gh]

And I tend to regard the existence of getAllRootHeading() as a compromise. It's best to merge getAllTocEntry() and getAllRootHeading() as one method getAllHeading(), but the current architecture makes it too hard.

Makes sense to me 👍

It's evident that GitHub, GitLab, VS Code, Azure DevOps ... all assign anchor ID to all headings no matter where they are.

Interesting. I didn't know it before.

Or you plan to use the AST in the getAllRootHeading function soon/later?

Probably in 2022. NOT now. I cannot make it within this month.

That is fine.

BTW, my next big plan is to implement the "export as PDF" feature and that will bring us to the next major update (v4.0.0), although I am not sure when it will be done 🤣. Just take your (our) time.

[yzhang-gh]

🍻