Highlight rustdoc

[zhaouv]

Do Rustaceans want these to be integrated into this extension?
If the answer is 'yes,' I will send a pull request.

[filipwiech]

IMHO it would be a very welcome feature, maybe it could be configurable so that users can choose the preferred experience. I assume that it's providing semantic tokens for the relevant parts of the documentation comments. 🙂

One idea though: perhaps the generated tokens shouldn't be exactly the same as the ones for the normal code, but ought to signify that they are coming from the rustdoc (maybe with a relevant prefix or suffix in their name). This way the theme could decide to show the highlighted parts with different colors (for example slightly muted ones) so that they could be distinguished more easily on the first glance from the actual implementation. 👍

[Veykril]

It will depend on the implementation, especially regarding the syntax highlighting of other languages inside the doc comment. We don't really wanna ship anything unrelated to rust in that regard.

[lnicola]

Also, I think we currently highlight in the second screenshot, which matches rustdoc?

[Veykril]

No we used to, but that implementation caused a bunch of problems so we removed it again.

[lnicola]

What's this, then?

[Veykril]

Oh, you mean the embedded rust code. Yes that we do. I thought you meant the markdown parts.

[zhaouv]

The current implementation (there is still room for adjustments):
package.json>contributes>grammars

            {
                "injectTo": [
                    "source.rust"
                ],
                "scopeName": "comment.markdown-cell-inject.double-slash-exclamation.LRSW",
                "path": "./syntaxes/double-slash-exclamation.json",
                "embeddedLanguages": {
                    "meta.embedded.block.everywhere.md": "markdown"
                }
            },
            {
                "injectTo": [
                    "source.rust"
                ],
                "scopeName": "comment.markdown-cell-inject.triple-slash.LRSW",
                "path": "./syntaxes/triple-slash.json",
                "embeddedLanguages": {
                    "meta.embedded.block.everywhere.md": "markdown"
                }
            },
            {
                "injectTo": [
                    "source.rust"
                ],
                "scopeName": "comment.markdown-cell-inject.slash-start-exclamation.BR",
                "path": "./syntaxes/slash-start-exclamation.json",
                "embeddedLanguages": {
                    "meta.embedded.block.everywhere.md": "markdown"
                }
            }

triple-slash.json:

{
	"injectionSelector": "L:source.rust -string -comment -meta.embedded.block.everywhere.md",
	"patterns": [
		{
			"include": "#comment-markdown-cell-inject-triple-slash-LRSW"
		}
	],
	"repository": {
		"comment-markdown-cell-inject-triple-slash-LRSW": {
			"begin": "(^|\\G)\\s*(///) ?",
			"captures": {
				"2": {
					"name": "comment.punctuation.definition.quote_code.triple-slash.LRSW"
				}
			},
			"name": "comment.quote_code.triple-slash.LRSW",
			"contentName": "meta.embedded.block.everywhere.md",
			"patterns": [
				{
					"include": "text.html.markdown"
				}
			],
			"while": "(^|\\G)\\s*(///) ?"
		}
	},
    "scopeName": "comment.markdown-cell-inject.triple-slash.LRSW"
}

slash-start-exclamation.json and double-slash-exclamation is similar
and with settings (user put it into settings.json themselves)

    "editor.tokenColorCustomizations": {
        "[Default Dark+]": {
            "textMateRules": [
                {
                    "scope": "meta.embedded.block.everywhere.md markup.heading, meta.embedded.block.everywhere.md markup.bold",
                    "settings": {
                        "foreground": "#61aa71",
                    }
                },
                {
                    "scope": "meta.embedded.block.everywhere.md punctuation.definition.list.begin.markdown, meta.embedded.block.everywhere.md entity.name.tag",
                    "settings": {
                        "foreground": "#599aa5",
                    }
                },
                {
                    "scope": "meta.embedded.block.everywhere.md entity.other.attribute-name",
                    "settings": {
                        "foreground": "#98bdc4",
                    }
                },
                {
                    "scope": "meta.embedded.block.everywhere.md markup.inline.raw, meta.embedded.block.everywhere.md string",
                    "settings": {
                        "foreground": "#ceca8b",
                    }
                },
                {
                    "scope": "meta.embedded.block.everywhere.md, meta.embedded.block.everywhere.md meta.embedded",
                    "settings": {
                        "foreground": "#9abb87",
                    }
                },
            ]
        }
    },

[zhaouv]

This is a pure static TextMate rule implementation that utilizes a small trick to consume the /// token, enabling the built-in syntax analysis of VCS to correctly highlight Markdown and its fenced code blocks. It still works perfectly without any additional overhead for Rust files without Cargo.toml. Any manually written TypeScript/JavaScript/Rust doesn't require any additional setup.

[Veykril]

We had a textmate injection for this before and that caused a bunch of problems, will need to dig the PRs for that out later to see what the problem was.

[lnicola]

#15116

[zhaouv]

I have gathered the previous several issues and the syntax removed in the previous two pull requests. The problem with them is that essential syntax was not excluded.

-string -comment

#15116
#15102
#15111
#15114
#15091

rules 15116

{
    "$schema": "https://raw.githubusercontent.com/martinring/tmlanguage/master/tmlanguage.json",
    "scopeName": "source.rustdoc.markdown.injection",
    "injectionSelector": "L:source.rust",
    "patterns": [
        {
            "include": "#doc-comment-line"
        }
    ],
    "repository": {
        "doc-comment-line": {
            "name": "comment.line.documentation.rust",
            "begin": "^\\s*//(/|!)",
            "end": "$",
            "contentName": "meta.embedded.block.markdown",
            "patterns": [
                {
                    "include": "text.html.markdown"
                }
            ]
        }
    }
}

rules 15102

{
    "$schema": "https://raw.githubusercontent.com/martinring/tmlanguage/master/tmlanguage.json",
    "scopeName": "source.rustdoc.markdown.injection",
    "injectionSelector": "L:source.rust",
    "patterns": [
        {
            "include": "#doc-comment-line"
        }
    ],
    "repository": {
        "doc-comment-line": {
            "name": "comment.line.documentation.rust",
            "begin": "^\\s*//(/|!)",
            "end": "$",
            "contentName": "meta.embedded.block.markdown",
            "patterns": [
                {
                    "include": "text.html.markdown"
                }
            ]
        }
    }
}

break 15111

/// bnuuy!
pub fn cool(){
    assert_eq!(1+1, 2);
}

/// not bnuuy
enum Ponies {
    Izzy,
    Sunny,
    Hitch
}

/// baz
struct Foo {
    bar: String
}

/// hello
enum E {
    A,
    B,
    C
}

/// Ping pong
#[poise::command(slash_command, ephemeral)]
pub async fn ping(ctx: CommandContext<'_>) ->Result<()> {
    ctx.say(format!(
        "Pong :3 ({}ms)",
        (ctx.created_at().time() - Utc::now().time()).num_milliseconds()
    ))
    .await?; 
    
    Ok(()) 
}

break 15114

/// Doc comments seem to be rendered in normal color,rather than comment color
/// ```
/// Comment code blocks similarly aren't colored or formatted differently
/// ```
/// Weirdly,VS Code's parentheses matching kicks in, however: ((())
fn main(){
    println!("Hello world!"); 
}

break 15091

/*********************/
let _foo = "/**";


/* ////////////////////////////

//////////////////////////// */

let _foo = "*/";

/*
/// */

let _foo = "/** */";

Let's take a look at some classic examples of syntax injection.

https://github.com/mjbvz/vscode-comment-tagged-templates

"injectionSelector": "L:source.js -comment -(string - meta.embedded), L:source.jsx -comment -(string - meta.embedded), L:source.js.jsx -comment -(string - meta.embedded), L:source.ts -comment -(string - meta.embedded), L:source.tsx -comment -(string - meta.embedded)",

https://github.com/mjbvz/vscode-jsdoc-markdown-highlighting

"injectionSelector": "L:comment.block.documentation",

(mjbvz: This guy, who works for vscode, is a real expert on this.)

First, you should choose to inject into the smallest possible scope.
When you need to include a larger range, be sure to exclude strings and comments.
And you should add meta.embedded tags to the injected content, as a standard practice.

As for relevant information, it's actually TextMate's documentation. VSCode adopted this design, so no separate references are provided.

https://www.apeth.com/nonblog/stories/textmatebundle.html
https://macromates.com/manual/en/all_pages
https://macromates.com/manual/en/language_grammars#naming-conventions

Final result:

[zhaouv]

And next step is adjust the behavior of fenced code. There are two ways to do it.

• reinjection to embedded markdown
• eat the fenced code token before sent to markdown

The more elegant way is to create a language called rustdoc.

{
	"name": "rustdoc",
	"patterns": [
		{
			"include": "#fenced_code_block"
		},
		{
			"include": "#markdown"
		}
	],
	"scopeName": "text.html.markdown.rustdoc",
	"repository": {
		"markdown":{
			"patterns": [
				{
					"include": "text.html.markdown"
				}
			]
		},
		"fenced_code_block":{
			"patterns": [
                {
                    "include": "#fenced_code_block_rust"
                },
                {
                    "include": "#fenced_code_block_unknown"
                }
			]
		},
		"fenced_code_block_rust": {
            "begin": "(^|\\G)(\\s*)(`{3,}|~{3,})\\s*(?i:(rust|not run|not_run)?((\\s+|:|,|\\{|\\?)[^`~]*)?$)",
            "name": "markup.fenced_code.block.markdown",
            "end": "(^|\\G)(\\2|\\s{0,3})(\\3)\\s*$",
            "beginCaptures": {
                "3": {
                    "name": "punctuation.definition.markdown"
                },
                "4": {
                    "name": "fenced_code.block.language.markdown"
                },
                "5": {
                    "name": "fenced_code.block.language.attributes.markdown"
                }
            },
            "endCaptures": {
                "3": {
                    "name": "punctuation.definition.markdown"
                }
            },
            "patterns": [
                {
                    "begin": "(^|\\G)(\\s*)(.*)",
                    "while": "(^|\\G)(?!\\s*([`~]{3,})\\s*$)",
                    "contentName": "meta.embedded.block.rust",
                    "patterns": [
                        {
                            "include": "source.rust"
                        }
                    ]
                }
            ]
        },
		"fenced_code_block_unknown": {
            "begin": "(^|\\G)(\\s*)(`{3,}|~{3,})\\s*(?=([^`~]+)?$)",
            "beginCaptures": {
                "3": {
                    "name": "punctuation.definition.markdown"
                },
                "4": {
                    "name": "fenced_code.block.language"
                }
            },
            "end": "(^|\\G)(\\2|\\s{0,3})(\\3)\\s*$",
            "endCaptures": {
                "3": {
                    "name": "punctuation.definition.markdown"
                }
            },
            "name": "markup.fenced_code.block.markdown"
        }
	}
}

And then. only rust|not run|not_run and block without description will highlight as rust, other will be ignored.

[zhaouv]

And also more things can be done:
when you type a Enter at red O, auto fill /// in next line
(by vscode.languages.setLanguageConfiguration('rustdoc', { onEnterRules: [...] })

[Veykril]

Alright, seems like you figured out the problems from the previous attempts. I'm happy for us to try this again in that case!