rustc_errors: Add (heuristic) Syntax Highlighting for rustc --explain

[JayanAXHF]

This PR adds a feature that enables rustc --explain <error> to have syntax highlighted code blocks. Due to performance, size and complexity constraints, the highlighter is very heuristc, relying on conventions for capitalizations and such to infer what an identifier represents. The details for the implementation are specified below.

Changes

1. Change term::entrypoint to term::entrypoint_with_formatter, which takes an optional third argument, which is a function pointer to a formatter. (compiler/rustc_errors/src/markdown/mod.rs)
2. Change MdStream::write_anstream_buf to be a wrapper around a new function, MdStream::write_anstream_buf_with_formatter, which takes a function pointer to a formatter. (compiler/rustc_errors/src/markdown/mod.rs)
3. Change compiler/rustc_driver_impl/src/lib.rs to call MdStream::write_anstream_buf_with_formatter instead of MdStream::write_anstream_buf.
4. Add a compiler/rustc_driver_impl/src/highlighter.rs file, which contains the actual syntax highlighter.

Implementation Details

1. The highlighter starts from the highlight function defined in compiler/rustc_driver_impl/src/highlighter.rs. It creates a new instance of the Highlighter struct, and calls its highlight_rustc_lexer function to start highlighting.
2. The highlight_rustc_lexer function uses rustc_lexer to lex the code into Tokens. rustc_lexer was chosen since it preserves the newlines after scanning.
3. Based on the kind of token (TokenKind), we color the corresponding lexeme.

Highlighter Implementation

Identifiers

1. All identifiers that match a (non-exhaustive and minimal) list of keywords are coloured magenta.
2. An identifier that begins with a capital letter is assumed as a type. There is no distinction between a Trait and a type, since that would involve name resolution, and the parts of rustc that perform name resolution on code do not preserve the original formatting. (An attempt to use rustc_parse's lexer and TokenStream was made, which was then printed with the pretty printer, but failed to preserve the formatting and was generally more complex to work with)
3. An identifier that is immediately followed by a parenthesis is recognized as a function identifier, and coloured blue.

Literals

5. A String literal (or its corresponding Raw, C and Byte versions) is colored green.
6. All other literals are colored bright red (orange-esque)

Everything Else

Everything else is colored bright white and dimmed, to create a grayish colour.

---

Demo

Command: rustc --explain E0520

---

This description was not generated by an LLM (:p)

cc: @bjorn3

[rustbot]

These commits modify the Cargo.lock file. Unintentional changes to Cargo.lock can be introduced when switching branches and rebasing PRs.

If this was unintentional then you should revert the changes before this PR is merged.
Otherwise, you can ignore this comment.

[rustbot]

r? @fee1-dead

rustbot has assigned @fee1-dead.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

[JayanAXHF]

PS (Link to Zulip Topic) : #t-compiler/diagnostics > Colored output for `rustc --explain`

[Kivooeo]

Is this change requires MCP?

[JayanAXHF]

Is this change requires MCP?

Could you clarify what you mean by MCP? I'm very new to rustc (this is my first contribution)

[Kivooeo]

A process where you propose a change and other members brainstorm it whether it worth adding or not

[Kivooeo]

https://forge.rust-lang.org/compiler/proposals-and-stabilization.html

[JayanAXHF]

A process where you propose a change and other members brainstorm it whether it worth adding or not

Oh thanks. I don't think that this qualifies as a major change. It just adds some very simple code that basically colours tokens by matching on its kind and lexeme. IMO it falls in the light-weight proposal category, as it doesn't break anything (if it errors, the plain block is printed)

[tgross35]

I don't think this needs an MCP, it's a reasonably local change that could be backed out at any time.

[Kivooeo]

r? me

[tgross35]

Fyi this will need a test somehow, a UI test is probably fine

[JayanAXHF]

Fyi this will need a test somehow, a UI test is probably fine

The test could probably me giving it a string of code and comparing its colored output to what was expected.

[JayanAXHF]

The job aarch64-gnu-llvm-20-2 failed! Check out the build log: (web) (plain enhanced) (plain)

The existing tests need to be updated

[tgross35]

IIRC that test can be updated with RUSTC_BLESS=1

[JayanAXHF]

IIRC that test can be updated with RUSTC_BLESS=1

Could you elaborate please (I'm sorry I'm new to the codebase)

[tgross35]

Sorry, you don't need to set that env - just run --bless when you run the tests locally (./x t <relevant path>).

rust/compiler/rustc_errors/src/markdown/tests/term.rs

Line 65 in a3f2d5a

let bless = std::env::var_os("RUSTC_BLESS").is_some_and(|v| v != "0");

. Just means to auto update test that check output and can be annoying to update manually, so you can look at the diff (of course, doesn't mean it is correct!).

Not actually sure why that test failed, the code formatter isn't hooked there.

[JayanAXHF]

I have fixed the failing test.

[JayanAXHF]

Fyi this will need a test somehow, a UI test is probably fine

Where would I create the test?

[Kivooeo]

Can we consider colouring all primitives types as well? The same way it was made for keywords

https://doc.rust-lang.org/rust-by-example/primitives.html

[Kivooeo]

Can write_anstream_buf now be deleted? I don't see any reasons to keep it for now, since it just calls write_anstream_buf_with_formatter, why not call it directly

Same for write_stream

[JayanAXHF]

Can we consider colouring all primitives types as well? The same way it was made for keywords

https://doc.rust-lang.org/rust-by-example/primitives.html

Sure!

[JayanAXHF]

Same for write_stream

This would require having to change stuff in a few places (in the same file i believe)

Can write_anstream_buf now be deleted? I don't see any reasons to keep it for now, since it just calls write_anstream_buf_with_formatter, why not call it directly

This can done, since iirc write_anstream_buf is not called anywhere except the current test (which can be updated)

[Kivooeo]

This would require having to change stuff in a few places

Feel free to do this changes

[JayanAXHF]

but adding --color=always

What will this look like in the .stdout file?

I believe it should just show the escape codes, similar to the output.stdout file that's in this PR now (though code highlighting isn't hooked there of course)

Would it be fine if the test just focused on the highlighter?

[JayanAXHF]

@Kivooeo (cc: @tgross35) I have figured out (i think) why --color always doesn't always print out in color. Apparently, within rustc_driver_impl,

        if io::stdout().is_terminal() {
            show_md_content_with_pager(&text, color);
        } else {
            safe_print!("{text}");
        }

prevents printing the output with color if stdout isn't a terminal. This is why --bless didn't output a .stdout file with ANSI escapes

[JayanAXHF]

Following my previous message, I have added the tests for the highlighter. However, I did have to change rustc_driver_impl's handle_explain function to be indifferent to stdout when --color always is passed. I hope that is alright

[Kivooeo]

ok, looks good now

thanks for this

@bors r+

[rust-bors]

📌 Commit 19b04f2 has been approved by Kivooeo

It is now in the queue for this repository.

[Kivooeo]

@bors r-

wait, actually not

can you please squash commits into one before merge?

[rust-bors]

Commit 19b04f2 has been unapproved.

[JayanAXHF]

Done!

cc: @Kivooeo

[Kivooeo]

@bors r+

[rust-bors]

📌 Commit 67c45b7 has been approved by Kivooeo

It is now in the queue for this repository.