Implement RFC 3503: frontmatters #140035

fee1-dead · 2025-04-19T09:36:52Z

Supercedes #137193. This implements RFC 3503.

This might break rust-analyzer. Will look into how to fix that.

Suggestions welcome for how to improve diagnostics.

rustbot · 2025-04-19T09:36:57Z

rustbot has assigned @jieyouxu.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

rust-log-analyzer · 2025-04-19T09:51:12Z

The job mingw-check failed! Check out the build log: (web) (plain)

Click to see the possible cause of the failure (guessed by this bot)

[RUSTC-TIMING] stdx test:false 0.210
    Checking parser v0.0.0 (/checkout/src/tools/rust-analyzer/crates/parser)
[RUSTC-TIMING] chalk_derive test:false 0.840
    Checking tt v0.0.0 (/checkout/src/tools/rust-analyzer/crates/tt)
error[E0004]: non-exhaustive patterns: `&TokenKind::Frontmatter { .. }` not covered
   --> src/tools/rust-analyzer/crates/parser/src/lexed_str.rs:171:19
    |
171 |             match kind {
    |                   ^^^^ pattern `&TokenKind::Frontmatter { .. }` not covered
    |
note: `TokenKind` defined here
   --> /checkout/compiler/rustc_lexer/src/lib.rs:58:1
    |
58  | pub enum TokenKind {
    | ^^^^^^^^^^^^^^^^^^
...
76  |     Frontmatter {
    |     ----------- not covered
    = note: the matched value is of type `&TokenKind`
help: ensure that all possible cases are being handled by adding a match arm with a wildcard pattern or an explicit pattern as shown
    |
254 ~                 rustc_lexer::TokenKind::Eof => EOF,
255 ~                 &TokenKind::Frontmatter { .. } => todo!(),
    |

For more information about this error, try `rustc --explain E0004`.
[RUSTC-TIMING] parser test:false 0.378
error: could not compile `parser` (lib) due to 1 previous error

jieyouxu · 2025-04-19T10:54:18Z

I'll do a review pass next Monday/Tuesday, but in the meantime, I'll roll this to Wesley who I think have more context on the general vibes.

r? @wesleywiser

jieyouxu

Thanks for the implementation! I left some (adversarial) review notes under the impression based on reading back #137193 that frontmatter is a Rust language syntax. However, the existing diagnostics here I find is already quite nice.

I especially appreciate using error annotations in between frontmatter openers/closers, I found that very clever (and cute) 😆

jieyouxu · 2025-04-22T12:24:09Z

compiler/rustc_lexer/src/lib.rs

-    let mut cursor = Cursor::new(input);
+    let mut cursor = Cursor::new(input, false);


Nit (readability): I find this at call sites somewhat hard to understand. Maybe introduce an enum like FrontmatterAllowed::{Yes,No} explicitly?

jieyouxu · 2025-04-22T12:51:26Z

tests/ui/frontmatter/frontmatter-whitespace-4.rs

+--- cargo
+//~^ ERROR: invalid infostring for frontmatter
+---


Question [FRONT-TERM 1/2]: wait, why is this rejected? My reading of the RFC text

Opens with 3+ dashes followed by 0+ whitespace, an optional term (one or more characters excluding whitespace and commas), 0+ whitespace, and a newline

is that ---cargo and --- cargo and --- cargo are considered valid due to "opens with 3+ dashes followed by 0+ whitespace" ? 🤔 Maybe I'm misreading it? As in, my interpretation of this grammar is sth like (EBNF-esque syntax)

frontmatter start = three or more dashes , [ whitespaces ] , [ term ] , [ whitespaces ] , '\n'

where each of the whitespaces and term are optional.

(See also [FRONT-TERM 2/2], I think this as described in the RFC text is ambiguous)

jieyouxu · 2025-04-22T12:56:42Z

tests/ui/frontmatter/frontmatter-whitespace-4.stderr

+LL | --- cargo
+   |    ^^^^^^
+   |
+   = note: frontmatter infostrings must be a single identifier immediately following the opening


Question [FRONT-TERM 2/2]: (Adversarial review, thinking out loud here)

and re. this diagnostics, my reading of the RFC is that all the following are considered "valid" openers:

--- no infostring like cargo or whatever, w.r.t. "an optional term"

---cargo or --- cargo or --- cargo infostrings ("term" is what the RFC uses I guess), "0+ whitespace" between --- and cargo

--- -: note the whitespace between --- and the "term" here -. The RFC says "an optional term (one or more characters excluding whitespace and commas)" which notably does not exclude -.

And these are considered "invalid" openers:

---,, --- ,: violates "excluding whitespace and commas"

--- a b: space between "term" characters

--- edition2024,no_run: comma between "term" characters

Was this changed after the RFC was accepted? As in, my interpretation of the "term" is that it's only (EBNF-esque syntax)

term = { all characters - ( whitespace | ',' ) }

where { .. } indicates repetition. Notably, it accepts -.

Hmm, actually, isn't this ambiguous? For instance, take the following opener

----\n can be considered:

"3+ dashes" (4 in fact), optional 0+ whitespace (take 0), optional term (none), optional 0+ whitespace (take 0), and a newline

But it can also be considered 3 dashes followed by optional 0+ whitespace (take 0), an optional term (one or more characters excluding whitespace and commas) (take - as the character), optional 0+ whitespace (take 0), and a newline

🤔

jieyouxu · 2025-04-22T13:28:42Z

tests/ui/frontmatter/frontmatter-after-tokens.rs

+#![feature(frontmatter)]
+
+---
+//~^ ERROR: expected item, found `-`
+// FIXME(frontmatter): make this diagnostic better
+---


Remark: yeah, I'm not sure how to detect this properly without making some convoluted logic, even though I imagine this would be one of the more common mistakes to make. At least this does error, maybe we could provide some kind of contextual HELP like if we see --- (3+ starting dashes)

error: expected item, found `-` --> $DIR/frontmatter-after-tokens.rs:3:1 | LL | --- | ^ expected item | = note: for a full list of items that can appear in modules, see <https://doc.rust-lang.org/reference/items.html> = help: if you meant to write a frontmatter, the frontmatter must come after an optional shebang but before any regular source code

(With some better wording, I find it tricky to explain)

Words are hard 🙂

If the file doesn't start with a shebang line, maybe we could just say something like

= help: if you meant to write a frontmatter, the frontmatter must appear first in the file

Shebangs are not super commonly used so if we can leave that detail out, it would simplify the error message.

jieyouxu · 2025-04-22T13:29:58Z

compiler/rustc_feature/src/unstable.rs

+    /// Frontmatter `---` blocks for use by external tools.
+    (unstable, frontmatter, "CURRENT_RUSTC_VERSION", Some(136889)),


Discussion: (stealing this to force an inline comment, not about this line of code itself)

Some positive/negative test coverage that you may or may not want to consider (but not necessarily blocking for this initial PR, but may be blockers prior to stabilization):

How does frontmatter interact with include!()?

Maybe also a smoke test for interactions with file!()/line!()?

(Probably not for this PR) How does frontmatter interact with -Zunpretty as mentioned by @fmease in Add unstable frontmatter support #137193 (comment)?

(A stretch) maybe a positive smoke test that the offsets are correctly reflected in an example debuginfo test? But I guess that might be overkill since the span offsets should already be handled by that time.

(Before stabilization) we should add a run-make test that exercises how cargo and rustc works together w.r.t. frontmatter, depending on what the final cooperation scheme we end up using.

(Before stabilization) we should add a positive smoke test to check that rustfix is able to account for frontmatters (or properly ignore), i.e. //@ run-rustfix.

jieyouxu · 2025-04-22T13:34:26Z

compiler/rustc_feature/src/unstable.rs

+    /// Frontmatter `---` blocks for use by external tools.
+    (unstable, frontmatter, "CURRENT_RUSTC_VERSION", Some(136889)),


Discussion (meta, not necessarily for this PR): hm, frontmatter is going to be interesting when it comes to cooking it sufficiently for stabilization. Like rustfmt, clippy, rust-analyzer, cargo might all at least need to learn how to account for (or properly ignore) frontmatters in source files, since it's a language syntax and not just sth tools can be expected to naively strip away prior to passing a modified source file to rustc.

jieyouxu · 2025-04-22T13:41:30Z

tests/ui/frontmatter/escape.rs

Suggestion (meta): it'd also be quite helpful if you could add a brief comment for each of these frontmatter tests regarding their intention. I can guess of their intention, but I still need to stare at them (and compare against the RFC text) for a few moments.

bors · 2025-04-23T19:35:43Z

☔ The latest upstream changes (presumably #140180) made this pull request unmergeable. Please resolve the merge conflicts.

wesleywiser · 2025-04-23T22:13:40Z

compiler/rustc_lexer/src/lib.rs

+        }
+
+        if !found {
+            // recovery strategy: a closing statement might have precending whitespace/newline


This looks reasonable! If we don't find a ---, perhaps we could also look for a use statement or a crate attribute or a //! doc comment as these might plausibly be expected to often follow a frontmatter block.

wesleywiser · 2025-04-23T22:21:40Z

tests/ui/frontmatter/frontmatter-after-tokens.rs

+#![feature(frontmatter)]
+
+---
+//~^ ERROR: expected item, found `-`
+// FIXME(frontmatter): make this diagnostic better
+---


Words are hard 🙂

If the file doesn't start with a shebang line, maybe we could just say something like

= help: if you meant to write a frontmatter, the frontmatter must appear first in the file

Shebangs are not super commonly used so if we can leave that detail out, it would simplify the error message.

Implement RFC 3503: frontmatters

f9112c6

Supercedes rust-lang#137193

rustbot assigned jieyouxu Apr 19, 2025

jieyouxu added the F-frontmatter `#![feature(frontmatter)]` label Apr 19, 2025

rustbot assigned wesleywiser and unassigned jieyouxu Apr 19, 2025

jieyouxu self-assigned this Apr 19, 2025

jieyouxu reviewed Apr 22, 2025

View reviewed changes

wesleywiser reviewed Apr 23, 2025

View reviewed changes

jieyouxu added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Apr 25, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement RFC 3503: frontmatters #140035

Implement RFC 3503: frontmatters #140035

fee1-dead commented Apr 19, 2025

rustbot commented Apr 19, 2025

rust-log-analyzer commented Apr 19, 2025

jieyouxu commented Apr 19, 2025

jieyouxu left a comment

jieyouxu Apr 22, 2025

jieyouxu Apr 22, 2025 •

edited

Loading

jieyouxu Apr 22, 2025 •

edited

Loading

jieyouxu Apr 22, 2025

wesleywiser Apr 23, 2025

jieyouxu Apr 22, 2025

jieyouxu Apr 22, 2025

jieyouxu Apr 22, 2025

bors commented Apr 23, 2025

wesleywiser Apr 23, 2025

wesleywiser Apr 23, 2025

		let mut cursor = Cursor::new(input);
		let mut cursor = Cursor::new(input, false);

		/// Frontmatter `---` blocks for use by external tools.
		(unstable, frontmatter, "CURRENT_RUSTC_VERSION", Some(136889)),

Implement RFC 3503: frontmatters #140035

Are you sure you want to change the base?

Implement RFC 3503: frontmatters #140035

Conversation

fee1-dead commented Apr 19, 2025

rustbot commented Apr 19, 2025

rust-log-analyzer commented Apr 19, 2025

jieyouxu commented Apr 19, 2025

jieyouxu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jieyouxu Apr 22, 2025 • edited Loading

Choose a reason for hiding this comment

jieyouxu Apr 22, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bors commented Apr 23, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jieyouxu Apr 22, 2025 •

edited

Loading

jieyouxu Apr 22, 2025 •

edited

Loading