Remove `Nonterminal` and `TokenKind::Interpolated` #124141

nnethercote · 2024-04-18T23:28:21Z

A third attempt at this; the first attempt was #96724 and the second was #114647.

rustbot · 2024-04-18T23:28:25Z

Some changes occurred in src/tools/rustfmt

cc @rust-lang/rustfmt

ijackson · 2024-05-10T15:28:11Z

❤️ @nnethercote for working on this. Thank you! I'm not sure if there's a way for me to help, as someone who doesn't really know much about the compiler innards, but please LMK if you think of something.

nnethercote · 2024-05-10T20:45:22Z

@ijackson: thanks! I'm curious why you are interested in this change, given that it's a compiler internals rearrangement?

nnethercote · 2024-05-10T20:47:31Z

@ijackson: Oh, I see, you are interested in #67062 being fixed. Unfortunately my current thoughts are that this PR alone won't be enough to fix that issue, though it's a necessary stepping stone.

ijackson · 2024-05-10T22:40:41Z

@ijackson: Oh, I see, you are interested in #67062 being fixed. Unfortunately my current thoughts are that this PR alone won't be enough to fix that issue, though it's a necessary stepping stone.

Right. It seems ... quite nontrivial. So, thanks.

dev-ardi · 2024-05-12T21:49:53Z

After this is done TokenKind will become Copy right?

nnethercote · 2024-05-12T22:35:11Z

After this is done TokenKind will become Copy right?

Yes.

Instead of using AST pretty printing. This is a step towards removing `token::Interpolated`, which will eventually (in rust-lang#124141) be replaced with a token stream within invisible delimiters. This changes (improves) the output of the `stringify!` macro in some cases. This is allowed. As the `stringify!` docs say: "Note that the expanded results of the input tokens may change in the future. You should be careful if you rely on the output." Test changes: - tests/ui/macros/stringify.rs: this used to test both token stream pretty printing and AST pretty printing via different ways of invoking of `stringify!` (i.e. `$expr` vs `$tt`). But those two different invocations now give the same result, which is a nice consistency improvement. This removes the need for the `c2!` macro. - tests/ui/macros/trace_faulty_macros.rs: there is some sub-optimal spacing in the printing of `A { a : a, b : 0, c : _, .. }`, which will be fixed in the next commit. The spacing of `1+1` improves -- it now matches the formatting in the source code. - tests/ui/proc-macro/*: minor improvements where small differences between `INPUT (DISPLAY)` output and `DEEP-RE-COLLECTED (DISPLAY)` output disappear.

…, r=<try> Print `token::Interpolated` with token stream pretty printing. This is a step towards removing `token::Interpolated` (rust-lang#124141). It unavoidably changes the output of the `stringify!` macro, generally for the better. r? `@petrochenkov`

nnethercote · 2024-05-17T06:47:27Z

#125174 carves off a piece of this PR so it can be merged separately.

bors · 2024-05-18T00:46:59Z

☔ The latest upstream changes (presumably #123865) made this pull request unmergeable. Please resolve the merge conflicts.

Add tests for `-Zunpretty=expanded` ported from stringify's tests This PR adds a new set of tests for the AST pretty-printer. Previously, pretty-printer edge cases were tested by way of `stringify!` in [tests/ui/macros/stringify.rs](https://github.com/rust-lang/rust/blob/1.78.0/tests/ui/macros/stringify.rs), such as the tests added by rust-lang@419b269 and rust-lang@527e2ea. Those tests will no longer provide effective coverage of the AST pretty-printer after rust-lang#124141. `Nonterminal` and `TokenKind::Interpolated` are being removed, and a consequence is that `stringify!` will perform token stream pretty printing, instead of AST pretty printing, in all of the `stringify!` cases including $:expr and all other interpolations. This PR adds 2 new ui tests with `compile-flags: -Zunpretty=expanded`: - **tests/ui/unpretty/expanded-exhaustive.rs** — this test aims for exhaustive coverage of all the variants of `ExprKind`, `ItemKind`, `PatKind`, `StmtKind`, `TyKind`, and `VisibilityKind`. Some parts could use being fleshed out further, but the current state is roughly on par with what exists in the old stringify-based tests. - **tests/ui/unpretty/expanded-interpolation.rs** — this test covers tricky macro metavariable edge cases that require the AST pretty printer to synthesize parentheses in order for the printed code to be valid Rust syntax. r? `@nnethercote`

Rollup merge of rust-lang#125236 - dtolnay:expandtest, r=nnethercote Add tests for `-Zunpretty=expanded` ported from stringify's tests This PR adds a new set of tests for the AST pretty-printer. Previously, pretty-printer edge cases were tested by way of `stringify!` in [tests/ui/macros/stringify.rs](https://github.com/rust-lang/rust/blob/1.78.0/tests/ui/macros/stringify.rs), such as the tests added by rust-lang@419b269 and rust-lang@527e2ea. Those tests will no longer provide effective coverage of the AST pretty-printer after rust-lang#124141. `Nonterminal` and `TokenKind::Interpolated` are being removed, and a consequence is that `stringify!` will perform token stream pretty printing, instead of AST pretty printing, in all of the `stringify!` cases including $:expr and all other interpolations. This PR adds 2 new ui tests with `compile-flags: -Zunpretty=expanded`: - **tests/ui/unpretty/expanded-exhaustive.rs** — this test aims for exhaustive coverage of all the variants of `ExprKind`, `ItemKind`, `PatKind`, `StmtKind`, `TyKind`, and `VisibilityKind`. Some parts could use being fleshed out further, but the current state is roughly on par with what exists in the old stringify-based tests. - **tests/ui/unpretty/expanded-interpolation.rs** — this test covers tricky macro metavariable edge cases that require the AST pretty printer to synthesize parentheses in order for the printed code to be valid Rust syntax. r? `@nnethercote`

Instead of using AST pretty printing. This is a step towards removing `token::Interpolated`, which will eventually (in rust-lang#124141) be replaced with a token stream within invisible delimiters. This changes (improves) the output of the `stringify!` macro in some cases. This is allowed. As the `stringify!` docs say: "Note that the expanded results of the input tokens may change in the future. You should be careful if you rely on the output." Test changes: - tests/ui/macros/stringify.rs: this used to test both token stream pretty printing and AST pretty printing via different ways of invoking of `stringify!` (i.e. `$expr` vs `$tt`). But those two different invocations now give the same result, which is a nice consistency improvement. This removes the need for all the `c2*` macros. The AST pretty printer now has more thorough testing thanks to rust-lang#125236. - tests/ui/proc-macro/*: minor improvements where small differences between `INPUT (DISPLAY)` output and `DEEP-RE-COLLECTED (DISPLAY)` output disappear.

petrochenkov · 2024-05-23T20:01:24Z

It's great to see that enum InvisibleOrigin allows to migrate the parser to delimited groups relatively simply, with just the maybe_whole to maybe_reparse_metavar_seq replacement.

Of course it prevents a lot of interesting stuff like reparsing expr as pat and similar, like it would work in a purely token-based model, but all that can be carefully introduced later, when it's possible to do backward compatibly.

petrochenkov · 2024-05-23T20:02:23Z

How hard would it be to get this to a perf run?
(With or without the NtExpr/NtLiteral stuff.)

petrochenkov · 2024-05-23T20:03:04Z

Blocked on #125174.
@rustbot blocked

mcol · 2024-10-01T06:21:23Z

Note this sentence in your report (3rd section): "However, if the code is split across two crates, rustc0 will accept it, while rustc1 accepts it. "

nnethercote · 2024-10-01T22:26:57Z

Note this sentence in your report (3rd section): "However, if the code is split across two crates, rustc0 will accept it, while rustc1 accepts it. "

Fixed, thanks.

It was added in rust-lang#130349, but it's not used meaningfully, and causes difficulties for Nonterminal removal in rust-lang#124141.

bors · 2024-10-29T15:32:44Z

☔ The latest upstream changes (presumably #132317) made this pull request unmergeable. Please resolve the merge conflicts.

…r=estebank Use `token_descr` more in error messages This is the first two commits from rust-lang#124141, put into their own PR to get things rolling. Commit messages have the details. r? `@estebank` cc `@petrochenkov`

…r=estebank Use `token_descr` more in error messages This is the first two commits from rust-lang#124141, put into their own PR to get things rolling. Commit messages have the details. r? ``@estebank`` cc ``@petrochenkov``

Rollup merge of rust-lang#132332 - nnethercote:use-token_descr-more, r=estebank Use `token_descr` more in error messages This is the first two commits from rust-lang#124141, put into their own PR to get things rolling. Commit messages have the details. r? ``@estebank`` cc ``@petrochenkov``

It's not used meaningfully yet, but will be needed to get rid of interpolated tokens.

Pasted metavariables are wrapped in invisible delimiters, which pretty-print as empty strings, and changing that can break some proc macros. But error messages saying "expected identifer, found ``" are bad. So this commit adds support for metavariables in `TokenDescription` so they print as "metavariable" in error messages, instead of "``". It's not used meaningfully yet, but will be needed to get rid of interpolated tokens.

Current places where `Interpolated` is used are going to change to instead use invisible delimiters. This prepares for that. - It adds invisible delimiter cases to the `can_begin_*`/`may_be_*` methods and the `failed_to_match_macro` that are equivalent to the existing `Interpolated` cases. - It adds panics/asserts in some places where invisible delimiters should never occur. - In `Parser::parse_struct_fields` it excludes an ident + invisible delimiter from special consideration in an error message, because that's quite different to an ident + paren/brace/bracket.

We now use invisible delimiters for expanded `vis` fragments, instead of `Token::Interpolated`.

Notes about tests: - tests/ui/parser/macro/trait-object-macro-matcher.rs: the syntax error is duplicated, because it occurs now when parsing the decl macro input, and also when parsing the expanded decl macro. But this won't show up for normal users due to error de-duplication. - tests/ui/associated-consts/issue-93835.rs: ditto. - The changes to metavariable descriptions in this PR's earlier commits are now visible in error message for several tests.

The one notable test change is `tests/ui/macros/trace_faulty_macros.rs`. This commit removes the complicated `Interpolated` handling in `expected_expression_found` that results in a longer error message. But I think the new, shorter message is actually an improvement. The original complaint was in rust-lang#71039, when the error message started with "error: expected expression, found `1 + 1`". That was confusing because `1 + 1` is an expression. Other than that, the reporter said "the whole error message is not too bad if you ignore the first line". Subsequently, extra complexity and wording was added to the error message. But I don't think the extra wording actually helps all that much. In particular, it still says of the `1+1` that "this is expected to be expression". This repeats the problem from the original complaint! This commit removes the extra complexity, reverting to a simpler error message. This is primarily because the traversal is a pain without `Interpolated` tokens. Nonetheless, I think the error message is *improved*. It now starts with "expected expression, found `pat` metavariable", which is much clearer and the real problem. It also doesn't say anything specific about `1+1`, which is good, because the `1+1` isn't really relevant to the error -- it's the `$e:pat` that's important.

This involves replacing `nt_pretty_printing_compatibility_hack` with `stream_pretty_printing_compatibility_hack`. The handling of statements in `transcribe` is slightly different to other nonterminal kinds, due to the lack of `from_ast` implementation for empty statements. Notable test changes: - `tests/ui/proc-macro/expand-to-derive.rs`: the diff looks large but the only difference is the insertion of a single invisible-delimited group around a metavar.

This time when converting them to proc-macro `Group` form.

Note: there was an existing code path involving `Interpolated` in `MetaItem::from_tokens` that was dead. This commit transfers that to the new form, but puts an `unreachable!` call inside it.

It was added in rust-lang#130349, but it's not used meaningfully, and causes difficulties for Nonterminal removal in rust-lang#124141.

Notes about tests: - tests/ui/rfcs/rfc-2294-if-let-guard/feature-gate.rs: some messages are now duplicated due to repeated parsing. - tests/ui/rfcs/rfc-2497-if-let-chains/disallowed-positions*.rs: ditto. - `tests/ui/proc-macro/macro-rules-derive-cfg.rs`: the diff looks large but the only difference is the insertion of a single invisible-delimited group around a metavar.

`NtBlock` is the last remaining variant of `Nonterminal`, so once it is gone then `Nonterminal` can be removed as well.

It's no longer needed. This does slightly worsen the error message for a single test, but that test contains code that is so badly broken that I'm not worried about it.

It was added in rust-lang#130349, but it's not used meaningfully, and causes difficulties for Nonterminal removal in rust-lang#124141.

…<try> rust-lang#124141 preliminaries Preliminary changes required to start removing `Nonterminal`. r? `@petrochenkov`

bors · 2024-11-10T08:32:51Z

☔ The latest upstream changes (presumably #132831) made this pull request unmergeable. Please resolve the merge conflicts.

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Apr 18, 2024

nnethercote marked this pull request as draft April 18, 2024 23:28

nnethercote mentioned this pull request Apr 18, 2024

Start removing Nonterminal #114647

Closed

nnethercote force-pushed the rm-Nonterminal-and-TokenKind-Interpolated branch 2 times, most recently from 42a623a to c133e16 Compare April 23, 2024 03:51

petrochenkov self-assigned this Apr 28, 2024

nnethercote force-pushed the rm-Nonterminal-and-TokenKind-Interpolated branch from c133e16 to 7aef5db Compare May 16, 2024 10:50

This comment has been minimized.

Sign in to view

nnethercote mentioned this pull request May 17, 2024

Print token::Interpolated with token stream pretty printing. #125174

Merged

dtolnay mentioned this pull request May 18, 2024

Add tests for -Zunpretty=expanded ported from stringify's tests #125236

Merged

rustbot removed the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label May 23, 2024

nnethercote force-pushed the rm-Nonterminal-and-TokenKind-Interpolated branch from 4c46a2a to c7cf783 Compare October 28, 2024 03:47

nnethercote mentioned this pull request Oct 29, 2024

Use token_descr more in error messages #132332

Merged

nnethercote added 16 commits October 31, 2024 06:05

Introduce InvisibleOrigin on invisible delimiters.

4bddbd5

It's not used meaningfully yet, but will be needed to get rid of interpolated tokens.

Remove NtVis.

2ccfa1a

We now use invisible delimiters for expanded `vis` fragments, instead of `Token::Interpolated`.

More denesting of invisibly-delimited groups.

b30a69f

This time when converting them to proc-macro `Group` form.

Remove NtMeta.

8c7b344

Note: there was an existing code path involving `Interpolated` in `MetaItem::from_tokens` that was dead. This commit transfers that to the new form, but puts an `unreachable!` call inside it.

Remove NtPath.

8540c76

Remove ErrorGuaranteed retval from error_unexpected_after_dot.

76abcce

It was added in rust-lang#130349, but it's not used meaningfully, and causes difficulties for Nonterminal removal in rust-lang#124141.

Fix a problem with metavars and inner attributes.

423dd94

Remove NtBlock, Nonterminal, and TokenKind::Interpolated.

7cd05e9

`NtBlock` is the last remaining variant of `Nonterminal`, so once it is gone then `Nonterminal` can be removed as well.

Impl Copy for Token and TokenKind.

de3cf26

Remove TokenStream::flattened.

9dfb022

It's no longer needed. This does slightly worsen the error message for a single test, but that test contains code that is so badly broken that I'm not worried about it.

nnethercote force-pushed the rm-Nonterminal-and-TokenKind-Interpolated branch from c7cf783 to 9dfb022 Compare October 30, 2024 21:10

bors added a commit to rust-lang-ci/rust that referenced this pull request Nov 5, 2024

Auto merge of rust-lang#132629 - nnethercote:124141-preliminaries, r=…

feea15e

…<try> rust-lang#124141 preliminaries Preliminary changes required to start removing `Nonterminal`. r? `@petrochenkov`

petrochenkov mentioned this pull request Nov 9, 2024

#124141 preliminaries #132629

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove `Nonterminal` and `TokenKind::Interpolated` #124141

Remove `Nonterminal` and `TokenKind::Interpolated` #124141

nnethercote commented Apr 18, 2024 •

edited

Loading

rustbot commented Apr 18, 2024

ijackson commented May 10, 2024

nnethercote commented May 10, 2024

nnethercote commented May 10, 2024

ijackson commented May 10, 2024

dev-ardi commented May 12, 2024

nnethercote commented May 12, 2024

This comment has been minimized.

nnethercote commented May 17, 2024 •

edited

Loading

bors commented May 18, 2024

petrochenkov commented May 23, 2024 •

edited

Loading

petrochenkov commented May 23, 2024

petrochenkov commented May 23, 2024

mcol commented Oct 1, 2024

nnethercote commented Oct 1, 2024

bors commented Oct 29, 2024

bors commented Nov 10, 2024

Remove Nonterminal and TokenKind::Interpolated #124141

Are you sure you want to change the base?

Remove Nonterminal and TokenKind::Interpolated #124141

Conversation

nnethercote commented Apr 18, 2024 • edited Loading

rustbot commented Apr 18, 2024

ijackson commented May 10, 2024

nnethercote commented May 10, 2024

nnethercote commented May 10, 2024

ijackson commented May 10, 2024

dev-ardi commented May 12, 2024

nnethercote commented May 12, 2024

This comment has been minimized.

nnethercote commented May 17, 2024 • edited Loading

bors commented May 18, 2024

petrochenkov commented May 23, 2024 • edited Loading

petrochenkov commented May 23, 2024

petrochenkov commented May 23, 2024

mcol commented Oct 1, 2024

nnethercote commented Oct 1, 2024

bors commented Oct 29, 2024

bors commented Nov 10, 2024

Remove `Nonterminal` and `TokenKind::Interpolated` #124141

Remove `Nonterminal` and `TokenKind::Interpolated` #124141

nnethercote commented Apr 18, 2024 •

edited

Loading

nnethercote commented May 17, 2024 •

edited

Loading

petrochenkov commented May 23, 2024 •

edited

Loading