Start removing Nonterminal
#114647
Closed
Start removing Nonterminal
#114647
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rustbot
added
S-waiting-on-review
|
This is a successor to #96724, where I started doing this with The code is draft quality. So far I have removed It's unclear to me how to best stage this work, whether to do it in pieces, or all in a single huge PR. |
This comment has been minimized.
This comment has been minimized.
nnethercote
force-pushed
the
start-rm-Nonterminal
branch
3 times, most recently
from
86999c2
to
1034bc0
Compare
|
The latest update removes |
nnethercote
force-pushed
the
start-rm-Nonterminal
branch
3 times, most recently
from
390563a
to
be30bb6
Compare
|
The latest update removes |
|
The PR is marked as draft, so marking is as waiting on author as well (I'll be busy for the next few days, and won't be able to review anyway). Once this is ready I'll ask to split all the preparational commits + one of the smaller nonterminals ( |
rustbot
added
S-waiting-on-author
|
#114571 should be considered ahead of this one. It improves token stream pretty printing greatly, which helps minimize changes to tests in this PR. |
nnethercote
force-pushed
the
start-rm-Nonterminal
branch
from
be30bb6
to
3db455d
Compare
|
The latest update removes |
nnethercote
force-pushed
the
start-rm-Nonterminal
branch
from
3db455d
to
a93445e
Compare
|
The latest update removes |
|
And now |
|
☔ The latest upstream changes (presumably #114803) made this pull request unmergeable. Please resolve the merge conflicts. |
nnethercote
force-pushed
the
start-rm-Nonterminal
branch
from
d5df8c4
to
67f4e3c
Compare
|
The latest updates remove I haven't yet added handling of invisible delimiters from proc macros, which will be required to fix #67062. @petrochenkov: it might be worth taking a look now, to see what you think of the overall approach. |
This comment has been minimized.
This comment has been minimized.
|
That test failure above on the I have opened #114915 for all the preliminary cleanup commits. |
Merged
Note: there was an existing code path involving `Interpolated` in `MetaItem::from_tokens` that was dead. This commit transfers that to the new form, but puts an `unreachable!` call inside it.
[xt] move from_token
Getting a test failure here:
```
Building tool error_index_generator (stage1 -> stage2, x86_64-unknown-linux-gnu)
Compiling cfg-if v1.0.0
...
Compiling mdbook v0.4.31
error: internal compiler error: the following error was constructed but not emitted
error: unexpected token: keyword `self`
--> /home/njn/.cargo/registry/src/index.crates.io-6f17d22bba15001f/mdbook-0.4.31/src/book/summary.rs:275:31
|
275 | bail!(self.parse_error("Suffix chapters cannot be followed by a list"));
| ^^^^
thread 'rustc' panicked at compiler/rustc_errors/src/diagnostic_builder.rs:775:21:
error was constructed but not emitted
```
Remnants of `Token::Interpolated` still exist in the form of the new `token::IdentMv` and `token::LifetimeMv` tokens. I did them like that because there's a lot of code that assumes an interpolated ident/lifetime fits in a single token, and changing all that code to work with invisible delimiters would have been a pain. (Maybe it could be done in a follow-up.) Fully kills off the "captured metavariables except for `:tt`, `:ident` and `:lifetime` cannot be compared to other tokens" restriction.
nnethercote
force-pushed
the
start-rm-Nonterminal
branch
from
d2664fa
to
848d2d2
Compare
|
The last commit partially implements removal of invisible delimiter skipping. This is enough to make the examples in #67062 work, but the problems mentioned above occur. The code in that commit is very rough. |
|
@nnethercote any updates on this? this has ended up bitrotting heavily |
bors
added a commit
to rust-lang-ci/rust
that referenced
this pull request
Feb 18, 2024
…ors,cjgillot macro_rules: Preserve all metavariable spans in a global side table This PR preserves spans of `tt` metavariables used to pass tokens to declarative macros. Such metavariable spans can then be used in span combination operations like `Span::to` to improve all kinds of diagnostics. Spans of non-`tt` metavariables are currently kept in nonterminal tokens, but the long term plan is remove all nonterminal tokens from rustc parser and rely on the proc macro model with invisible delimiters (rust-lang#114647, rust-lang#67062). In particular, `NtIdent` nonterminal (corresponding to `ident` metavariables) becomes easy to remove when this PR lands (rust-lang#119412 does it). The metavariable spans are kept in a global side table keyed by `Span`s of original tokens. The alternative to the side table is keeping them in `SpanData` instead, but the performance regressions would be large because any spans from tokens passed to declarative macros would stop being inline and would work through span interner instead, and the penalty would be paid even if we never use the metavar span for the given original span. (But also see the comment on `fn maybe_use_metavar_location` describing the map collision issues with the side table approach.) There are also other alternatives - keeping the metavar span in `Token` or `TokenTree`, but associating it with `Span` itsel is the most natural choice because metavar spans are used in span combining operations, and those operations are not necessarily tied to tokens.
bors
added a commit
to rust-lang-ci/rust
that referenced
this pull request
Feb 18, 2024
…ors,cjgillot macro_rules: Preserve all metavariable spans in a global side table This PR preserves spans of `tt` metavariables used to pass tokens to declarative macros. Such metavariable spans can then be used in span combination operations like `Span::to` to improve all kinds of diagnostics. Spans of non-`tt` metavariables are currently kept in nonterminal tokens, but the long term plan is remove all nonterminal tokens from rustc parser and rely on the proc macro model with invisible delimiters (rust-lang#114647, rust-lang#67062). In particular, `NtIdent` nonterminal (corresponding to `ident` metavariables) becomes easy to remove when this PR lands (rust-lang#119412 does it). The metavariable spans are kept in a global side table keyed by `Span`s of original tokens. The alternative to the side table is keeping them in `SpanData` instead, but the performance regressions would be large because any spans from tokens passed to declarative macros would stop being inline and would work through span interner instead, and the penalty would be paid even if we never use the metavar span for the given original span. (But also see the comment on `fn maybe_use_metavar_location` describing the map collision issues with the side table approach.) There are also other alternatives - keeping the metavar span in `Token` or `TokenTree`, but associating it with `Span` itsel is the most natural choice because metavar spans are used in span combining operations, and those operations are not necessarily tied to tokens.
|
There are still some good ideas in here that I'd like to pursue, so I'll keep it open. |
|
My next attempt at this is now in #124141. |
nnethercote
added a commit
to nnethercote/rust
that referenced
this pull request
Apr 22, 2024
The extra span is now recorded in the new `TokenKind::InterpolatedIdent` and `TokenKind::InterpolatedLifetime`. These both consist of a single token, and so there's no operator precedence problems with inserting them directly into the token stream. The other way to do this would be to wrap the ident/lifetime in invisible delimiters, but there's a lot of code that assumes an interpolated ident/lifetime fits in a single token, and changing all that code to work with invisible delimiters would have been a pain. (Maybe it could be done in a follow-up.) This change might not seem like much of a win, but it's a first step toward the much bigger and long-desired removal of `Nonterminal` and `TokenKind::Interpolated`. That change is big and complex enough that it's worth doing this piece separately. (Indeed, this commit is based on part of a late commit in rust-lang#114647, a prior attempt at that big and complex change.)
bors
added a commit
to rust-lang-ci/rust
that referenced
this pull request
Apr 22, 2024
…=<try> Remove `NtIdent` and `NtLifetime` This is one part of the bigger "remove `Nonterminal` and `TokenKind::Interpolated`" change drafted in rust-lang#114647. More details in the individual commit messages. r? `@petrochenkov`
nnethercote
added a commit
to nnethercote/rust
that referenced
this pull request
May 13, 2024
The extra span is now recorded in the new `TokenKind::InterpolatedIdent` and `TokenKind::InterpolatedLifetime`. These both consist of a single token, and so there's no operator precedence problems with inserting them directly into the token stream. The other way to do this would be to wrap the ident/lifetime in invisible delimiters, but there's a lot of code that assumes an interpolated ident/lifetime fits in a single token, and changing all that code to work with invisible delimiters would have been a pain. (Maybe it could be done in a follow-up.) This change might not seem like much of a win, but it's a first step toward the much bigger and long-desired removal of `Nonterminal` and `TokenKind::Interpolated`. That change is big and complex enough that it's worth doing this piece separately. (Indeed, this commit is based on part of a late commit in rust-lang#114647, a prior attempt at that big and complex change.)
nnethercote
added a commit
to nnethercote/rust
that referenced
this pull request
May 13, 2024
The extra span is now recorded in the new `TokenKind::NtIdent` and `TokenKind::NtLifetime`. These both consist of a single token, and so there's no operator precedence problems with inserting them directly into the token stream. The other way to do this would be to wrap the ident/lifetime in invisible delimiters, but there's a lot of code that assumes an interpolated ident/lifetime fits in a single token, and changing all that code to work with invisible delimiters would have been a pain. (Maybe it could be done in a follow-up.) This change might not seem like much of a win, but it's a first step toward the much bigger and long-desired removal of `Nonterminal` and `TokenKind::Interpolated`. That change is big and complex enough that it's worth doing this piece separately. (Indeed, this commit is based on part of a late commit in rust-lang#114647, a prior attempt at that big and complex change.)
nnethercote
added a commit
to nnethercote/rust
that referenced
this pull request
May 13, 2024
The extra span is now recorded in the new `TokenKind::NtIdent` and `TokenKind::NtLifetime`. These both consist of a single token, and so there's no operator precedence problems with inserting them directly into the token stream. The other way to do this would be to wrap the ident/lifetime in invisible delimiters, but there's a lot of code that assumes an interpolated ident/lifetime fits in a single token, and changing all that code to work with invisible delimiters would have been a pain. (Maybe it could be done in a follow-up.) This change might not seem like much of a win, but it's a first step toward the much bigger and long-desired removal of `Nonterminal` and `TokenKind::Interpolated`. That change is big and complex enough that it's worth doing this piece separately. (Indeed, this commit is based on part of a late commit in rust-lang#114647, a prior attempt at that big and complex change.)
bors
added a commit
to rust-lang-ci/rust
that referenced
this pull request
May 14, 2024
…=petrochenkov Remove `NtIdent` and `NtLifetime` This is one part of the bigger "remove `Nonterminal` and `TokenKind::Interpolated`" change drafted in rust-lang#114647. More details in the individual commit messages. r? `@petrochenkov`
bors
added a commit
to rust-lang-ci/rust
that referenced
this pull request
Jun 23, 2024
…Kind-Interpolated, r=<try> Remove `Nonterminal` and `TokenKind::Interpolated` A third attempt at this; the first attempt was rust-lang#96724 and the second was rust-lang#114647. r? `@ghost`
3 tasks
bors
added a commit
to rust-lang-ci/rust
that referenced
this pull request
Jun 24, 2024
…Kind-Interpolated, r=<try> Remove `Nonterminal` and `TokenKind::Interpolated` A third attempt at this; the first attempt was rust-lang#96724 and the second was rust-lang#114647. r? `@ghost`
bors
added a commit
to rust-lang-ci/rust
that referenced
this pull request
Sep 10, 2024
…Kind-Interpolated, r=<try> Remove `Nonterminal` and `TokenKind::Interpolated` A third attempt at this; the first attempt was rust-lang#96724 and the second was rust-lang#114647. r? `@ghost`
bors
added a commit
to rust-lang-ci/rust
that referenced
this pull request
Sep 25, 2024
…Kind-Interpolated, r=<try> Remove `Nonterminal` and `TokenKind::Interpolated` A third attempt at this; the first attempt was rust-lang#96724 and the second was rust-lang#114647. r? `@ghost`
bors
added a commit
to rust-lang-ci/rust
that referenced
this pull request
Sep 25, 2024
…Kind-Interpolated, r=<try> Remove `Nonterminal` and `TokenKind::Interpolated` A third attempt at this; the first attempt was rust-lang#96724 and the second was rust-lang#114647. r? `@ghost`
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
A second attempt at this; the first attempt was #96724.
r? @ghost