Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use token::Lit in ast::ExprKind::Lit. #102944

Merged
merged 3 commits into from
Nov 17, 2022

Conversation

nnethercote
Copy link
Contributor

Instead of ast::Lit.

Literal lowering now happens at two different times. Expression literals are lowered when HIR is crated. Attribute literals are lowered during parsing.

r? @petrochenkov

@rustbot rustbot added the T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. label Oct 12, 2022
@rustbot
Copy link
Collaborator

rustbot commented Oct 12, 2022

Some changes occurred in src/tools/clippy

cc @rust-lang/clippy

@rust-highfive rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Oct 12, 2022
@nnethercote
Copy link
Contributor Author

Another alternative to the literal changes in #101562 and #101885.

I don't feel entirely happy with this approach. Various things:

  • The output change to src/test/ui/lint/lint-impl-fn.rs is bogus, but I don't understand why it happens.
  • rustc_ast_lowering now depends on rustc_parse, for report_lit_error, because lit lowering now happens in both of those crates.
  • ast::Lit still has redundancy, which two literal "kinds" within it.
  • There are some almost duplicate paths in the parser:
    • parse_opt_token_lit / parse_opt_ast_lit
    • parse_token_lit / parse_ast_lit`

@nnethercote
Copy link
Contributor Author

Also, it's not yet complete. I haven't updated clippy and rustfmt. I'll do that if we decide that this is the right path forward.

@rust-log-analyzer

This comment has been minimized.

@petrochenkov
Copy link
Contributor

rustc_ast_lowering now depends on rustc_parse, for report_lit_error, because lit lowering now happens in both of those crates.

The literal lowering logic, including error reporting, is independent of the rest of the parser and can be moved to some common dependency of AST->HIR lowering and Attribute -> MetaItem lowering, or passed through a function pointer like nt_to_tokenstream before #97251.

ast::Lit still has redundancy, which two literal "kinds" within it.

I'd expect it to be renamed to something like MetaItemLit, and the token_lit part to be removed (not in this PR).

@petrochenkov
Copy link
Contributor

petrochenkov commented Oct 13, 2022

Change description for the lang team.

This PR changes some errors related to literals from syntactic-but-not-in-macros to semantic.
These are the error names in the compiler:

    InvalidSuffix,
    InvalidIntSuffix,
    InvalidFloatSuffix,
    NonDecimalFloat(u32),
    IntTooLarge,

Examples:

"string"any_suffix
10u123
10.0f123
0b10f32
999340282366920938463463374607431768211455999

All these examples are valid tokens, that can be passed to macros without errors, but previously they were reported as errors when parsing proper Rust code, including code that will be later removed by cfg or attribute macros.

// Before
sink! {
    "string"any_suffix // OK
    10u123 // OK
    10.0f123 // OK
    0b10f32 // OK
    999340282366920938463463374607431768211455999 // OK
}

#[cfg(FALSE)]
fn configured_out() {
    "string"any_suffix // ERROR
    10u123 // ERROR
    10.0f123 // ERROR
    0b10f32 // ERROR
    999340282366920938463463374607431768211455999 // ERROR
}

fn main() {
    "string"any_suffix // ERROR
    10u123 // ERROR
    10.0f123 // ERROR
    0b10f32 // ERROR
    999340282366920938463463374607431768211455999 // ERROR
}

This PR makes them fully semantic errors so they are only reported after all macros are expanded.

// After
sink! {
    "string"any_suffix // OK
    10u123 // OK
    10.0f123 // OK
    0b10f32 // OK
    999340282366920938463463374607431768211455999 // OK
}

#[cfg(FALSE)]
fn configured_out() {
    "string"any_suffix // OK
    10u123 // OK
    10.0f123 // OK
    0b10f32 // OK
    999340282366920938463463374607431768211455999 // OK
}

fn main() {
    "string"any_suffix // ERROR
    10u123 // ERROR
    10.0f123 // ERROR
    0b10f32 // ERROR
    999340282366920938463463374607431768211455999 // ERROR
}

This way

  • we make behavior of function-like and attribute-like macros more consistent
  • we make other literals more consistent with floating point literals which are already converted from strings to floats lately in the compiler (probably due to expensiveness of the conversion)
  • we avoid unnecessary work in the compiler - literals don't need to be checked/converted unless they are really used, and size of expression AST nodes can be shrank.

@petrochenkov petrochenkov added T-lang Relevant to the language team, which will review and decide on the PR/issue. S-waiting-on-team Status: Awaiting decision from the relevant subteam (see the T-<team> label). I-lang-nominated Nominated for discussion during a lang team meeting. and removed T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Oct 13, 2022
@nnethercote
Copy link
Contributor Author

we make other literals more consistent with floating point literals which are already converted from strings to floats lately in the compiler (probably due to expensiveness of the conversion)

Floats are stored as Symbols rather than f64 in ast::LitKind so that ast::LitKind can implement Eq and Hash. I think this is the real reason they are converted later, rather than cost.

@y86-dev
Copy link
Contributor

y86-dev commented Oct 18, 2022

0x10.0 // OK

This is currently not accepted by the compiler: https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=316edfc43c4ad955ee7b667883a10287

@petrochenkov
Copy link
Contributor

petrochenkov commented Oct 18, 2022

@y86-dev
Ah, you are right, 0x10.0 -> 0b10f32 in all the examples.
UPD: I've updated the examples in #102944 (comment).

@nnethercote
Copy link
Contributor Author

@scottmcm: The meeting notes indicate some support for this, but also that a firm conclusion wasn't reached. Should we do an FCP?

@bors

This comment was marked as resolved.

@scottmcm
Copy link
Member

@nnethercote I've been trying to distill my thoughts here into something less fuzzy. I think I see three different situations here.

  1. Things that match currently-existing token forms, but are hitting arbitrary length restrictions.

For example, decimal literals that are obviously lexically fine, like

#[cfg(nope)]
pub fn foo() {
    let x = 1111111111222222222233333333334444444444;
}

For that kind, I think we had lang consensus that they definitely shouldn't be a tokenizing nor parsing time error, but rather a semantic error later. So making those work for macros and cfg'd-out code as I understand this PR would do makes great sense to me.

  1. Extensions of existing lexical forms to other widths.

I don't think we're likely to add u1024 any time soon, but if we ever did I think it's very obvious that 0_u1024 would be a valid value of the type, as would 0xFFFF_u1024 and the other obvious examples. Similarly for 0.0_f16, or other such things with different bitwidths. (0.0_f11 is massively unlikely, but I still don't think it's something we'd need to block.)

I think lang also felt positive about supporting these, though I don't think we talked about them as much. Personally I'm definitely in favour.

  1. Things where the combination has no meaning today.

This is where I'm not so confident about things. When it explicitly rejects hexadecimal and binary floating point, it's not at all obvious to me that 0b0f32 is something we want to support for situations like this. What does that even mean, especially when 0x0f32 does something completely related to what an f32 suffix means near a .? Similarly, it's not obvious to me at all that "hello"match should even be one token. I guess we can't break that in macros until an edition, so it's stuck being passable there, but that doesn't mean it's something that makes sense to me to allow through the parser.


Hopefully that's understandable 🤞

How do you feel about those distinctions? Is the line between cases 1&2 and case 3 something that you think is meaningful? How obnoxious would it be to separate those cases in the code? Am I missing something about how Rust is set up lexically/syntactically?

cc @rust-lang/lang for comments in hopes this makes more sense than my rambling in the meeting.

@nnethercote
Copy link
Contributor Author

I have no strong opinions about the language design issues here. I'm just interested in shrinking the size of ast::Literal, in order to shrink ast::Expr, in order to reduce compile times.

@petrochenkov: what do think about Scott's comments?

@petrochenkov
Copy link
Contributor

@scottmcm

How do you feel about those distinctions? Is the line between cases 1&2 and case 3 something that you think is meaningful?

I wouldn't make any of these distinctions.
For me the important part is that all of them are considered valid tokens (accepted as tt or proc macro input), so it's too late to make the distinctions and they should all be accepted by Rust parser as well.
If you want to make distinctions, then perhaps the moral of this story is that we need to pay more attention when stabilizing something as a valid token.

How obnoxious would it be to separate those cases in the code? Am I missing something about how Rust is set up lexically/syntactically?

  • "string"any_suffix - less obnoxious, the suffix check is cheap and AST expressions can still keep token::Lit, although it may be annoying to not report this error twice.
  • everything else - more obnoxious, AST either needs to keep lowered literals, or the compiler will have to lower literals twice doing more work instead of less work.

@joshtriplett
Copy link
Member

We discussed this in today's @rust-lang/lang meeting, and we decided that we're willing to evaluate this via FCP. We're not thrilled that these were handled in the first place, but given that they work in macros, we should let them work in cfg(FALSE). That doesn't stop us from giving them language-level meanings in the parser later.

@joshtriplett
Copy link
Member

@rfcbot merge

@rfcbot
Copy link

rfcbot commented Oct 25, 2022

Team member @joshtriplett has proposed to merge this. The next step is review by the rest of the tagged team members:

No concerns currently listed.

Once a majority of reviewers approve (and at most 2 approvals are outstanding), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

See this document for info about what commands tagged team members can give me.

@rfcbot rfcbot added proposed-final-comment-period Proposed to merge/close by relevant subteam, see T-<team> label. Will enter FCP once signed off. disposition-merge This issue / PR is in PFCP or FCP with a disposition to merge it. labels Oct 25, 2022
@joshtriplett joshtriplett removed the I-lang-nominated Nominated for discussion during a lang team meeting. label Oct 25, 2022
@rfcbot rfcbot added final-comment-period In the final comment period and will be merged soon unless new substantive objections are raised. and removed proposed-final-comment-period Proposed to merge/close by relevant subteam, see T-<team> label. Will enter FCP once signed off. labels Oct 26, 2022
flip1995 pushed a commit to flip1995/rust-clippy that referenced this pull request Nov 19, 2022
Instead of `ast::Lit`.

Literal lowering now happens at two different times. Expression literals
are lowered when HIR is crated. Attribute literals are lowered during
parsing.

This commit changes the language very slightly. Some programs that used
to not compile now will compile. This is because some invalid literals
that are removed by `cfg` or attribute macros will no longer trigger
errors. See this comment for more details:
rust-lang/rust#102944 (comment)
mattheww added a commit to mattheww/reference that referenced this pull request Nov 20, 2022
This reflects the changes made in in rust-lang/rust#102944 .

Previously, unknown suffixes were rejected by the parser. Now they are
accepted by the parser and rejected at a later stage.

Similarly, integer literals too large to fit in u128 are now accepted by the
parser.

Forms like 5f32 are now INTEGER_LITERAL rather than FLOAT_LITERAL.

The notion of a 'pseudoliteral' is no longer required.
flip1995 pushed a commit to flip1995/rust that referenced this pull request Nov 21, 2022
…, r=petrochenkov

Use `token::Lit` in `ast::ExprKind::Lit`.

Instead of `ast::Lit`.

Literal lowering now happens at two different times. Expression literals are lowered when HIR is crated. Attribute literals are lowered during parsing.

r? `@petrochenkov`
@rylev
Copy link
Member

rylev commented Nov 22, 2022

Note from perf triage: the regressions are largely concentrated in primary benchmarks while the improvements are in secondary. So while the regressions and improvements balance out when treating all benchmarks equally, I do think that primary regressions should be considered to have more weight than secondary improvements.

ehuss added a commit to rust-lang/reference that referenced this pull request Nov 25, 2022
@ehuss ehuss mentioned this pull request Nov 27, 2022
@rustbot rustbot mentioned this pull request Dec 5, 2022
matthiaskrgr added a commit to matthiaskrgr/rust that referenced this pull request Dec 5, 2022
Update books

## rust-lang/book

1 commits in 3f64052c048c6def93b94a2b514ee88bba918744..a60f4316ec923a5ac2ed6a2eba6960edb832d855
2022-11-16 15:07:18 UTC to 2022-11-16 15:07:18 UTC

- Fix Install MdBook command (rust-lang/book#3424)

## rust-embedded/book

4 commits in c533348edd69f11a8f4225d633a05d7093fddbf3..19f798d448835a4888e3b3eae7fe69f1d61d8681
2022-11-17 15:08:11 UTC to 2022-11-08 23:27:57 UTC

- start/hardware.md: Fix typo (rust-embedded/book#336)
- doc: Fix `arm-none-eabi-gdb` installation instruction for Fedora 27 or newer to just use `gdb` (rust-embedded/book#335)
- Update singletons.md (rust-embedded/book#334)
- Remove incorrect claim HashMap is avail in no_std (rust-embedded/book#333)

## rust-lang/nomicon

2 commits in 05532356e7a4dbea2330aabb77611f5179493bb8..ae406aa5287a9e025abb72343aaceec98458c117
2022-11-21 22:48:20 UTC to 2022-11-15 00:29:20 UTC

- Improve chapter about `Vec<T>` (rust-lang/nomicon#381)
- Grammar change for 3.4: Limits of Lifetimes (lifetime-mismatch.md) (rust-lang/nomicon#386)

## rust-lang/reference

9 commits in 9f0cc13ffcd27c1fbe1ab766a9491e15ddcf4d19..3ae62681ff236d5528ef7c8c28ba7c6b2ecc6731
2022-12-05 00:51:50 UTC to 2022-11-15 20:43:30 UTC

- Document that type parameter `Self` is unsized by default (rust-lang/reference#1285)
- replace `crateid` term with `crate_name` (rust-lang/reference#1310)
- Document native library modifier `verbatim` (rust-lang/reference#1299)
- Update literal suffix docs for rust-lang#102944 (rust-lang/reference#1305)
- update aliasing rules section of the reference (rust-lang/reference#1290)
- Document RFC 2867: instruction_set attribute (rust-lang/reference#1253)
- Fix a minor typo in the "Higher-ranked trait bounds" section (rust-lang/reference#1288)
- Disallow newline directly following `//` (rust-lang/reference#1294)
- Add an anchor to the "forwarding macro fragments" paragraph (rust-lang/reference#1300)

## rust-lang/rust-by-example

5 commits in 2b15c0abf2bada6e00553814336bc3e2d8399097..a9869b4a3c4cac3bc6099b41f088679e268400b8
2022-11-27 19:03:05 UTC to 2022-11-11 18:54:53 UTC

- Migrate from highfive to triagebot (rust-lang/rust-by-example#1647)
- Simpler version of the read_lines script. (rust-lang/rust-by-example#1641)
- Fix comment in "Formatted print" example code (rust-lang/rust-by-example#1638)
- Added a missing backtick in a comment in chapter 15.4. (rust-lang/rust-by-example#1642)
- Clarify the confusing closure example rust-lang#1611 (rust-lang/rust-by-example#1643)

## rust-lang/rustc-dev-guide

13 commits in d0dc6c97a6486f68bac782fff135086eae6d77ec..e269950a57fa6fcda356426545fb5aa3691a7ced
2022-12-03 23:09:24 UTC to 2022-11-08 21:35:38 UTC

- Remove duplicate paragraph (rust-lang/rustc-dev-guide#1523)
- clarify subtree tool policy (rust-lang/rustc-dev-guide#1518)
- Typo (rust-lang/rustc-dev-guide#1520)
- Link directly to the section on `--keep-stage` (rust-lang/rustc-dev-guide#1515)
- do an actual link to detect if it breaks in future (rust-lang/rustc-dev-guide#1517)
- Triage some date-check items (rust-lang/rustc-dev-guide#1513)
- Update path for `try_mark_green` implementation (rust-lang/rustc-dev-guide#1512)
- Fix a broken design docs link about unused substs bug (rust-lang/rustc-dev-guide#1511)
- updating-llvm: keep a calm tone (rust-lang/rustc-dev-guide#1449)
- date-check: updating-llvm (rust-lang/rustc-dev-guide#1424)
- rewrite the section about regions in the trait solver (rust-lang/rustc-dev-guide#1508)
- Consistent ordered list indexing (rust-lang/rustc-dev-guide#1509)
- Document multiple alternative suggestions on diagnostic structs (rust-lang/rustc-dev-guide#1486)
matthiaskrgr added a commit to matthiaskrgr/rust that referenced this pull request Dec 5, 2022
Update books

## rust-lang/book

1 commits in 3f64052c048c6def93b94a2b514ee88bba918744..a60f4316ec923a5ac2ed6a2eba6960edb832d855
2022-11-16 15:07:18 UTC to 2022-11-16 15:07:18 UTC

- Fix Install MdBook command (rust-lang/book#3424)

## rust-embedded/book

4 commits in c533348edd69f11a8f4225d633a05d7093fddbf3..19f798d448835a4888e3b3eae7fe69f1d61d8681
2022-11-17 15:08:11 UTC to 2022-11-08 23:27:57 UTC

- start/hardware.md: Fix typo (rust-embedded/book#336)
- doc: Fix `arm-none-eabi-gdb` installation instruction for Fedora 27 or newer to just use `gdb` (rust-embedded/book#335)
- Update singletons.md (rust-embedded/book#334)
- Remove incorrect claim HashMap is avail in no_std (rust-embedded/book#333)

## rust-lang/nomicon

2 commits in 05532356e7a4dbea2330aabb77611f5179493bb8..ae406aa5287a9e025abb72343aaceec98458c117
2022-11-21 22:48:20 UTC to 2022-11-15 00:29:20 UTC

- Improve chapter about `Vec<T>` (rust-lang/nomicon#381)
- Grammar change for 3.4: Limits of Lifetimes (lifetime-mismatch.md) (rust-lang/nomicon#386)

## rust-lang/reference

9 commits in 9f0cc13ffcd27c1fbe1ab766a9491e15ddcf4d19..3ae62681ff236d5528ef7c8c28ba7c6b2ecc6731
2022-12-05 00:51:50 UTC to 2022-11-15 20:43:30 UTC

- Document that type parameter `Self` is unsized by default (rust-lang/reference#1285)
- replace `crateid` term with `crate_name` (rust-lang/reference#1310)
- Document native library modifier `verbatim` (rust-lang/reference#1299)
- Update literal suffix docs for rust-lang#102944 (rust-lang/reference#1305)
- update aliasing rules section of the reference (rust-lang/reference#1290)
- Document RFC 2867: instruction_set attribute (rust-lang/reference#1253)
- Fix a minor typo in the "Higher-ranked trait bounds" section (rust-lang/reference#1288)
- Disallow newline directly following `//` (rust-lang/reference#1294)
- Add an anchor to the "forwarding macro fragments" paragraph (rust-lang/reference#1300)

## rust-lang/rust-by-example

5 commits in 2b15c0abf2bada6e00553814336bc3e2d8399097..a9869b4a3c4cac3bc6099b41f088679e268400b8
2022-11-27 19:03:05 UTC to 2022-11-11 18:54:53 UTC

- Migrate from highfive to triagebot (rust-lang/rust-by-example#1647)
- Simpler version of the read_lines script. (rust-lang/rust-by-example#1641)
- Fix comment in "Formatted print" example code (rust-lang/rust-by-example#1638)
- Added a missing backtick in a comment in chapter 15.4. (rust-lang/rust-by-example#1642)
- Clarify the confusing closure example rust-lang#1611 (rust-lang/rust-by-example#1643)

## rust-lang/rustc-dev-guide

13 commits in d0dc6c97a6486f68bac782fff135086eae6d77ec..e269950a57fa6fcda356426545fb5aa3691a7ced
2022-12-03 23:09:24 UTC to 2022-11-08 21:35:38 UTC

- Remove duplicate paragraph (rust-lang/rustc-dev-guide#1523)
- clarify subtree tool policy (rust-lang/rustc-dev-guide#1518)
- Typo (rust-lang/rustc-dev-guide#1520)
- Link directly to the section on `--keep-stage` (rust-lang/rustc-dev-guide#1515)
- do an actual link to detect if it breaks in future (rust-lang/rustc-dev-guide#1517)
- Triage some date-check items (rust-lang/rustc-dev-guide#1513)
- Update path for `try_mark_green` implementation (rust-lang/rustc-dev-guide#1512)
- Fix a broken design docs link about unused substs bug (rust-lang/rustc-dev-guide#1511)
- updating-llvm: keep a calm tone (rust-lang/rustc-dev-guide#1449)
- date-check: updating-llvm (rust-lang/rustc-dev-guide#1424)
- rewrite the section about regions in the trait solver (rust-lang/rustc-dev-guide#1508)
- Consistent ordered list indexing (rust-lang/rustc-dev-guide#1509)
- Document multiple alternative suggestions on diagnostic structs (rust-lang/rustc-dev-guide#1486)
matthiaskrgr added a commit to matthiaskrgr/rust that referenced this pull request Dec 5, 2022
Update books

## rust-lang/book

1 commits in 3f64052c048c6def93b94a2b514ee88bba918744..a60f4316ec923a5ac2ed6a2eba6960edb832d855
2022-11-16 15:07:18 UTC to 2022-11-16 15:07:18 UTC

- Fix Install MdBook command (rust-lang/book#3424)

## rust-embedded/book

4 commits in c533348edd69f11a8f4225d633a05d7093fddbf3..19f798d448835a4888e3b3eae7fe69f1d61d8681
2022-11-17 15:08:11 UTC to 2022-11-08 23:27:57 UTC

- start/hardware.md: Fix typo (rust-embedded/book#336)
- doc: Fix `arm-none-eabi-gdb` installation instruction for Fedora 27 or newer to just use `gdb` (rust-embedded/book#335)
- Update singletons.md (rust-embedded/book#334)
- Remove incorrect claim HashMap is avail in no_std (rust-embedded/book#333)

## rust-lang/nomicon

2 commits in 05532356e7a4dbea2330aabb77611f5179493bb8..ae406aa5287a9e025abb72343aaceec98458c117
2022-11-21 22:48:20 UTC to 2022-11-15 00:29:20 UTC

- Improve chapter about `Vec<T>` (rust-lang/nomicon#381)
- Grammar change for 3.4: Limits of Lifetimes (lifetime-mismatch.md) (rust-lang/nomicon#386)

## rust-lang/reference

9 commits in 9f0cc13ffcd27c1fbe1ab766a9491e15ddcf4d19..3ae62681ff236d5528ef7c8c28ba7c6b2ecc6731
2022-12-05 00:51:50 UTC to 2022-11-15 20:43:30 UTC

- Document that type parameter `Self` is unsized by default (rust-lang/reference#1285)
- replace `crateid` term with `crate_name` (rust-lang/reference#1310)
- Document native library modifier `verbatim` (rust-lang/reference#1299)
- Update literal suffix docs for rust-lang#102944 (rust-lang/reference#1305)
- update aliasing rules section of the reference (rust-lang/reference#1290)
- Document RFC 2867: instruction_set attribute (rust-lang/reference#1253)
- Fix a minor typo in the "Higher-ranked trait bounds" section (rust-lang/reference#1288)
- Disallow newline directly following `//` (rust-lang/reference#1294)
- Add an anchor to the "forwarding macro fragments" paragraph (rust-lang/reference#1300)

## rust-lang/rust-by-example

5 commits in 2b15c0abf2bada6e00553814336bc3e2d8399097..a9869b4a3c4cac3bc6099b41f088679e268400b8
2022-11-27 19:03:05 UTC to 2022-11-11 18:54:53 UTC

- Migrate from highfive to triagebot (rust-lang/rust-by-example#1647)
- Simpler version of the read_lines script. (rust-lang/rust-by-example#1641)
- Fix comment in "Formatted print" example code (rust-lang/rust-by-example#1638)
- Added a missing backtick in a comment in chapter 15.4. (rust-lang/rust-by-example#1642)
- Clarify the confusing closure example rust-lang#1611 (rust-lang/rust-by-example#1643)

## rust-lang/rustc-dev-guide

13 commits in d0dc6c97a6486f68bac782fff135086eae6d77ec..e269950a57fa6fcda356426545fb5aa3691a7ced
2022-12-03 23:09:24 UTC to 2022-11-08 21:35:38 UTC

- Remove duplicate paragraph (rust-lang/rustc-dev-guide#1523)
- clarify subtree tool policy (rust-lang/rustc-dev-guide#1518)
- Typo (rust-lang/rustc-dev-guide#1520)
- Link directly to the section on `--keep-stage` (rust-lang/rustc-dev-guide#1515)
- do an actual link to detect if it breaks in future (rust-lang/rustc-dev-guide#1517)
- Triage some date-check items (rust-lang/rustc-dev-guide#1513)
- Update path for `try_mark_green` implementation (rust-lang/rustc-dev-guide#1512)
- Fix a broken design docs link about unused substs bug (rust-lang/rustc-dev-guide#1511)
- updating-llvm: keep a calm tone (rust-lang/rustc-dev-guide#1449)
- date-check: updating-llvm (rust-lang/rustc-dev-guide#1424)
- rewrite the section about regions in the trait solver (rust-lang/rustc-dev-guide#1508)
- Consistent ordered list indexing (rust-lang/rustc-dev-guide#1509)
- Document multiple alternative suggestions on diagnostic structs (rust-lang/rustc-dev-guide#1486)
JohnTitor pushed a commit to JohnTitor/rust that referenced this pull request Dec 6, 2022
Update books

## rust-lang/book

1 commits in 3f64052c048c6def93b94a2b514ee88bba918744..a60f4316ec923a5ac2ed6a2eba6960edb832d855
2022-11-16 15:07:18 UTC to 2022-11-16 15:07:18 UTC

- Fix Install MdBook command (rust-lang/book#3424)

## rust-embedded/book

4 commits in c533348edd69f11a8f4225d633a05d7093fddbf3..19f798d448835a4888e3b3eae7fe69f1d61d8681
2022-11-17 15:08:11 UTC to 2022-11-08 23:27:57 UTC

- start/hardware.md: Fix typo (rust-embedded/book#336)
- doc: Fix `arm-none-eabi-gdb` installation instruction for Fedora 27 or newer to just use `gdb` (rust-embedded/book#335)
- Update singletons.md (rust-embedded/book#334)
- Remove incorrect claim HashMap is avail in no_std (rust-embedded/book#333)

## rust-lang/nomicon

2 commits in 05532356e7a4dbea2330aabb77611f5179493bb8..ae406aa5287a9e025abb72343aaceec98458c117
2022-11-21 22:48:20 UTC to 2022-11-15 00:29:20 UTC

- Improve chapter about `Vec<T>` (rust-lang/nomicon#381)
- Grammar change for 3.4: Limits of Lifetimes (lifetime-mismatch.md) (rust-lang/nomicon#386)

## rust-lang/reference

9 commits in 9f0cc13ffcd27c1fbe1ab766a9491e15ddcf4d19..3ae62681ff236d5528ef7c8c28ba7c6b2ecc6731
2022-12-05 00:51:50 UTC to 2022-11-15 20:43:30 UTC

- Document that type parameter `Self` is unsized by default (rust-lang/reference#1285)
- replace `crateid` term with `crate_name` (rust-lang/reference#1310)
- Document native library modifier `verbatim` (rust-lang/reference#1299)
- Update literal suffix docs for rust-lang#102944 (rust-lang/reference#1305)
- update aliasing rules section of the reference (rust-lang/reference#1290)
- Document RFC 2867: instruction_set attribute (rust-lang/reference#1253)
- Fix a minor typo in the "Higher-ranked trait bounds" section (rust-lang/reference#1288)
- Disallow newline directly following `//` (rust-lang/reference#1294)
- Add an anchor to the "forwarding macro fragments" paragraph (rust-lang/reference#1300)

## rust-lang/rust-by-example

5 commits in 2b15c0abf2bada6e00553814336bc3e2d8399097..a9869b4a3c4cac3bc6099b41f088679e268400b8
2022-11-27 19:03:05 UTC to 2022-11-11 18:54:53 UTC

- Migrate from highfive to triagebot (rust-lang/rust-by-example#1647)
- Simpler version of the read_lines script. (rust-lang/rust-by-example#1641)
- Fix comment in "Formatted print" example code (rust-lang/rust-by-example#1638)
- Added a missing backtick in a comment in chapter 15.4. (rust-lang/rust-by-example#1642)
- Clarify the confusing closure example rust-lang#1611 (rust-lang/rust-by-example#1643)

## rust-lang/rustc-dev-guide

13 commits in d0dc6c97a6486f68bac782fff135086eae6d77ec..e269950a57fa6fcda356426545fb5aa3691a7ced
2022-12-03 23:09:24 UTC to 2022-11-08 21:35:38 UTC

- Remove duplicate paragraph (rust-lang/rustc-dev-guide#1523)
- clarify subtree tool policy (rust-lang/rustc-dev-guide#1518)
- Typo (rust-lang/rustc-dev-guide#1520)
- Link directly to the section on `--keep-stage` (rust-lang/rustc-dev-guide#1515)
- do an actual link to detect if it breaks in future (rust-lang/rustc-dev-guide#1517)
- Triage some date-check items (rust-lang/rustc-dev-guide#1513)
- Update path for `try_mark_green` implementation (rust-lang/rustc-dev-guide#1512)
- Fix a broken design docs link about unused substs bug (rust-lang/rustc-dev-guide#1511)
- updating-llvm: keep a calm tone (rust-lang/rustc-dev-guide#1449)
- date-check: updating-llvm (rust-lang/rustc-dev-guide#1424)
- rewrite the section about regions in the trait solver (rust-lang/rustc-dev-guide#1508)
- Consistent ordered list indexing (rust-lang/rustc-dev-guide#1509)
- Document multiple alternative suggestions on diagnostic structs (rust-lang/rustc-dev-guide#1486)
@apiraino apiraino removed the to-announce Announce this issue on triage meeting label Jan 5, 2023
Aaron1011 pushed a commit to Aaron1011/rust that referenced this pull request Jan 6, 2023
…, r=petrochenkov

Use `token::Lit` in `ast::ExprKind::Lit`.

Instead of `ast::Lit`.

Literal lowering now happens at two different times. Expression literals are lowered when HIR is crated. Attribute literals are lowered during parsing.

r? `@petrochenkov`
calebcartwright pushed a commit to calebcartwright/rustfmt that referenced this pull request Jan 24, 2023
Instead of `ast::Lit`.

Literal lowering now happens at two different times. Expression literals
are lowered when HIR is crated. Attribute literals are lowered during
parsing.

This commit changes the language very slightly. Some programs that used
to not compile now will compile. This is because some invalid literals
that are removed by `cfg` or attribute macros will no longer trigger
errors. See this comment for more details:
rust-lang/rust#102944 (comment)
wip-sync pushed a commit to NetBSD/pkgsrc-wip that referenced this pull request Jan 27, 2023
Pkgsrc changes:
 * Adjust patches and cargo checksums to new versions,
   but also one strange "mips" conditional.

Upstream changes:

Version 1.67.0 (2023-01-26)
==========================

Language
--------

- [Make `Sized` predicates coinductive, allowing cycles.]
  (rust-lang/rust#100386)
- [`#[must_use]` annotations on `async fn` also affect the
  `Future::Output`.] (rust-lang/rust#100633)
- [Elaborate supertrait obligations when deducing closure signatures.]
  (rust-lang/rust#101834)
- [Invalid literals are no longer an error under `cfg(FALSE)`.]
  (rust-lang/rust#102944)
- [Unreserve braced enum variants in value namespace.]
  (rust-lang/rust#103578)

Compiler
--------

- [Enable varargs support for calling conventions other than `C`
  or `cdecl`.] (rust-lang/rust#97971)
- [Add new MIR constant propagation based on dataflow analysis.]
  (rust-lang/rust#101168)
- [Optimize field ordering by grouping m\*2^n-sized fields with
  equivalently aligned ones.] (rust-lang/rust#102750)
- [Stabilize native library modifier `verbatim`.]
  (rust-lang/rust#104360)

Added and removed targets:

- [Add a tier 3 target for PowerPC on AIX]
  (rust-lang/rust#102293), `powerpc64-ibm-aix`.
- [Add a tier 3 target for the Sony PlayStation 1]
  (rust-lang/rust#102689), `mipsel-sony-psx`.
- [Add tier 3 `no_std` targets for the QNX Neutrino RTOS]
  (rust-lang/rust#102701),
  `aarch64-unknown-nto-qnx710` and `x86_64-pc-nto-qnx710`.
- [Remove tier 3 `linuxkernel` targets]
  (rust-lang/rust#104015) (not used by the
  actual kernel).

Refer to Rust's [platform support page][platform-support-doc]
for more information on Rust's tiered platform support.

Libraries
---------

- [Merge `crossbeam-channel` into `std::sync::mpsc`.]
  (rust-lang/rust#93563)
- [Fix inconsistent rounding of 0.5 when formatted to 0 decimal places.]
  (rust-lang/rust#102935)
- [Derive `Eq` and `Hash` for `ControlFlow`.]
  (rust-lang/rust#103084)
- [Don't build `compiler_builtins` with `-C panic=abort`.]
  (rust-lang/rust#103786)

Stabilized APIs
---------------

- [`{integer}::checked_ilog`]
  (https://doc.rust-lang.org/stable/std/primitive.i32.html#method.checked_ilog)
- [`{integer}::checked_ilog2`]
  (https://doc.rust-lang.org/stable/std/primitive.i32.html#method.checked_ilog2)
- [`{integer}::checked_ilog10`]
  (https://doc.rust-lang.org/stable/std/primitive.i32.html#method.checked_ilog10)
- [`{integer}::ilog`]
  (https://doc.rust-lang.org/stable/std/primitive.i32.html#method.ilog)
- [`{integer}::ilog2`]
  (https://doc.rust-lang.org/stable/std/primitive.i32.html#method.ilog2)
- [`{integer}::ilog10`]
  (https://doc.rust-lang.org/stable/std/primitive.i32.html#method.ilog10)
- [`NonZeroU*::ilog2`]
  (https://doc.rust-lang.org/stable/std/num/struct.NonZeroU32.html#method.ilog2)
- [`NonZeroU*::ilog10`]
  (https://doc.rust-lang.org/stable/std/num/struct.NonZeroU32.html#method.ilog10)
- [`NonZero*::BITS`]
  (https://doc.rust-lang.org/stable/std/num/struct.NonZeroU32.html#associatedconstant.BITS)

These APIs are now stable in const contexts:

- [`char::from_u32`]
  (https://doc.rust-lang.org/stable/std/primitive.char.html#method.from_u32)
- [`char::from_digit`]
  (https://doc.rust-lang.org/stable/std/primitive.char.html#method.from_digit)
- [`char::to_digit`]
  (https://doc.rust-lang.org/stable/std/primitive.char.html#method.to_digit)
- [`core::char::from_u32`]
  (https://doc.rust-lang.org/stable/core/char/fn.from_u32.html)
- [`core::char::from_digit`]
  (https://doc.rust-lang.org/stable/core/char/fn.from_digit.html)

Compatibility Notes
-------------------

- [The layout of `repr(Rust)` types now groups m\*2^n-sized fields
  with equivalently aligned ones.]
  (rust-lang/rust#102750) This is intended
  to be an optimization, but it is also known to increase type
  sizes in a few cases for the placement of enum tags. As a reminder,
  the layout of `repr(Rust)` types is an implementation detail,
  subject to change.
- [0.5 now rounds to 0 when formatted to 0 decimal places.]
  (rust-lang/rust#102935)
  This makes it consistent with the rest of floating point formatting that
  rounds ties toward even digits.
- [Chains of `&&` and `||` will now drop temporaries from their
  sub-expressions in evaluation order, left-to-right.]
  (rust-lang/rust#103293) Previously, it
  was "twisted" such that the _first_ expression dropped its
  temporaries _last_, after all of the other expressions dropped
  in order.
- [Underscore suffixes on string literals are now a hard error.]
  (rust-lang/rust#103914)
  This has been a future-compatibility warning since 1.20.0.
- [Stop passing `-export-dynamic` to `wasm-ld`.]
  (rust-lang/rust#105405)
- [`main` is now mangled as `__main_void` on `wasm32-wasi`.]
  (rust-lang/rust#105468)
- [Cargo now emits an error if there are multiple registries in
  the configuration with the same index URL.]
  (rust-lang/cargo#10592)

Internal Changes
----------------

These changes do not affect any public interfaces of Rust, but they
represent significant improvements to the performance or internals
of rustc and related tools.

- [Rewrite LLVM's archive writer in Rust.]
  (rust-lang/rust#97485)
netbsd-srcmastr pushed a commit to NetBSD/pkgsrc that referenced this pull request Apr 8, 2023
Pkgsrc changes:
 * Adjust patches (add & remove) and cargo checksums to new versions.
 * It's conceivable that the workaround for LLVM based NetBSD works
   even less in this version (ref. PKGSRC_HAVE_LIBCPP not having a
   corresponding patch anymore).

Upstream changes:

Version 1.68.2 (2023-03-28)
===========================

- [Update the GitHub RSA host key bundled within Cargo]
  (rust-lang/cargo#11883).
  The key was [rotated by GitHub]
  (https://github.blog/2023-03-23-we-updated-our-rsa-ssh-host-key/)
  on 2023-03-24 after the old one leaked.
- [Mark the old GitHub RSA host key as revoked]
  (rust-lang/cargo#11889).
  This will prevent Cargo from accepting the leaked key even when
  trusted by the system.
- [Add support for `@revoked` and a better error message for
  `@cert-authority` in Cargo's SSH host key verification]
  (rust-lang/cargo#11635)

Version 1.68.1 (2023-03-23)
===========================

- [Fix miscompilation in produced Windows MSVC artifacts]
  (rust-lang/rust#109094)
  This was introduced by enabling ThinLTO for the distributed rustc
  which led to miscompilations in the resulting binary. Currently
  this is believed to be limited to the -Zdylib-lto flag used for
  rustc compilation, rather than a general bug in ThinLTO, so only
  rustc artifacts should be affected.
- [Fix --enable-local-rust builds]
  (rust-lang/rust#109111)
- [Treat `$prefix-clang` as `clang` in linker detection code]
  (rust-lang/rust#109156)
- [Fix panic in compiler code]
  (rust-lang/rust#108162)

Version 1.68.0 (2023-03-09)
===========================

Language
--------

- [Stabilize default_alloc_error_handler]
  (rust-lang/rust#102318)
  This allows usage of `alloc` on stable without requiring the
  definition of a handler for allocation failure. Defining custom
  handlers is still unstable.
- [Stabilize `efiapi` calling convention.]
  (rust-lang/rust#105795)
- [Remove implicit promotion for types with drop glue]
  (rust-lang/rust#105085)

Compiler
--------

- [Change `bindings_with_variant_name` to deny-by-default]
  (rust-lang/rust#104154)
- [Allow .. to be parsed as let initializer]
  (rust-lang/rust#105701)
- [Add `armv7-sony-vita-newlibeabihf` as a tier 3 target]
  (rust-lang/rust#105712)
- [Always check alignment during compile-time const evaluation]
  (rust-lang/rust#104616)
- [Disable "split dwarf inlining" by default.]
  (rust-lang/rust#106709)
- [Add vendor to Fuchsia's target triple]
  (rust-lang/rust#106429)
- [Enable sanitizers for s390x-linux]
  (rust-lang/rust#107127)

Libraries
---------

- [Loosen the bound on the Debug implementation of Weak.]
  (rust-lang/rust#90291)
- [Make `std::task::Context` !Send and !Sync]
  (rust-lang/rust#95985)
- [PhantomData layout guarantees]
  (rust-lang/rust#104081)
- [Don't derive Debug for `OnceWith` & `RepeatWith`]
  (rust-lang/rust#104163)
- [Implement DerefMut for PathBuf]
  (rust-lang/rust#105018)
- [Add O(1) `Vec -> VecDeque` conversion guarantee]
  (rust-lang/rust#105128)
- [Leak amplification for peek_mut() to ensure BinaryHeap's invariant
  is always met]
  (rust-lang/rust#105851)

Stabilized APIs
---------------

- [`{core,std}::pin::pin!`]
  (https://doc.rust-lang.org/stable/std/pin/macro.pin.html)
- [`impl From<bool> for {f32,f64}`]
  (https://doc.rust-lang.org/stable/std/primitive.f32.html#impl-From%3Cbool%3E-for-f32)
- [`std::path::MAIN_SEPARATOR_STR`]
  (https://doc.rust-lang.org/stable/std/path/constant.MAIN_SEPARATOR_STR.html)
- [`impl DerefMut for PathBuf`]
  (https://doc.rust-lang.org/stable/std/path/struct.PathBuf.html#impl-DerefMut-for-PathBuf)

These APIs are now stable in const contexts:

- [`VecDeque::new`]
  (https://doc.rust-lang.org/stable/std/collections/struct.VecDeque.html#method.new)

Cargo
-----

- [Stabilize sparse registry support for crates.io]
  (rust-lang/cargo#11224)
- [`cargo build --verbose` tells you more about why it recompiles.]
  (rust-lang/cargo#11407)
- [Show progress of crates.io index update even `net.git-fetch-with-cli`
  option enabled]
  (rust-lang/cargo#11579)

Misc
----

Compatibility Notes
-------------------

- [Add `SEMICOLON_IN_EXPRESSIONS_FROM_MACROS` to future-incompat report]
  (rust-lang/rust#103418)
- [Only specify `--target` by default for `-Zgcc-ld=lld` on wasm]
  (rust-lang/rust#101792)
- [Bump `IMPLIED_BOUNDS_ENTAILMENT` to Deny + ReportNow]
  (rust-lang/rust#106465)
- [`std::task::Context` no longer implements Send and Sync]
  (rust-lang/rust#95985)

nternal Changes
----------------

These changes do not affect any public interfaces of Rust, but they represent
significant improvements to the performance or internals of rustc and related
tools.

- [Encode spans relative to the enclosing item]
  (rust-lang/rust#84762)
- [Don't normalize in AstConv]
  (rust-lang/rust#101947)
- [Find the right lower bound region in the scenario of partial order relations]
  (rust-lang/rust#104765)
- [Fix impl block in const expr]
  (rust-lang/rust#104889)
- [Check ADT fields for copy implementations considering regions]
  (rust-lang/rust#105102)
- [rustdoc: simplify JS search routine by not messing with lev distance]
  (rust-lang/rust#105796)
- [Enable ThinLTO for rustc on `x86_64-pc-windows-msvc`]
  (rust-lang/rust#103591)
- [Enable ThinLTO for rustc on `x86_64-apple-darwin`]
  (rust-lang/rust#103647)

Version 1.67.0 (2023-01-26)
==========================

Language
--------

- [Make `Sized` predicates coinductive, allowing cycles.]
  (rust-lang/rust#100386)
- [`#[must_use]` annotations on `async fn` also affect the
  `Future::Output`.] (rust-lang/rust#100633)
- [Elaborate supertrait obligations when deducing closure signatures.]
  (rust-lang/rust#101834)
- [Invalid literals are no longer an error under `cfg(FALSE)`.]
  (rust-lang/rust#102944)
- [Unreserve braced enum variants in value namespace.]
  (rust-lang/rust#103578)

Compiler
--------

- [Enable varargs support for calling conventions other than `C`
  or `cdecl`.] (rust-lang/rust#97971)
- [Add new MIR constant propagation based on dataflow analysis.]
  (rust-lang/rust#101168)
- [Optimize field ordering by grouping m\*2^n-sized fields with
  equivalently aligned ones.] (rust-lang/rust#102750)
- [Stabilize native library modifier `verbatim`.]
  (rust-lang/rust#104360)

Added and removed targets:

- [Add a tier 3 target for PowerPC on AIX]
  (rust-lang/rust#102293), `powerpc64-ibm-aix`.
- [Add a tier 3 target for the Sony PlayStation 1]
  (rust-lang/rust#102689), `mipsel-sony-psx`.
- [Add tier 3 `no_std` targets for the QNX Neutrino RTOS]
  (rust-lang/rust#102701),
  `aarch64-unknown-nto-qnx710` and `x86_64-pc-nto-qnx710`.
- [Remove tier 3 `linuxkernel` targets]
  (rust-lang/rust#104015) (not used by the
  actual kernel).

Refer to Rust's [platform support page][platform-support-doc]
for more information on Rust's tiered platform support.

Libraries
---------

- [Merge `crossbeam-channel` into `std::sync::mpsc`.]
  (rust-lang/rust#93563)
- [Fix inconsistent rounding of 0.5 when formatted to 0 decimal places.]
  (rust-lang/rust#102935)
- [Derive `Eq` and `Hash` for `ControlFlow`.]
  (rust-lang/rust#103084)
- [Don't build `compiler_builtins` with `-C panic=abort`.]
  (rust-lang/rust#103786)

Stabilized APIs
---------------

- [`{integer}::checked_ilog`]
  (https://doc.rust-lang.org/stable/std/primitive.i32.html#method.checked_ilog)
- [`{integer}::checked_ilog2`]
  (https://doc.rust-lang.org/stable/std/primitive.i32.html#method.checked_ilog2)
- [`{integer}::checked_ilog10`]
  (https://doc.rust-lang.org/stable/std/primitive.i32.html#method.checked_ilog10)
- [`{integer}::ilog`]
  (https://doc.rust-lang.org/stable/std/primitive.i32.html#method.ilog)
- [`{integer}::ilog2`]
  (https://doc.rust-lang.org/stable/std/primitive.i32.html#method.ilog2)
- [`{integer}::ilog10`]
  (https://doc.rust-lang.org/stable/std/primitive.i32.html#method.ilog10)
- [`NonZeroU*::ilog2`]
  (https://doc.rust-lang.org/stable/std/num/struct.NonZeroU32.html#method.ilog2)
- [`NonZeroU*::ilog10`]
  (https://doc.rust-lang.org/stable/std/num/struct.NonZeroU32.html#method.ilog10)
- [`NonZero*::BITS`]
  (https://doc.rust-lang.org/stable/std/num/struct.NonZeroU32.html#associatedconstant.BITS)

These APIs are now stable in const contexts:

- [`char::from_u32`]
  (https://doc.rust-lang.org/stable/std/primitive.char.html#method.from_u32)
- [`char::from_digit`]
  (https://doc.rust-lang.org/stable/std/primitive.char.html#method.from_digit)
- [`char::to_digit`]
  (https://doc.rust-lang.org/stable/std/primitive.char.html#method.to_digit)
- [`core::char::from_u32`]
  (https://doc.rust-lang.org/stable/core/char/fn.from_u32.html)
- [`core::char::from_digit`]
  (https://doc.rust-lang.org/stable/core/char/fn.from_digit.html)

Compatibility Notes
-------------------

- [The layout of `repr(Rust)` types now groups m\*2^n-sized fields
  with equivalently aligned ones.]
  (rust-lang/rust#102750) This is intended
  to be an optimization, but it is also known to increase type
  sizes in a few cases for the placement of enum tags. As a reminder,
  the layout of `repr(Rust)` types is an implementation detail,
  subject to change.
- [0.5 now rounds to 0 when formatted to 0 decimal places.]
  (rust-lang/rust#102935)
  This makes it consistent with the rest of floating point formatting that
  rounds ties toward even digits.
- [Chains of `&&` and `||` will now drop temporaries from their
  sub-expressions in evaluation order, left-to-right.]
  (rust-lang/rust#103293) Previously, it
  was "twisted" such that the _first_ expression dropped its
  temporaries _last_, after all of the other expressions dropped
  in order.
- [Underscore suffixes on string literals are now a hard error.]
  (rust-lang/rust#103914)
  This has been a future-compatibility warning since 1.20.0.
- [Stop passing `-export-dynamic` to `wasm-ld`.]
  (rust-lang/rust#105405)
- [`main` is now mangled as `__main_void` on `wasm32-wasi`.]
  (rust-lang/rust#105468)
- [Cargo now emits an error if there are multiple registries in
  the configuration with the same index URL.]
  (rust-lang/cargo#10592)

Internal Changes
----------------

These changes do not affect any public interfaces of Rust, but they
represent significant improvements to the performance or internals
of rustc and related tools.

- [Rewrite LLVM's archive writer in Rust.]
  (rust-lang/rust#97485)
nnethercote added a commit to nnethercote/rust that referenced this pull request Dec 11, 2023
Currently string literals are unescaped twice.
- Once during lexing in `cook_quoted`/`cook_c_string`/`cook_common`.
  This one just checks for errors.
- Again in `LitKind::from_token_lit`, which is mostly called when
  lowering AST to HIR, but also in a few other places during expansion.
  This one actually constructs the unescaped string. It also has error
  checking code, but that code handling the error cases is actually dead
  (and has several bugs) because the check during lexing catches all
  errors!

This commit removes the checking during lexing, and fixes up
`LitKind::from_token_lit` so it properly does both checking and
construction. This is a language change: some programs now compile that
previously did not. For example, it is now possible for macros to be
passed "invalid" string literals like "\a\b\c". This is a continuation
of a trend of delaying semantic error checking of literals to after
expansion, e.g. rust-lang#102944 did this for some cases for numeric literals,
and the detection of NUL chars in C string literals is already delayed
in this way.

XXX: have Session::report_lit_errors?
XXX: have LitKind::from_token_lit so you don't need the .0?

Things to note:
- `LitError` has a new `EscapeError` variant.
- `LitKind::from_token_lit`'s return value changed, to produce multiple
  errors/warnings, and also to handle lexer warnings. This latter case
  is annoying but necessary to preserve existing warning behaviour.
- `report_lit_error` becomes `report_lit_errors`, in order to handle
  multiple errors in a single string literal.

Notes about test changes:

- `tests/rustdoc-ui/ignore-block-help.rs`: this relies on a parsing
  error occurring. The error present was an unescaping error, which is
  now delayed to after parsing. So the commit changes it to an
  "unterminated character literal" error which continues to occurs
  during parsing.

- Several tests had unescaping errors combined with unterminated literal
  errors. The former are now delayed but the latter remain as lexing
  errors. So the unterminated literal part needed to be split into a
  separate test file otherwise compilation would end before the other
  errors were reported.

- issue-62913.rs: The structure and output changed a bit. Issue rust-lang#62913
  was about an ICE due to an unterminated string literal, so the new
  version should be good enough.

- literals-are-validated-before-expansion.rs: this tests exactly the
  behaviour that has been changed, and so was removed

  XXX: insert a new test covering more of that

- A couple of other test produce the same errors, just in a different
  order.
nnethercote added a commit to nnethercote/rust that referenced this pull request Dec 12, 2023
XXX: need more tests

Currently string literals are unescaped twice.

- Once during lexing in `cook_quoted`/`cook_c_string`/`cook_common`.
  This one just checks for errors.

- Again in `LitKind::from_token_lit`, which is called when lowering AST
  to HIR, and also in a few other places during expansion. This one
  actually constructs the unescaped string. It also has error checking
  code, but that part of the code is actually dead (and has several
  bugs) because the check during lexing catches all errors!

This commit removes the error-check-only unescaping during lexing, and
fixes up `LitKind::from_token_lit` so it properly does both checking and
construction. This is a backwards-compatible language change: some
programs now compile that previously did not. For example, it is now
possible for macros to consume "invalid" string literals like "\a\b\c".
This is a continuation of a trend of delaying semantic error checking of
literals to after expansion:
- rust-lang#102944 did this for some cases for numeric literals
- The detection of NUL chars in C string literals is already delayed in
  this way.

Notes about test changes:

- `tests/rustdoc-ui/ignore-block-help.rs`: this requires a parse
  error occurring. The error used was an unescaping error, which is now
  delayed to after parsing. So the commit changes it to an "unterminated
  character literal" error which still occurs during parsing.

- Several tests had unescaping errors combined with unterminated literal
  errors. The former are now delayed but the latter remain as lexing
  errors. So the unterminated literal part needed to be split into a
  separate test file otherwise compilation would end before the other
  errors were reported.

- issue-62913.rs: The structure and output changed a bit. Issue rust-lang#62913
  was about an ICE due to an unterminated string literal, so the new
  version should be good enough.

- literals-are-validated-before-expansion.rs: this tests exactly the
  behaviour that has been changed, and so was removed

- A couple of other test produce the same errors, just in a different
  order.
nnethercote added a commit to nnethercote/rust that referenced this pull request Dec 12, 2023
Currently string literals are unescaped twice.

- Once during lexing in `cook_quoted`/`cook_c_string`/`cook_common`.
  This one just checks for errors.

- Again in `LitKind::from_token_lit`, which is called when lowering AST
  to HIR, and also in a few other places during expansion. This one
  actually constructs the unescaped string. It also has error checking
  code, but that part of the code is actually dead (and has several
  bugs) because the check during lexing catches all errors!

This commit removes the error-check-only unescaping during lexing, and
fixes up `LitKind::from_token_lit` so it properly does both checking and
construction. This is a backwards-compatible language change: some
programs now compile that previously did not. For example, it is now
possible for macros to consume "invalid" string literals like "\a\b\c".
This is a continuation of a trend of delaying semantic error checking of
literals to after expansion:
- rust-lang#102944 did this for some cases for numeric literals
- The detection of NUL chars in C string literals is already delayed in
  this way.

Notes about test changes:

- `ignore-block-help.rs`: this requires a parse error for the test to
  work. The error used was an unescaping error, which is now delayed to
  after parsing. So the commit changes it to an "unterminated character
  literal" error which still occurs during parsing.

- `tests/ui/lexer/error-stage.rs`: this shows the newly allowed cases,
  due to delayed literal unescaping.

- Several tests had unescaping errors combined with unterminated literal
  errors. The former are now delayed but the latter remain as lexing
  errors. So the unterminated literal part needed to be split into a
  separate test file otherwise compilation would end before the other
  errors were reported.

- issue-62913.rs: The structure and output changed a bit. Issue rust-lang#62913
  was about an ICE due to an unterminated string literal, so the new
  version should be good enough.

- literals-are-validated-before-expansion.rs: this tests exactly the
  behaviour that has been changed, and so was removed

- A couple of other test produce the same errors, just in a different
  order.
nnethercote added a commit to nnethercote/rust that referenced this pull request Dec 12, 2023
Currently string literals are unescaped twice.

- Once during lexing in `cook_quoted`/`cook_c_string`/`cook_common`.
  This one just checks for errors.

- Again in `LitKind::from_token_lit`, which is called when lowering AST
  to HIR, and also in a few other places during expansion. This one
  actually constructs the unescaped string. It also has error checking
  code, but that part of the code is actually dead (and has several
  bugs) because the check during lexing catches all errors!

This commit removes the error-check-only unescaping during lexing, and
fixes up `LitKind::from_token_lit` so it properly does both checking and
construction. This is a backwards-compatible language change: some
programs now compile that previously did not. For example, it is now
possible for macros to consume "invalid" string literals like "\a\b\c".
This is a continuation of a trend of delaying semantic error checking of
literals to after expansion:
- rust-lang#102944 did this for some cases for numeric literals
- The detection of NUL chars in C string literals is already delayed in
  this way.

Notes about test changes:

- `ignore-block-help.rs`: this requires a parse error for the test to
  work. The error used was an unescaping error, which is now delayed to
  after parsing. So the commit changes it to an "unterminated character
  literal" error which still occurs during parsing.

- `tests/ui/lexer/error-stage.rs`: this shows the newly allowed cases,
  due to delayed literal unescaping.

- Several tests had unescaping errors combined with unterminated literal
  errors. The former are now delayed but the latter remain as lexing
  errors. So the unterminated literal part needed to be split into a
  separate test file otherwise compilation would end before the other
  errors were reported.

- issue-62913.rs: The structure and output changed a bit. Issue rust-lang#62913
  was about an ICE due to an unterminated string literal, so the new
  version should be good enough.

- literals-are-validated-before-expansion.rs: this tests exactly the
  behaviour that has been changed, and so was removed

- A couple of other test produce the same errors, just in a different
  order.
nnethercote added a commit to nnethercote/rust that referenced this pull request Dec 18, 2023
Currently string literals are unescaped twice.

- Once during lexing in `cook_quoted`/`cook_c_string`/`cook_common`.
  This one just checks for errors.

- Again in `LitKind::from_token_lit`, which is called when lowering AST
  to HIR, and also in a few other places during expansion. This one
  actually constructs the unescaped string. It also has error checking
  code, but that part of the code is actually dead (and has several
  bugs) because the check during lexing catches all errors!

This commit removes the error-check-only unescaping during lexing, and
fixes up `LitKind::from_token_lit` so it properly does both checking and
construction. This is a backwards-compatible language change: some
programs now compile that previously did not. For example, it is now
possible for macros to consume "invalid" string literals like "\a\b\c".
This is a continuation of a trend of delaying semantic error checking of
literals to after expansion:
- rust-lang#102944 did this for some cases for numeric literals
- The detection of NUL chars in C string literals is already delayed in
  this way.

Notes about test changes:

- `ignore-block-help.rs`: this requires a parse error for the test to
  work. The error used was an unescaping error, which is now delayed to
  after parsing. So the commit changes it to an "unterminated character
  literal" error which still occurs during parsing.

- `tests/ui/lexer/error-stage.rs`: this shows the newly allowed cases,
  due to delayed literal unescaping.

- Several tests had unescaping errors combined with unterminated literal
  errors. The former are now delayed but the latter remain as lexing
  errors. So the unterminated literal part needed to be split into a
  separate test file otherwise compilation would end before the other
  errors were reported.

- issue-62913.rs: The structure and output changed a bit. Issue rust-lang#62913
  was about an ICE due to an unterminated string literal, so the new
  version should be good enough.

- literals-are-validated-before-expansion.rs: this tests exactly the
  behaviour that has been changed, and so was removed

- A couple of other test produce the same errors, just in a different
  order.
nnethercote added a commit to nnethercote/rust that referenced this pull request Dec 18, 2023
Currently string literals are unescaped twice.

- Once during lexing in `cook_quoted`/`cook_c_string`/`cook_common`.
  This one just checks for errors.

- Again in `LitKind::from_token_lit`, which is called when lowering AST
  to HIR, and also in a few other places during expansion. This one
  actually constructs the unescaped string. It also has error checking
  code, but that part of the code is actually dead (and has several
  bugs) because the check during lexing catches all errors!

This commit removes the error-check-only unescaping during lexing, and
fixes up `LitKind::from_token_lit` so it properly does both checking and
construction. This is a backwards-compatible language change: some
programs now compile that previously did not. For example, it is now
possible for macros to consume "invalid" string literals like "\a\b\c".
This is a continuation of a trend of delaying semantic error checking of
literals to after expansion:
- rust-lang#102944 did this for some cases for numeric literals
- The detection of NUL chars in C string literals is already delayed in
  this way.

Notes about test changes:

- `ignore-block-help.rs`: this requires a parse error for the test to
  work. The error used was an unescaping error, which is now delayed to
  after parsing. So the commit changes it to an "unterminated character
  literal" error which still occurs during parsing.

- `tests/ui/lexer/error-stage.rs`: this shows the newly allowed cases,
  due to delayed literal unescaping.

- Several tests had unescaping errors combined with unterminated literal
  errors. The former are now delayed but the latter remain as lexing
  errors. So the unterminated literal part needed to be split into a
  separate test file otherwise compilation would end before the other
  errors were reported.

- issue-62913.rs: The structure and output changed a bit. Issue rust-lang#62913
  was about an ICE due to an unterminated string literal, so the new
  version should be good enough.

- literals-are-validated-before-expansion.rs: this tests exactly the
  behaviour that has been changed, and so was removed

- A couple of other test produce the same errors, just in a different
  order.
nnethercote added a commit to nnethercote/rust that referenced this pull request Dec 19, 2023
Currently string literals are unescaped twice.

- Once during lexing in `cook_quoted`/`cook_c_string`/`cook_common`.
  This one just checks for errors.

- Again in `LitKind::from_token_lit`, which is called when lowering AST
  to HIR, and also in a few other places during expansion. This one
  actually constructs the unescaped string. It also has error checking
  code, but that part of the code is actually dead (and has several
  bugs) because the check during lexing catches all errors!

This commit removes the error-check-only unescaping during lexing, and
fixes up `LitKind::from_token_lit` so it properly does both checking and
construction. This is a backwards-compatible language change: some
programs now compile that previously did not. For example, it is now
possible for macros to consume "invalid" string literals like "\a\b\c".
This is a continuation of a trend of delaying semantic error checking of
literals to after expansion:
- rust-lang#102944 did this for some cases for numeric literals
- The detection of NUL chars in C string literals is already delayed in
  this way.

Notes about test changes:

- `ignore-block-help.rs`: this requires a parse error for the test to
  work. The error used was an unescaping error, which is now delayed to
  after parsing. So the commit changes it to an "unterminated character
  literal" error which still occurs during parsing.

- `tests/ui/lexer/error-stage.rs`: this shows the newly allowed cases,
  due to delayed literal unescaping.

- Several tests had unescaping errors combined with unterminated literal
  errors. The former are now delayed but the latter remain as lexing
  errors. So the unterminated literal part needed to be split into a
  separate test file otherwise compilation would end before the other
  errors were reported.

- issue-62913.rs: The structure and output changed a bit. Issue rust-lang#62913
  was about an ICE due to an unterminated string literal, so the new
  version should be good enough.

- literals-are-validated-before-expansion.rs: this tests exactly the
  behaviour that has been changed, and so was removed

- A couple of other test produce the same errors, just in a different
  order.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
disposition-merge This issue / PR is in PFCP or FCP with a disposition to merge it. finished-final-comment-period The final comment period is finished for this PR / Issue. merged-by-bors This PR was explicitly merged by bors. perf-regression Performance regression. perf-regression-triaged The performance regression has been triaged. S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) T-lang Relevant to the language team, which will review and decide on the PR/issue.
Projects
None yet
Development

Successfully merging this pull request may close these issues.