regression: macro expansion/resolution change of some kind

[Mark-Simulacrum]

• https://crater-reports.s3.amazonaws.com/beta-1.76-3/beta-2024-01-09/gh/fishfight.FishFightThePrequel/log.txt

[INFO] [stdout] error[E0424]: expected value, found module `self`
[INFO] [stdout]   --> src/capabilities/weapon.rs:13:23
[INFO] [stdout]    |
[INFO] [stdout] 13 | #[derive(Clone, Copy, CapabilityTrait)]
[INFO] [stdout]    |                       ^^^^^^^^^^^^^^^
[INFO] [stdout]    |                       |
[INFO] [stdout]    |                       `self` value is a keyword only available in methods with a `self` parameter
[INFO] [stdout]    |                       this function doesn't have a `self` parameter
[INFO] [stdout]    |
[INFO] [stdout]    = note: this error originates in the derive macro `CapabilityTrait` (in Nightly builds, run with -Z macro-backtrace for more info)
[INFO] [stdout] help: add a `self` receiver parameter to make the associated `fn` a method
[INFO] [stdout]    |
[INFO] [stdout] 13 | #[derive(Clone, Copy, &self, CapabilityTrait)]

• https://crater-reports.s3.amazonaws.com/beta-1.76-3/beta-2024-01-09/reg/repeated-0.1.2/log.txt
• https://crater-reports.s3.amazonaws.com/beta-1.76-3/beta-2024-01-09/reg/movie-0.1.2/log.txt
  • https://crater-reports.s3.amazonaws.com/beta-1.76-3/beta-2024-01-09/reg/movie_example-0.1.0/log.txt

[INFO] [stdout] error: expected one of `->`, `<`, `where`, or `{`, found `%`
[INFO] [stdout]   --> tests/smoke.rs:5:5
[INFO] [stdout]    |
[INFO] [stdout] 5  | /     repeated!(for x in [0;9;3] {
[INFO] [stdout] 6  | |         fn Welcome_%%x%%() {
[INFO] [stdout] 7  | |             repeated!(for y in [1;%%x%%;2] {
[INFO] [stdout] 8  | |                 println!("From within the macro %%x%%:%%y%%!");
[INFO] [stdout] 9  | |             });
[INFO] [stdout] 10 | |         }
[INFO] [stdout] 11 | |     });
[INFO] [stdout]    | |______^ expected one of `->`, `<`, `where`, or `{`
[INFO] [stdout]    |
[INFO] [stdout]    = note: this error originates in the macro `repeated` (in Nightly builds, run with -Z macro-backtrace for more info)

[compiler-errors]

This regressed in #114571, cc @nnethercote @petrochenkov.

This breakage seems to be acknowledged here: #114571 (comment)

I'm surprised this was just r+'d rather than going through some team consensus or comment period given that it breaks crates in the wild.

[nnethercote]

There are a small number of crates that implement proc macros by calling .to_string() on the input token stream and then doing some half-assed "parsing" (i.e. regex or substring matching) of the output. This is not the right way to do it, because it relies on the output of .to_string() having a particular form, something that isn't guaranteed.

We judged that the number of affected crates was small, and their use was small, and therefore that this was acceptable. I'm happy to listen if others disagree.

[wesleywiser]

Reviewed during T-compiler triage. We agree with @nnethercote and @petrochenkov's judgment that the breakage here is acceptable. In particular, the documentation for impl Display for TokenTree only guarantees that the output can be converted back into a TokenTree not that the output conforms to any particular formatting with regards to whitespace.

Normally, I would suggest we open PRs to the affected crates but all of the crates mentioned in the crater report have been dormant for >3 years. As such, I don't think those PRs would be merged so I'm ok with this shipping as-is in 1.76.

[nnethercote]

Note: in #120220 I augmented the documentation for Token{Stream,Tree}::Display to explicitly warn against the substring matching approach to implementing a proc macro.

@Mark-Simulacrum: In light of all the above discussion, can we close this issue?

[Mark-Simulacrum]

Ah, I want to double check there's a note in the release notes but will close after that.

[Mark-Simulacrum]

Confirmed: https://github.com/rust-lang/rust/pull/120004/files#diff-4a00447b47c36ebd3ded77321deff288e424f4605496ee1199fc9953a20f4473R76

[nnethercote]

@Mark-Simulacrum: there will be a tiny bit more similar breakage in 1.77 due to further tweaks to TokenStream pretty-printing in #120227. See this comment for details about the crater run. (Note: that's the last such change to pretty-printing that I have planned.) So it might be worth adding to the release notes for 1.77 as well?

[Mark-Simulacrum]

Those aren't even in draft yet, but if we pick up on the breakage through crater we'll certainly do so. (Also tagged the PR so that we are more likely to see it while drafting).