proc_macro: Avoid cached TokenStream more often

[alexcrichton]

This commit adds even more pessimization to use the cached TokenStream inside
of an AST node. As a reminder the proc_macro API requires taking an arbitrary
AST node and transforming it back into a TokenStream to hand off to a
procedural macro. Such functionality isn't actually implemented in rustc today,
so the way proc_macro works today is that it stringifies an AST node and then
reparses for a list of tokens.

This strategy unfortunately loses all span information, so we try to avoid it
whenever possible. Implemented in #43230 some AST nodes have a TokenStream
cache representing the tokens they were originally parsed from. This
TokenStream cache, however, has turned out to not always reflect the current
state of the item when it's being tokenized. For example #[cfg] processing or
macro expansion could modify the state of an item. Consequently we've seen a
number of bugs (#48644 and #49846) related to using this stale cache.

This commit tweaks the usage of the cached TokenStream to compare it to our
lossy stringification of the token stream. If the tokens that make up the cache
and the stringified token stream are the same then we return the cached version
(which has correct span information). If they differ, however, then we will
return the stringified version as the cache has been invalidated and we just
haven't figured that out.

Closes #48644
Closes #49846

[rust-highfive]

r? @eddyb

(rust_highfive has picked a reviewer for you, use r? to override)

[alexcrichton]

r? @nrc

cc @dtolnay

[alexcrichton]

Note that the real underlying issue here, #43081, I intend to leave open as that'll just get more pressing over time

[nrc]

I don't mind landing this, but it feels like a hack on top of a hack :-( We should definitely be thinking about a long-term solution.

Have you confirmed that the simple equality check works in practice? I could imagine it failing if there has been macro expansion resulting in interpolated tokens or with tokens which are otherwise ignored (comments, whitespace, etc.), etc.

Assuming that it works most of the time and it is thus an improvement, even if imperfect then r=me.

[alexcrichton]

I did some testing locally and it looks like this at least papers over the issue for now. I definitely agree this is piling hacks on hacks :(

The only real way forward that I know though is to take a strategy like syn and refactor the entire AST to have a lossless to_tokens method. That's a pretty significant undertaking though and a pretty massive refactoring in the compiler...

@bors: r=nrc

[bors]

📌 Commit 6d7cfd4 has been approved by nrc