Minor refactorings & implement --pretty expanded,hygiene to see syntax context & gensym information

[huonw]

Different Identifiers and Names can have identical textual representations, but different internal representations, due to the macro hygiene machinery (syntax contexts and gensyms). This provides a way to see these internals by compiling with --pretty expanded,hygiene.

This is useful for debugging & hacking on macros (e.g. diagnosing #15750 likely would've been faster with this functionality).

E.g.

#![feature(macro_rules)]
// minimal junk
#![no_std]

macro_rules! foo {
    ($x: ident) => { y + $x }
}

fn bar() {
    foo!(x)
}

#![feature(macro_rules)]
// minimal junk
#![no_std]


fn bar /* 61#0 */() { y /* 60#2 */ + x /* 58#3 */ }

[huonw]

This isn't as informative as it could be (#16420), but the gensym information is useful and the syntax context stuff is still useful as a basic positive test (i.e. if two identifiers with the same name also have the same syntax context, then they're the same).

[pnkfelix]

(I will review)

[pnkfelix]

@huonw so while I regard this as a net-win (and thus I r-plused it), I idly wonder: What is more important for a person using this -- that the names correspond to the names that would be printed via things like stringify!, with the hygiene information only rendered in a comment? Or would it be better if we tried to maintain an invariant that if the input code compiled successfully, then the output of pretty printing will also compile successfully, which would imply that we should in fact render the hygiene information using a semantically-significant format, such as e.g. re-rendering your example

fn bar_61_0 /* 61#0 */() { y /* 60#2 */ + x /* 58#3 */ }

as this instead:

#![feature(non_ascii_idents)]

fn barʃ61ʃ0() { yʃ60ʃ2 + xʃ58ʃ3 }

(where I have chosen to put a non-ascii character as the marker for where the meta-data starts to reduce the chance that it will end up actually colliding with an actual variable from the source text -- another, more robust option would be to dynamically choose between _, __, ___, etc until we find one that will not yield collisions with the identifiers from the original text.)

[huonw]

Ah, that's an interesting point. I hadn't thought of it, and it does seem like it could be better than just comments.

(On that note, if we want pretty printing to always compile & not change semantics, then --pretty expanded would need to do something like this too.)

[pnkfelix]

Nonetheless we can adopt this PR for now and put my suggestion down as a back burner task

[huonw]

re-r? @pnkfelix

I have just changed the test to avoid checking the exact numbers, since it makes the test very sensitive to compiler internals. I'm not sure if the new implementation of the test is reasonable.