Parse tokens, not strings

[alexcrichton]

This commit builds off @mystor's work to finish porting the parser to working over tokens, not strings as it was previously. The tokens here come directly from the proc_macro2 crate which is the intended-to-be-stabilized interface of proc_macro itself.

In the end this should mean that conversion from a TokenStream to a syn AST and back should be a lossless transformation in terms of contextual information like spans. This'll lead to far better error messages from the compiler and such.

This comes with quite heavy changes to the synom crate, including deletion of a number of macros that were no longer uses and aren't as useful in this "lossless" world. Comments are of course always welcome!

Closes #137
Closes #142

[alexcrichton]

Ah and it's worth noting that of course parsing strings is still supported. The TokenStream type can be parsed from a string and then from there we can parse those tokens. It's also worth noting that this currently has what is likely an excessively inefficient representation of the "slice of what to parse", especially around delimited token streams. I was hoping that this could be improved before 0.12 but perhaps after this PR.

[mystor]

Thanks for rebasing this @alexcrichton (And also introducing the Synom trait / rewriting most of it :S which simplifies things a lot).

Ah and it's worth noting that of course parsing strings is still supported. The TokenStream type can be parsed from a string and then from there we can parse those tokens. It's also worth noting that this currently has what is likely an excessively inefficient representation of the "slice of what to parse", especially around delimited token streams. I was hoping that this could be improved before 0.12 but perhaps after this PR.

I have an idea of how to improve this, and fix the None-delimited Sequence and operator associativity problem with Expressions, but I think I'm going to wait until this gets merged to work on it.

Unfortunately, even once proc-macro is in nightly, we'll probably need something like proc-macro2 even in nightly builds for a while, as right now you can't parse a string into a TokenStream with the real proc-macro unless you're running inside the compiler. There are a few use cases for syn right now which require being outside of the compiler, such as in build scripts. (hopefully this will be changed before the new API is stable though).

[dtolnay]

I am not going to review this thoroughly right now but based on a spot check, this is exactly what I had in mind. Merge when ready!

Any idea what the future looks like for combining adjacent spans? Ideally the += token would have a single span. As I understand it, our parsing guarantees that + and = are adjacent bytes in the input, so we could even make do with an implementation that panics when merging spans that are not adjacent.

[dtolnay]

There is a circular dependency here between proc-macro2 and synom. I will make a PR to drop the dependency of proc-macro2 on synom.

[alexcrichton]

Aw shucks, I was hoping we'd have an infinite bootstrap chain where if you compile proc-macro 0.N it requires syn 0.M which requires proc-macro 0.N-1 which requires syn 0.M-1 which requires...

[alexcrichton]

Any idea what the future looks like for combining adjacent spans?

Oh I haven't thought about this much, but I think it'd be quite plausible! If proc_macro itself exposes and API for this it shouldn't be hard to take advantage. Right now tokens like <<= keep track of three spans but there's only one helper function to change to convert that all to one span.

[dtolnay]

I was hoping we'd have an infinite bootstrap chain where if you compile proc-macro 0.N it requires syn 0.M which requires proc-macro 0.N-1 which requires syn 0.M-1 which requires...

I hope you're not serious. dtolnay/proc-macro2#4

[Arnavion]

How much of https://github.com/dtolnay/syn/issues/created_by/Arnavion is fixed by this?

[mystor]

@Arnavion It definitely fixes #114 once we switch to using the real proc-macro (IIRC proc-macro2 doesn't support 128-bit integers yet, but that shouldn't be too hard to change).

Not sure about the others, but I can test. I might fix some of them as part of #148.

[Arnavion]

Also, I'm converting one of my proc macro crates to master to try it out. Shall I file bugs for useful / necessary features that were removed (no more Eq on syn::Path, no more way to check if a syn::Lit is specifically a string literal and get the string out of it, etc) ? Or should I wait?

[mystor]

The second is a known issue. We'll probably solve it either in proc-macro2 (if it looks like proc-macro proper is going go gain that functionality), or I'll make a separate LiteralExt crate to add the methods I want which doesn't have to be built unless you need the feature dtolnay/proc-macro2#9

[mystor]

(we should probably have an issue for both of them on syn though - go ahead and file :))