Conversion from Tokens to proc_macro::TokenStream

[dtolnay]

What is the story behind dtolnay/quote@0b8ebcb? Relevant to #178.

[alexcrichton]

Oh I added that commit initially to quote because I thought it was necessary-ish for #165 but it ended up not being so, so I made the API more conservative. I've been thinking about the proc_macro vs not situation in the sense that crates like quote and syn are useful in standalone programs because there's no dependency on proc_macro. With recent changes, however, I wasn't sure if we wanted to preserve that or specifically avoid that. For example there's no need for proc-macro2 to depend on the proc_macro crate itself.

In any case ergonomically it seems quite nice to have, so any objections to re-adding?

[mystor]

I talked at the all hands a bit about this, and I think the plan for the future is to make sure that proc_macro works outside of the compiler in standalone crates before it is stabilized. I think that we shouldn't be trying to avoid syn or quote from depending on proc_macro, as it will make the API a lot cleaner, and keep the interface uniform between being within procedural macros and in independent programs.

In addition, eventually being able to drop having to build the lexer could really help with compile times ^.^.

[alexcrichton]

I've opened dtolnay/quote#40 to "officially" add it