Expand docs on Macros By Example.

[alercah]

The primary motivation here was to increase clarity and fully address the
scoping and naming details. The inclusion of RFC 550's formal specification is
to move it to the reference where it can be updated. I made several changes,
motivated by accommodating ? and new fragment specifiers, but there are some
other things which need highlighting so that they can be double-checked for
correctness.

• Permit the empty string to follow on in the first invariant; this is a
  technical oversight in the definition I believe.
• Added a requirement that repetitions obey the follow rules; this was an
  oversight in the original RFC and currently planned for fix.
• Rewrote the definition of FIRST for complex NTs to be more clear.
• Added a case to LAST for ? repetitions
• Removed the last example of LAST, because it is wrong.
• Rearranged the definition of FOLLOW to be more clear
• Added Shl to FOLLOW(ty) and FOLLOW(path), as documented in the Reference
  already.
• Added missing follow sets for newer fragment specifiers.

The scoping text is probably not completely accurate, but it's certainly much
better than what was there before (i.e. basically nothing).

[alercah]

cc @Centril @alexreg @estebank @petrochenkov

[petrochenkov]

Reviewed.
Great work.

[nikomatsakis]

Not sure what they are up to these days, but @paulstansifer was the original author of the algorithm we use and I don't think the approach has changed much. There is at least some chance they have the time to review this PR =)

[ehuss]

This is great, I have been looking forward to seeing this! Just yesterday I got a weird "import resolution is stuck, try simplifying macro imports" error, and I think this makes it clearer for me!

I've been wondering if it would be useful to make it clear somewhere that $crate is two separate tokens? There are a few places in the reference that abut tokens in such a way that makes it not clear that they are distinct. It's a minor issue, and can be inferred from the tokens chapter, just a thought.

[alexreg]

It looks like the macro guru @petrochenkov has already reviewed, so my input probably isn't needed -- but in brief, good work. :-)

[petrochenkov]

Typo: repititions -> repetitions here and in few other places below.

[Centril]

Mostly proofreading...

[Centril]

I'd break into a section for #[macro_use] here.

[jethrogb]

Can you add something along these lines?

When forwarding a matched fragment to another macro-by-example, matchers in the second macro will see an opaque AST of the fragment type. The second macro can't use literal tokens to match this in the matcher, only a fragment specifier of the same type. The ident, lifetime, and tt fragment types are an exception, and can be matched by literal tokens.

This explains the difference between:

// compiles OK
macro_rules! foo {
    ($l:tt) => { bar!($l); }
}

macro_rules! bar {
    (3) => {}
}

fn main() {
    foo!(3);
}

macro_rules! foo {
    ($l:expr) => { bar!($l); }
// ERROR:               ^^ no rules expected this token in macro call
}

macro_rules! bar {
    (3) => {}
}

fn main() {
    foo!(3);
}

[Centril]

Ping @alercah, there are some unaddressed review comments but otherwise this looks real good :)

[alercah]

I'm unable to work on this at the moment, someone else is welcome to bring it home or else I will get to it when I can.

[ehuss]

I'd be happy to address the final bits of the review. I think this is great content, and would like to see it published. I have rebased and pushed a small update.

[paulstansifer]

@nikomatsakis Thanks for thinking of me! I'm too busy to do anything other than skim this and say that it looks like a great resource. I also couldn't vet the technical stuff too well; the stuff with FIRST and FOLLOW is from after my time; I was pleasantly surprised when the compiler sprung it on me.

[Centril]

Thanks @alercah; @ehuss thanks also for the fixes; Merge when you want :)

[ehuss]

Merging so I can use it, thanks everyone!