When tokenising, use semi-colons to hint error recovery

[nrc]

e.g.,

pub fn trace_option(option: Option<isize>) {
    option.map(|some| 42;
}

When we hit }, we know that the delimiters are mismatched, we currently close all the token trees to recover, but that then gives us a bunch of errors in parsing. We should rewind to the ; and treat that kind of like an EOF, closing the ( and then continuing to tokenise from the ;.

Annoyingly we can't do this eagerly since ; could always be expected. For parens though, this can only happen in macros, so I think for error recovery we can assume the ; terminates a token tree.

We could do better by tracking whether we are in macro context during tokenising.

[oli-obk]

What about the semi-colon in array types and repeat expressions? Won't those complicate things?

[nrc]

We would only do this when we know we are outside {} and [] (using the token tree rules).