Switch from using "while" to "end" and adjust nested rules accordingly

[DanTup]

While investigating #11 I found a difference in how VS Code and GitHub/linguist handle when in the grammars.

In VS Code, when a when pattern stops matching, any open scopes from nested rules are ended, whereas on GitHub they are not. This means an unterminated triple-backtick block on GitHub can escape out of the surrounding comment.

The discussion in github-linguist/linguist#7015 concluded that VS Code's behaviour is incorrect, and GitHub's behaviour (to just keep processing the nested rules until they end before re-evaluating the while) matches TextMate.

Since this isn't well specified and it's not clear if VS Code will change behaviour to match, this change stops using while in the grammar and adjusts the end rules for the nested code blocks to detect when they are falling out of the comment (instead of assuming everything until the next triple-backticks is code.. which could be the entire file). It also fixes some issues where we didn't handle termination of block comments correctly either.

Before changes:

Although that looks worse because the unclosed comment breaks the whole doc. Without that, the rest of the file before the changes looks like this:

After changes:

[DanTup]

@devoncarew @jwren not sure who is good to review this. It migrates off while because behaviour is inconsistent in VS Code vs other tools (like TextMate and GitHub) and expands the tests for code blocks in dartdocs. The comment above includes screenshots showing the original behaviour (top two) and new behaviour (bottom).

[DanTup]

Looks like this might have broken nested multiline comments, so I'm changing it back to draft and will come back to it soon.

[DanTup]

I did some more testing, and I could not reproduce any issue with nested block comments today (and I noticed that we already have tests here for nested blocks). Foolishly, I did not include an example of what I thought was broken above - however I noticed when testing that when I was editing the opening tags for some nested comments, VS Code would sometimes "auto complete" the closing comment marker, meaning I had an extra closing marker, which of course does result in bad highlighting (and invalid code). I can only assume it was that that I saw when testing this.

So, in the absence of any reproducible issue, I think we should go with this and address any issues as they come up.

@jwren that means this is good to review :-)

[DanTup]

Ugh, I re-found the bug I'd noted above after merging. I've opened a revert at #78 and will work on getting the failure cases into the tests here to fix.

Edit: Aborting the revert, as I have a fix.