ECMAScript syntax: Comment affecting semantics of RegExp

[SMotaal]

• VSCode Version: Version 1.33.1 (1.33.1)
• OS Version: macOS 10.14.4 (18E226)

Steps to Reproduce:

The following is contrived to show this particular case.

/* prettier-ignore */
const dummy  = (div, g) => {
  ( []              /([(div)])/g / [] );
  ( ([])            /([(div)])/g / [] );
  ( [] /* */        /([(div)])/g / [] );
};

1. Paste this code into a new javascript file
2. Ignore any type-specific errors (because it is still valid ES syntax)
3. Note the last statement highlights as RegExp but still tokenizes correctly by tsserver.

Does this issue occur when all extensions are disabled?: Not tested

[SMotaal]

So, I too am playing this game apparently, it's whac-a-mole… the missing ingredient is ASI logic. The good thing is your highlighter logic is doing far better than github's especially with it all being client-side — the problem I am seeing is more of two tokenizers overlapping on a single source text, it is an anti-pattern to vscode's project structure itself (ie you can maybe summarize TypeScript's tokenizer in a few expressions, until they make change, you need a few weeks to dwell) to be fair to the amazing effort you are doing.

So more best shown in:

Feel free to use the dummy code to test.

Note: a false (div) token is followed by a #66b2b2-ish g

cc/ @mjbvz @sheetalkamat