Use the roslyn tokenizer#10702

Merged

333fred merged 17 commits intodotnet:features/roslyn-tokenizerfrom 333fred:roslyn-tokenizer

Aug 12, 2024

+1,745-1,229

Commits on Aug 1, 2024

Remove unused ITokenizer interface, make tokenizers disposable.
333fred
committed
Remove unneeded common states
333fred
committed
Allow setting the next tokenizer state after a razor comment body is tokenized.
333fred
committed
Plumb the Roslyn tokenizer through more tests
333fred
committed
Nullable enable the tokenizer
333fred
committed
Add consolidated syntax kinds. Since there aren't consumers that care about the individual differences between most C# operators, as well the difference between C# numeric types, we'll just use a s…
333fred
committed
Give better error messages when syntax tree verification fails.
333fred
committed
The big change: use the roslyn tokenizer for tokenizing C#.
333fred
committed
Update SyntaxKinds to the new consolidated kinds in tests.
333fred
committed
Increase test coverage of a few scenarios, and bulk up existing tests.
333fred
committed
Skip test for now, give a better exception.
333fred
committed
Name change
333fred
committed

Commits on Aug 2, 2024

Commits on Aug 6, 2024

Additional commenting and PR feedback. I've simplified the reset loop to just be from the back of the results, as that's the most common order for the parser to reset in. I've also refactored a com…
333fred
committed

Commits on Aug 9, 2024

More feedback
333fred
committed