Parser-guided fuzzer

[jsatdapr]

Issue To Be Solved

The grammar in docs/cadence.ebnf is not obviously correct, but not strictly necessary. We can have a parser-guided fuzzer, without a formal grammar, simply by refactoring the parser to specify which tokens it accepts, each time it asks the lexer for the next one.

Tasks

1. Make Token.Type and Token.Value be functions, not bare data
2. Annotate calls to Token.Type() and Token.Value() with lists of expected results
3. At each call to Token.Type() or Token.Value(), the fuzzer selects from the list of expected results.
4. Figure out how to structure code so that the behavior of Type() and Value() is unchanged when not running the fuzzer.

[jsatdapr]

Note: since the "grammar" here is implicit and generated on-the-fly during parsing, it's impossible to "balance" it ahead of time, something that is possible with a formal grammar: #1062

[jsatdapr]

Note: performance regression is a concern here, since we are hiding Token.Type and Token.Value behind a "getter" function call, where they were previously just plain data. I will run the benchmarks and post the results here for further discussion.

[jsatdapr]

Placing Token behind an interface caused no noticeable loss in performance, even with grammar-checking enabled at runtime.

[jsatdapr]

This is blocked on subtask #1094

[robert-e-davidson3]

Deprecating this ticket in favor of epic #1309.