Implement (contextual) keywords and use their versioning from v2

[Xanewok]

Closes #568

There is still one outstanding issue where we return a Vec<TokenKind> from next_token; it'd like to return a more specialized type and ideally pass it on stack (2x2 bytes), rather than on-heap (extra 3x8 bytes for the Vec handle + indirection). We should name it better and properly show that we can return at most 2 token kinds (single token kind or identifier + kw combo).

To do:

• Return tokens from next_token via stack

Apart from that, I think this is a more correct approach than #598, especially accounting for the new keyword definition format in DSL v2.

The main change is that we only check the keyword trie and additionally the (newly introduced) compound keyword scanners only after the token has been lexed as an identifier. For each context, we collect Identifier scanners used by the keywords and attempt promotion there.

The existing lexing performance is not impacted from what I've seen when running the sanctuary tests and I can verify (incl. CST tests) that we now properly parse source that uses contextual keywords (e.g. from) and that the compound keywords (e.g. ufixedMxN) are properly versioned.

This adapts the existing codegen_grammar interface that's a leftover from DSLv1; I did that to work on finishing #638; once this is merged and we now properly parse contextual keywords, I'll move to clean it up and reduce the parser codegen indirection (right now we go from v2 -> v1 model -> code generator -> Tera templates; it'd like to at least cut out the v1 model and/or simplify visiting v2 from the existing CodeGenerator).

Please excuse the WIP comments in the middle; the first and the last ones should make sense when reviewing. I can simplify this a bit for review, if needed.

[changeset-bot]

🦋 Changeset detected

Latest commit: 4fb350e

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
@nomicfoundation/slang	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[Xanewok]

Instead of next_token returning Vec (first pass), we now introduce the following type:

pub enum ScannedToken {
    Single(TokenKind),
    IdentifierOrKeyword {
        identifier: TokenKind,
        kw: KeywordScan,
    },
}

Not only we don't go through heap for the often-called next_token, we now explicitly have to handle the case of identifiers/keywords and whether a keyword is strictly reserved or "present", so usable in both keyword and identifier position.

To keep the existing parser flow mostly the same, a helper fn ScannedToken::unambiguous(self) -> TokenKind is introduced, that tries to map back to a single token kind if possible (for the "present" keyword it falls back to a general "identifier" token kind).

This also introduces another helper fn ScannedToken::accepted_as(self, expected: TokenKind) -> bool that is used in the parser functions as a convenience, which returns whether the scanned token is accepted in a given position, streamlining the underlying additional kw reservation checks for identifier token kinds.

[Xanewok]

Not perfect, I admit; this should probably be handled with an automaton for the keywords and executed/matched on the string till it's exhausted but let's punt on this for now

[OmarTawfik]

Left a few suggestions.