feat(linter/plugins): Token-related SourceCode APIs (TS ESLint implementation)

[lilnasy]

• "Slow but working" implementation for Linter plugins: Token-related SourceCode APIs #14829 (comment)
• Uses the parser from @typescript-eslint/typescript-estree for tokens. The source code is fully parsed and everything but the tokens and comments is discarded.
• Tests are directly adapted from eslint.
• Direct commits from maintainers welcome.

Tasks

• getTokens
• Move token types to src-js/plugins/tokens.ts
• Bundle typescript, handling _filename usage in ES modules.
• Prevent class name mangling by tsdown. Could be deferred. test(linter/plugins): avoid using scope constructor names in test snapshot #15906 (review)
• Test regex-divide ambiguous syntax.
• Test for conformance directly against eslint tests.
• Parse as JSX only if source text is JSX. Could be deferred.
• Lazy load the parser on demand. perf(linter/plugins): lazy-load tokens parsing code #16011

Future work

• getFirstToken feat(linter/plugins): implement SourceCode#getFirstToken() #16002
• getFirstTokensfeat(linter/plugins): implement SourceCode#getFirstTokens() #15976
• getLastToken feat(linter/plugins): implement SourceCode#getLastToken() #16003
• getLastTokens feat(linter/plugins): implement SourceCode#getLastTokens() #16000
• getTokenBefore feat(linter/plugins): implement SourceCode#getTokenBefore() #15956
• getTokenOrCommentBefore feat(linter/plugins): implement SourceCode#getTokenOrCommentBefore() #16044
• getTokensBefore feat(linter/plugins): implement SourceCode#getTokensBefore() #15955
• getTokenAfter feat(linter/plugins): implement SourceCode#getTokenAfter() #15973
• getTokenOrCommentAfter feat(linter/plugins): implement SourceCode#getTokenOrCommentAfter() #16045
• getTokensAfter feat(linter/plugins): implement SourceCode#getTokensAfter() #15971
• getTokensBetween feat(linter/plugins): implement SourceCode#getTokensBetween() #16034
• getFirstTokenBetween feat(linter/plugins): implement SourceCode#getFirstTokenBetween() #16032
• getFirstTokensBetween feat(linter/plugins): implement SourceCode#getFirstTokensBetween() #16019
• getLastTokenBetween feat(linter/plugins): implement SourceCode#getLastTokenBetween() #16008
• getLastTokensBetween feat(linter/plugins): implement SourceCode#getLastTokensBetween() #16033
• getTokenByRangeStart feat(linter/plugins): implement SourceCode#getTokenByRangeStart() #16043
• isSpaceBetween refactor(linter/plugins): reimplement isSpaceBetween() isSpaceBetweenTokens() using tokens #16055
• isSpaceBetweenTokens refactor(linter/plugins): reimplement isSpaceBetween() isSpaceBetweenTokens() using tokens #16055

Decisions

• @typescript-eslint/typescript-estree peer-depends on typescript. How should we package it? Currently, typescript is being made an optional dependency.
  • overlookmotel (on discord): ideally bundled, direct runtime dependency otherwise.
• Deprecated methods are being removed altogether in ESLint 10: getTokenOrCommentBefore, getTokenOrCommentAfter, and isSpaceBetweenTokens. These are surface level deprecations: the functionality was merged with other methods (the includeComments: true option) and plugins can migrate with a one line change. I'm guessing we are targeting fairly modern, actively developed projects. Should we expose them?
  • overlookmotel and camc314: yes.
• Program range is different between ESLint and TS-ESLint. Which one should we follow?
  • overlookmotel: they are going to both change and align soon, we should update our range calculation to match when they do.

[overlookmotel]

Deprecated methods are being removed altogether in ESLint 10: getTokenOrCommentBefore, getTokenOrCommentAfter, and isSpaceBetweenTokens. These are surface level deprecations: the functionality was merged with other methods (the includeComments: true option) and plugins can migrate with a one line change.

Cameron and I had a bit of an argument about exactly this question!

We concluded in the end to keep all the deprecated methods, to maximize compatibility with older plugins, which may take some time to get updated (or in some cases, will never get updated). Our rationale is that most of these methods are just aliases, so no maintenance burden to keep them. The only one which is slightly different from its non-deprecated "brother" is isSpaceBetweenTokens, and that's pretty simple to implement - it just treats JSXText differently (if I remember right).

I'm guessing we are targeting fairly modern, actively developed projects.

That's true, but even actively developed projects may use old unmaintained plugins.

Note: I added the stubs in #15645. That PR also added tests which illustrate the difference in behavior between isSpaceBetween and isSpaceBetweenTokens.

[overlookmotel]

Possibly premature review, as it's still marked as draft, but I'm keen to get it merged so I thought I'd go through it now.

Apart from the comments below, this looks good to me. Sorry for the volume of comments - most are pretty small details.

Once we're happy with getTokens impl, I think we should merge this, and we can add more methods in separate PRs. No need to do the whole API in a single PR.

A couple of points which we can also leave to follow-ups:

1. We should ideally lazy-load @typescript-eslint/typescript-estree package only when getTokens is first called.

2. I assume TS-ESLint's parser also generates a ScopeManager. If it does, we may as well cache it, to avoid running scope analysis again if plugin does sourceCode.getTokens() followed by getting sourceCode.scopeManager.

[lilnasy]

2. I assume TS-ESLint's parser also generates a ScopeManager. If it does, we may as well cache it, to avoid running scope analysis again if plugin does sourceCode.getTokens() followed by getting sourceCode.scopeManager.

@typescript-eslint/parser does. @typescript-eslint/typescript-estree does not.

The former delegates to the latter for parsing and lexing. I decided to use @typescript-eslint/typescript-estree directly to instantiate scope managers and parsers granularly.

[overlookmotel]

The former delegates to the latter for parsing and lexing. I decided to use @typescript-eslint/typescript-estree directly to instantiate scope managers and parsers granularly.

Ah ha that makes perfect sense. I had misunderstood.