feat(linter/plugins): add `tokens` property to `Program` #16020

overlookmotel · 2025-11-24T01:20:04Z

Add a tokens property to Program. Like ESLint, program.tokens contains only syntax tokens, and does not include comments.

Implemented as a getter, which lazily tokenizes source text on first access, using the same mechanism as other tokens APIs.

Note: If user has an AST to access the tokens getter on, sourceText must have been initialized, so the getter doesn't need to check that.

overlookmotel · 2025-11-24T01:20:19Z

feat(linter/plugins): add tokens property to Program #16020 👈 (View in Graphite)
main

How to use the Graphite Merge Queue

Add either label to this PR to merge it via the merge queue:

0-merge - adds this PR to the back of the merge queue
hotfix - for urgent hot fixes, skip the queue and merge this PR next

You must have a Graphite account in order to use the merge queue. Sign up using this link.

_{An organization admin has enabled the Graphite Merge Queue in this repository.} _{Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue.}

This stack of pull requests is managed by Graphite. Learn more about stacking.

Copilot

Pull request overview

This PR adds a tokens property to the Program AST node for the linter. Like ESLint, program.tokens contains only syntax tokens and excludes comments. The tokens are lazily initialized using TS-ESLint when first accessed.

Key Changes

Added tokens: Token[] field to the Program type in TypeScript type definitions
Implemented lazy getter for program.tokens that calls initTokens() from TS-ESLint on first access
Exported tokens variable and initTokens() function from the tokens module to support the new getter

Reviewed changes

Copilot reviewed 8 out of 10 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
tasks/ast_tools/src/generators/typescript.rs	Updated TypeScript type generator to add `tokens` field to `Program` interface and import/export `Token` type
tasks/ast_tools/src/generators/raw_transfer.rs	Added import statement for `tokens` and `initTokens` in generated deserializer for linter builds
crates/oxc_ast/src/serialize/mod.rs	Added lazy getter for `tokens` property in `Program` deserialization template
apps/oxlint/src-js/plugins/tokens.ts	Exported `tokens` variable and `initTokens()` function, added JSDoc clarifying caller must ensure `sourceText` is initialized
apps/oxlint/src-js/generated/types.d.ts	Generated TypeScript types with `tokens` field added to `Program` interface
apps/oxlint/src-js/generated/deserialize.js	Generated deserializer with `tokens` getter implementation and necessary imports
apps/oxlint/test/fixtures/tokens/plugin.ts	Test plugin that accesses `ast.tokens` and reports all tokens with their type, location, range, and value
apps/oxlint/test/fixtures/tokens/output.snap.md	Snapshot showing tokens are correctly exposed (10 syntax tokens) and comments are excluded
apps/oxlint/test/fixtures/tokens/files/index.js	Test fixture with multiple comments to verify they're excluded from `ast.tokens`
apps/oxlint/test/fixtures/tokens/.oxlintrc.json	Test configuration enabling the tokens test plugin

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

crates/oxc_ast/src/serialize/mod.rs

apps/oxlint/test/fixtures/tokens/files/index.js

codspeed-hq · 2025-11-24T01:28:54Z

CodSpeed Performance Report

Merging #16020 will not alter performance

_{Comparing 11-24-feat_linter_plugins_add_tokens_property_to_program_ (eddeedb) with main (3fe5225)¹}

Summary

✅ 42 untouched
⏩ 3 skipped²

No successful run was found on main (b8bf404) during the generation of this report, so 3fe5225 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩
3 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

overlookmotel · 2025-11-24T01:39:41Z

Merge activity

Nov 24, 1:39 AM UTC: The merge label '0-merge' was detected. This PR will be added to the Graphite merge queue once it meets the requirements.
Nov 24, 1:39 AM UTC: overlookmotel added this pull request to the Graphite merge queue.
Nov 24, 1:45 AM UTC: Merged by the Graphite merge queue.

overlookmotel · 2025-11-24T01:39:43Z

cc @lilnasy

Add a `tokens` property to `Program`. Like ESLint, `program.tokens` contains only syntax tokens, and does not include comments. Implemented as a getter, which lazily tokenizes source text on first access, using the same mechanism as other tokens APIs. Note: If user has an AST to access the `tokens` getter on, `sourceText` must have been initialized, so the getter doesn't need to check that.

@license

### 💥 BREAKING CHANGES - cbb27fd ast: [**BREAKING**] Add `TSGlobalDeclaration` type (#15712) (overlookmotel) ### 🚀 Features - 0c1f82b linter/plugins: Add `tokens` property to `Program` (#16020) (overlookmotel) - 6cff132 span: Add `Span::merge_within` method (#15869) (sapphi-red) - 102365d allocator/vec: Add `Vec::into_bump_slice` method (#15770) (Dunqing) ### 🐛 Bug Fixes - e2ca770 codegen: Add support for printing type arguments in new expressions (#15963) (Ives van Hoorne) - 2bd3cb6 apps, editors, napi: Fix `oxlint-disable` comments (#16014) (overlookmotel) - 622cb5e parser: Preserve legal comments with @preserve/@license when preceded by other annotations (#15929) (copilot-swe-agent) - 7c46a9e transformer/tagged-template-transform: Handle `\n` escape sequences (#15830) (Dunqing) - f386efc minifier: Avoid generating invalid spans (#15778) (sapphi-red) - d4ff004 parser: Forbid invalid modifiers on `module` and `global` (#15723) (overlookmotel) - 2191ae9 semantic: Allow reserved keywords in typescript ambient contexts (#15495) (sapphi-red) - 7d1ebad isolated-declarations: Incorrect nested namespace output in isolated declarations (#15800) (copilot-swe-agent) ### ⚡ Performance - b4b0ed8 transformer/typescript: Reverse order of checks (#15722) (overlookmotel) ### 📚 Documentation - c81a331 data_structures: Doc comments on fields of `Stack` (#15793) (overlookmotel) - cfae31d allocator: Use `allocator` as var name in examples (#15781) (overlookmotel) Co-authored-by: Boshen <1430279+Boshen@users.noreply.github.com>

# Oxlint ### 💥 BREAKING CHANGES - cbb27fd ast: [**BREAKING**] Add `TSGlobalDeclaration` type (#15712) (overlookmotel) ### 🚀 Features - 72660f7 linter: Support auto generate config document for tuple lint option (#15904) (Duc Nghiem Xuan) - 0c1f82b linter/plugins: Add `tokens` property to `Program` (#16020) (overlookmotel) - 9e61beb linter/plugins: Implement `SourceCode#getFirstToken()` (#16002) (Arsh) - 9a548dd linter/plugins: Implement `SourceCode#getLastTokens()` (#16000) (Arsh) - 0b6cb11 linter/plugins: Implement `SourceCode#getFirstTokens()` (#15976) (Arsh) - 166781e linter/plugins: Implement `SourceCode#getTokenAfter()` (#15973) (Arsh) - 6ae232b linter: Expose type errors via tsgolint (#15917) (camc314) - 2bfdd26 linter/plugins: Implement `SourceCode#getTokensAfter()` (#15971) (Arsh) - 45fffc1 linter/plugins: Implement `SourceCode#getTokenBefore()` (#15956) (Arsh) - 776e473 linter/plugins: Implement `SourceCode#getTokensBefore()` (#15955) (Arsh) - 595867a oxlint: Generate markdownDescription fields for oxlint JSON schema. (#15959) (connorshea) - 5569317 vscode: Add quick actions to status bar tooltip (#15962) (Sysix) - 986cac1 linter/plugins: Token-related `SourceCode` APIs (TS ESLint implementation) (#15861) (Arsh) - a21f9e4 linter: Implement unicorn/prefer-bigint-literals rule (#15923) (Michiel Vrins) - 4b9d8d2 linter/type-aware: Include range with tsconfig diagnostics (#15916) (camc314) - 220d01e editor: Improve the status bar item for the VS Code extension by adding a tooltip. (#15819) (connorshea) ### 🐛 Bug Fixes - 2bd3cb6 apps, editors, napi: Fix `oxlint-disable` comments (#16014) (overlookmotel) - 81f5360 linter/prefer-number-properties: Get fixer to replace entire call expr (#15979) (camc314) - e4ba07f language_server: Always write to memory file system (#15975) (Sysix) - a8a2032 linter: Support missing `range` for internal diagnostics (#15964) (camc314) - 619a226 oxlint/lsp: Don't register `textDocument/formatting` capability (#15882) (Sysix) - 6ab1a20 linter: Fix no_useless_spread producing invalid syntax when removing empty object spreads (#15905) (camc314) - be4b6df linter: Unicorn/prefer-string-replace-all incorrectly escapes backslashes (#15901) (camc314) - 9fa9ef2 linter: Gracefully fail when using import plugin, large file counf and JS plugins (#15864) (camc314) - c027398 linter/plugins: Correct bindings package names glob in TSDown config (#15871) (overlookmotel) - b622ef8 linter: Fix `oxc/bad_array_method_on_arguments` rule behavior. (#15854) (connorshea) - aa06c3f linter: Recognize NewExpression as value context in no-unused-private-class-members (#15843) (camc314) - e89c5ba typescript/prefer-namespace-keyword: Skip nested `TSModuleDeclaration`s (#15806) (overlookmotel) - 646d020 linter/exhaustive-dependencies: Prevent is_callee_of_call_expr flag from leaking into nested expressions (#15832) (camc314) - 46bd6bd linter/plugins: Pin `@typescript-eslint/scope-manager` dependency (#15807) (overlookmotel) - fba31fa linter: Patch `@typescript-eslint/scope manager` (#15214) (Arsh) - 50307c1 linter/jest: Ignore `expect` identifier in argument position (#15785) (camc314) ### ⚡ Performance - 024b48a linter/plugins: Lazy-load tokens parsing code (#16011) (overlookmotel) - 15365c9 linter/plugins: Reduce var assignments (#15953) (overlookmotel) - 84d1f4f linter/plugins: Downgrade some checks to debug-only (#15922) (overlookmotel) - a49f704 linter/typescript: Avoid searching source text unless required (#15805) (overlookmotel) ### 📚 Documentation - ceffa5a linter: Add config option docs for various rules. (#16024) (connorshea) - 9a0ed13 linter: Fix config option docs for eslint/operator-assignment rule. (#16030) (connorshea) - 0b18005 linter: Add config docs generation for rules with Regex arguments (#15978) (connorshea) - 48d18e0 linter: Improve diagnostic message for promise/catch-or-return rule (#15980) (connorshea) - 6c72e84 linter: Use backticks for code elements across more rule diagnostics (#15958) (connorshea) - a63dad7 linter/plugins: Add comment (#15952) (overlookmotel) - 81ea642 vscode: Use markdownDescription for better formatting in VSCode Settings (#15889) (connorshea) - db6a110 linter/plugins: Fix JSDoc comment (#15884) (overlookmotel) - 1487271 linter: Add config option docs for `jsdoc/require-param` and `jsdoc/require-returns` rules (#15857) (connorshea) - fbf0fd4 linter/plugins: Add JSDoc comments to `Plugin` and `Rule` types (#15815) (overlookmotel) - ac5e4b5 linter/plugins: Add JSDoc comments and improve comments (#15814) (overlookmotel) - 9b7b083 linter: Fix error in `curly` `"all"` example (#15801) (camc314) - 65a3520 linter: Improve diagnostic for consistent-type-definitions rule. (#15752) (connorshea) ### 🛡️ Security - f9b9276 deps: Update dependency rolldown to v1.0.0-beta.51 (#15856) (renovate[bot]) # Oxfmt ### 💥 BREAKING CHANGES - a937890 formatter: [**BREAKING**] Default to `lineWidth: 100` (#15933) (leaysgur) - 03d5f5a formatter/sort-imports: [**BREAKING**] Change default order to `natural` with `natord` crate (#15828) (leaysgur) - cbb27fd ast: [**BREAKING**] Add `TSGlobalDeclaration` type (#15712) (overlookmotel) ### 🚀 Features - 7818e22 formatter/sort-imports: Support `options.groups` (#15831) (leaysgur) - f9a502c oxfmt: `oxfmt --lsp` support (#15765) (leaysgur) ### 🐛 Bug Fixes - 4817486 formatter: Revert `FormatElement::BestFitting` printing logic (#16028) (Dunqing) - 5562dd6 formatter: Incorrect formatting method chain with trailing comments (#16027) (Dunqing) - 6d14c8b formatter: Comments in export class decorators are printing incorrectly (#15897) (Dunqing) - 683c764 formatter: Correct a few minor mismatched typescript tests (#15894) (Dunqing) - c11cc07 formatter: Improve formatting for default type on type parameters (#15893) (Dunqing) - 0bff596 formatter: Handle JSX expresssion dangling comment (#15890) (leaysgur) - 16a9dc8 formatter: Inconsistent printing of class extends and interface extends (#15892) (Dunqing) - 300b496 formatter: Inconsistent CallExpression and NewExpression around member chain and logical expression (#15858) (Dunqing) ### ⚡ Performance - 65174cc formatter: Reduce the size of `TextWidth` to 4 byte (#15827) (Dunqing) - 4fe3aac formatter: Use `ArenaVec` and `ArenaBox` (#15420) (Dunqing) Co-authored-by: overlookmotel <557937+overlookmotel@users.noreply.github.com>

github-actions bot added A-linter Area - Linter A-cli Area - CLI A-ast Area - AST A-ast-tools Area - AST tools A-linter-plugins Area - Linter JS plugins labels Nov 24, 2025

github-actions bot added the C-enhancement Category - New feature or request label Nov 24, 2025

overlookmotel marked this pull request as ready for review November 24, 2025 01:21

overlookmotel requested a review from camc314 as a code owner November 24, 2025 01:21

Copilot AI review requested due to automatic review settings November 24, 2025 01:21

overlookmotel self-assigned this Nov 24, 2025

Copilot started reviewing on behalf of overlookmotel November 24, 2025 01:21 View session

Copilot finished reviewing on behalf of overlookmotel November 24, 2025 01:23

Copilot AI reviewed Nov 24, 2025

View reviewed changes

crates/oxc_ast/src/serialize/mod.rs Show resolved Hide resolved

apps/oxlint/test/fixtures/tokens/files/index.js Show resolved Hide resolved

apps/oxlint/test/fixtures/tokens/files/index.js Show resolved Hide resolved

overlookmotel added the 0-merge Merge with Graphite Merge Queue label Nov 24, 2025

graphite-app bot force-pushed the 11-24-feat_linter_plugins_add_tokens_property_to_program_ branch from eddeedb to 0c1f82b Compare November 24, 2025 01:40

graphite-app bot merged commit 0c1f82b into main Nov 24, 2025
20 checks passed

graphite-app bot deleted the 11-24-feat_linter_plugins_add_tokens_property_to_program_ branch November 24, 2025 01:45

graphite-app bot removed the 0-merge Merge with Graphite Merge Queue label Nov 24, 2025

This was referenced Nov 24, 2025

release(crates): oxc v0.99.0 #16037

Merged

release(apps): oxlint v1.30.0 && oxfmt v0.15.0 #16039

Closed

Boshen mentioned this pull request Nov 24, 2025

release(apps): oxlint v1.30.0 && oxfmt v0.15.0 #16047

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

feat(linter/plugins): add `tokens` property to `Program` #16020

feat(linter/plugins): add `tokens` property to `Program` #16020

Uh oh!

overlookmotel commented Nov 24, 2025 •

edited

Loading

Uh oh!

overlookmotel commented Nov 24, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codspeed-hq bot commented Nov 24, 2025

Uh oh!

overlookmotel commented Nov 24, 2025 •

edited by graphite-app bot

Loading

Uh oh!

overlookmotel commented Nov 24, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

feat(linter/plugins): add tokens property to Program #16020

feat(linter/plugins): add tokens property to Program #16020

Uh oh!

Conversation

overlookmotel commented Nov 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

overlookmotel commented Nov 24, 2025

How to use the Graphite Merge Queue

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Key Changes

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codspeed-hq bot commented Nov 24, 2025

CodSpeed Performance Report

Merging #16020 will not alter performance

Summary

Footnotes

Uh oh!

overlookmotel commented Nov 24, 2025 • edited by graphite-app bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge activity

Uh oh!

overlookmotel commented Nov 24, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

feat(linter/plugins): add `tokens` property to `Program` #16020

feat(linter/plugins): add `tokens` property to `Program` #16020

overlookmotel commented Nov 24, 2025 •

edited

Loading

overlookmotel commented Nov 24, 2025 •

edited by graphite-app bot

Loading