[ty] Fix range filtering for tokens starting at the end of the requested range #21193
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
MichaReiser
added
server
MichaReiser
requested review from
AlexWaygood,
carljm and
sharkdp
as code owners
Contributor
Diagnostic diff on typing conformance testsNo changes detected when running ty on typing conformance tests ✅ |
Contributor
|
MichaReiser
force-pushed
the
micha/semantic-tokens-fix
branch
from
b6ec0b2 to
b3dc5ea
Compare
Fix semantic tokens to exclude ranges that only touch (are empty) when filtering by range. Previously, ranges that touched the filter range but were empty were incorrectly included. This ensures that empty ranges at range boundaries are properly filtered out.
MichaReiser
force-pushed
the
micha/semantic-tokens-fix
branch
from
b3dc5ea to
cc54de8
Compare
sharkdp
approved these changes
Nov 2, 2025
| " | ||
| x = 1 | ||
| y = 2 | ||
| z = 3<CURSOR> |
Contributor
There was a problem hiding this comment.
Aside: I found the <CURSOR> here distracting, as I initially thought it would somehow influence or determine the target range. Can it be removed?
Member
Author
There was a problem hiding this comment.
I agree that CURSOR is ony confusing here and that semantic tokens should probably not use the cursor infrastructure to begin with. But I'd rather clean this up in a separate PR
Co-authored-by: David Peter <sharkdp@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR fixes an issue where the semantic token provider returned tokens that
started at the end offset of the selected range. That is, the
token and requested range intersect, but the intersecting range is empty.
This isn't an issue for most documents because the only outcome is tha the server
sends one additional token to the client. However, this is an issue for notebooks
where the response can only contain tokens that all belong to the same cell. Sending one extra token can then have the effect of sending the first token of the next cell,
which isn't allowed.
Test Plan
Added test. My in draft notebook PR no longer panics during semantic syntax highlighting