Optimize TokenMap updates and traversals for modified files only

[JoshuaBatty]

Description

This PR introduces significant optimisations in how we handle TokenMap updates and file traversals, resulting in substantial performance improvements.

Key Changes:

1. Selective TokenMap Clearing: Instead of clearing the entire TokenMap on each keystroke, we now only clear tokens belonging to the modified file.
2. Targeted Traversal: We've limited the traversal to only the modified file, eliminating unnecessary processing of unchanged dependencies and workspace files.

Performance Impact:

These optimisations have led to substantial improvements in traverse times, particularly for subsequent traversals:

Build Type	Before (ms)	After (ms)	Absolute Improvement (ms)	Relative Improvement (%)
Release	16.218875	4.758958	11.459917	70.66%
Debug	30.196084	7.144166	23.051918	76.34%

closes #6292
related to #5445

Checklist

• I have linked to any relevant issues.
• I have commented my code, particularly in hard-to-understand areas.
• I have updated the documentation where relevant (API docs, the reference, and the Sway book).
  • If my change requires substantial documentation changes, I have requested support from the DevRel team
• I have added tests that prove my fix is effective or that my feature works.
• I have added (or requested a maintainer to add) the necessary Breaking* or New Feature labels where relevant.
• I have done my best to ensure that my PR adheres to the Fuel Labs Code Review Standards.
• I have requested a review from the relevant team or maintainers.

[github-actions]

Benchmark for fdbb9f9

Click to view benchmark

Test	Base	PR	%
code_action	5.0±0.04ms	5.1±0.08ms	+2.00%
code_lens	290.2±8.31ns	332.0±9.26ns	+14.40%
compile	2.6±0.07s	2.6±0.05s	0.00%
completion	4.6±0.11ms	4.6±0.01ms	0.00%
did_change_with_caching	2.5±0.05s	2.6±0.04s	+4.00%
document_symbol	916.3±35.78µs	941.3±33.66µs	+2.73%
format	70.2±2.94ms	69.5±1.26ms	-1.00%
goto_definition	332.4±7.66µs	344.0±6.42µs	+3.49%
highlight	8.7±0.14ms	8.7±0.12ms	0.00%
hover	493.0±6.43µs	499.3±8.07µs	+1.28%
idents_at_position	117.2±0.41µs	121.8±0.34µs	+3.92%
inlay_hints	617.1±25.82µs	642.9±37.58µs	+4.18%
on_enter	2.0±0.04µs	2.1±0.02µs	+5.00%
parent_decl_at_position	3.6±0.04ms	3.6±0.04ms	0.00%
prepare_rename	331.1±5.07µs	340.4±3.45µs	+2.81%
rename	9.0±0.16ms	9.0±0.12ms	0.00%
semantic_tokens	1200.5±58.84µs	1248.4±12.95µs	+3.99%
token_at_position	336.8±1.99µs	340.2±2.77µs	+1.01%
tokens_at_position	3.6±0.03ms	3.6±0.03ms	0.00%
tokens_for_file	402.1±9.96µs	400.7±3.46µs	-0.35%
traverse	33.4±1.02ms	34.6±1.01ms	+3.59%

[github-actions]

Benchmark for 22e6c71

Click to view benchmark

Test	Base	PR	%
code_action	5.0±0.10ms	5.0±0.10ms	0.00%
code_lens	283.8±5.89ns	337.8±10.79ns	+19.03%
compile	1889.1±53.62ms	1931.5±56.89ms	+2.24%
completion	4.4±0.02ms	4.4±0.06ms	0.00%
did_change_with_caching	1863.4±31.99ms	1818.5±43.05ms	-2.41%
document_symbol	902.7±36.32µs	949.6±14.32µs	+5.20%
format	84.0±0.84ms	82.3±0.54ms	-2.02%
goto_definition	336.1±3.04µs	343.9±6.33µs	+2.32%
highlight	8.7±0.13ms	8.8±0.25ms	+1.15%
hover	348.9±6.74µs	362.2±10.49µs	+3.81%
idents_at_position	117.8±0.36µs	118.0±0.77µs	+0.17%
inlay_hints	630.3±13.71µs	637.5±10.83µs	+1.14%
on_enter	2.0±0.09µs	2.1±0.02µs	+5.00%
parent_decl_at_position	3.6±0.02ms	3.6±0.07ms	0.00%
prepare_rename	339.3±4.28µs	341.0±7.52µs	+0.50%
rename	9.0±0.35ms	9.1±0.12ms	+1.11%
semantic_tokens	1243.9±13.01µs	1289.3±23.95µs	+3.65%
token_at_position	337.0±3.16µs	343.9±4.24µs	+2.05%
tokens_at_position	3.6±0.05ms	3.6±0.02ms	0.00%
tokens_for_file	397.2±2.50µs	400.0±1.61µs	+0.70%
traverse	34.0±0.96ms	34.2±0.75ms	+0.59%

[sdankel]

Nice one 👍

[github-actions]

Benchmark for 45418bd

Click to view benchmark

Test	Base	PR	%
code_action	5.1±0.10ms	5.6±0.09ms	+9.80%
code_lens	290.2±21.49ns	341.4±10.12ns	+17.64%
compile	2.1±0.08s	2.2±0.07s	+4.76%
completion	4.8±0.16ms	5.0±0.22ms	+4.17%
did_change_with_caching	1952.0±19.54ms	2.1±0.02s	+7.58%
document_symbol	954.1±46.78µs	870.1±13.75µs	-8.80%
format	83.9±0.78ms	85.0±1.30ms	+1.31%
goto_definition	337.9±6.22µs	343.3±17.11µs	+1.60%
highlight	9.5±0.18ms	8.8±0.13ms	-7.37%
hover	355.8±7.18µs	360.8±20.80µs	+1.41%
idents_at_position	120.2±0.38µs	118.3±0.85µs	-1.58%
inlay_hints	641.2±28.48µs	637.3±11.21µs	-0.61%
on_enter	2.1±0.07µs	1934.5±111.24ns	-7.88%
parent_decl_at_position	4.1±0.09ms	3.7±0.06ms	-9.76%
prepare_rename	339.3±6.71µs	340.6±6.83µs	+0.38%
rename	9.6±0.14ms	9.4±0.23ms	-2.08%
semantic_tokens	1252.4±13.51µs	1302.2±29.01µs	+3.98%
token_at_position	336.5±2.85µs	340.7±3.56µs	+1.25%
tokens_at_position	4.1±0.07ms	3.7±0.09ms	-9.76%
tokens_for_file	409.8±9.57µs	406.8±4.35µs	-0.73%
traverse	35.3±0.56ms	36.6±0.73ms	+3.68%