[Exploration] Tree-sitter tokenization exploration (Fixes #161256)#161479

Closed

aiday-mar wants to merge 64 commits intomainfrom aiday/treeSitterTokenization

+911-4

Commits on Aug 31, 2022

initial skeleton of tree sitter tokenization
aiday-mar
committed
package.json update
aiday-mar
committed

Commits on Sep 5, 2022

Work in progress
aiday-mar
committed
adding wasm file
aiday-mar
committed
Line colorization now works
aiday-mar
committed
Adding more queries into the scm
aiday-mar
committed

Commits on Sep 7, 2022

Asynchronous parsing and rendering
aiday-mar
committed
Change the name of the methods
aiday-mar
committed
Iterating only over the ranges that have changed
aiday-mar
committed
making initial synchronous operation before making it asynchronous
aiday-mar
committed
using FileAccess.asBrowserUri instead of readFileSync
aiday-mar
committed

Commits on Sep 8, 2022

Using the theme data for the colorization
aiday-mar
committed

Commits on Sep 9, 2022

Corrected incorrect colorization
aiday-mar
committed
Incorrect colorization corrected
aiday-mar
committed
Cleaning the code
aiday-mar
committed
Cleaning the code
aiday-mar
committed
Removed TODO
aiday-mar
committed
Adding back the check on the language
aiday-mar
committed

Commits on Sep 12, 2022

Commits on Sep 13, 2022

After cleaning the code and using inheritance
aiday-mar
committed

Commits on Sep 14, 2022

Using single instance of each tree in the tree service
aiday-mar
committed
Cleaning code
aiday-mar
committed
Placing the language setting into the treeSitterService.ts file. Need to also implement a system which keeps track of the changes.
aiday-mar
committed
tree sitter is updated in the onDidChangeContent listener
aiday-mar
committed
Taking into account model removal. Exploring the folding model.
aiday-mar
committed

Commits on Sep 21, 2022

Test file added, comparison md file
aiday-mar
committed
Update tokenization-comparison.md
Aiday Marlen Kyzy
authored
Update tokenization-comparison.md
Aiday Marlen Kyzy
authored

Commits on Sep 22, 2022

Update tokenization-comparison.md
Aiday Marlen Kyzy
authored
Update tokenization-comparison.md
Aiday Marlen Kyzy
authored
Update tokenization-comparison.md
Aiday Marlen Kyzy
authored
Update tokenization-comparison.md
Aiday Marlen Kyzy
authored
Update tokenization-comparison.md
Aiday Marlen Kyzy
authored
Update tokenization-comparison.md
Aiday Marlen Kyzy
authored
Update tokenization-comparison.md
Aiday Marlen Kyzy
authored
Update tokenization-comparison.md
Aiday Marlen Kyzy
authored
Update tokenization-comparison.md
Aiday Marlen Kyzy
authored
Update tokenization-comparison.md
Aiday Marlen Kyzy
authored
Update tokenization-comparison.md
Aiday Marlen Kyzy
authored
Update tokenization-comparison.md
Aiday Marlen Kyzy
authored
Update tokenization-comparison.md
Aiday Marlen Kyzy
authored
Update tokenization-comparison.md
Aiday Marlen Kyzy
authored
Adding again the tree-sitter library
aiday-mar
committed
Merge branch 'private/treeSitterTokenization' into aiday/treeSitterTokenization
aiday-mar
committed
Removed the md file since the analysis is placed inside of the PR
aiday-mar
committed
Cleaning the code
aiday-mar
committed
updating yarn.lock
aiday-mar
committed
removing all code relative to tree-sitter folding exploration
aiday-mar
committed
Removed the pre-commit
aiday-mar
committed
Removing the comments on current tokenization system. The tree-sitter implementation now sets the tokens to an empty array before assigning the new values.
aiday-mar
committed

Commits on Sep 23, 2022

Code added in order to meare the performance of the tree-sitter tokenization system
aiday-mar
committed
Setting the tokens at the end of every asynchronous call and setting them until the end of the file
aiday-mar
committed
Cleaning the code
aiday-mar
committed
Removing the --no-verify flag
aiday-mar
committed
Disable the local code import pattern for the Parser
aiday-mar
committed

Commits on Sep 28, 2022

Merge branch 'main' into aiday/treeSitterTokenization
aiday-mar
committed