Skip to content

Conversation

@Ayush2006128
Copy link

Added simple tokenizer to implement basic syntax highlighting respecting the simplicity of the project

Screenshot 2025-11-26 175716 Screenshot 2025-11-26 175933

Features:

  • Improves UX or DX
  • It's generic which means works on any language
  • Simple & performant

Note: Currently it only highlights basic things like numbers, identifier, strings, comments // or # and symbols.

This is an experimental feature provide --enable-exp-highlighting flag to enable

@Ayush2006128
Copy link
Author

It was a fascinating experience working on this project

@lhecker
Copy link
Member

lhecker commented Nov 26, 2025

Your implementation is a classic tokenizer. Unfortunately, we can't merge it because we would need flexible support for more languages. I'm super happy you liked experimenting on the project though!

(FYI I'm also working on this right now: #624 (comment))

@Ayush2006128
Copy link
Author

@lhecker I was going to do the same, but wouldn’t that make it more complex and slow down the app? 🤔 Here’s a new solution: we could add language-specific highlighting only for languages like HTML, XML/XAML, YML/YAML, CSS/SCSS, and TOML. For other languages, we could have the tokenizer handle basic elements like booleans or simple keywords using LSPs. 🤷

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants