[themes] Explore theming support for semantic colors

[aeschli]

• define a set of Token Classifications that can be used for semantic highlighting
• evaluate actual color/style values based on the current text mate theme
• allow themes to define such colors and styles

[aeschli]

related: #32813

[kanlukasz]

* allow themes to define such colors and styles

Do you mean that there will be option to set color variables and use them with themes making?
I mean something like in Sublime Text

Here is example what i talking about:
https://stackoverflow.com/questions/56473901/is-it-possible-to-use-color-variables-in-the-visual-studio-code-when-creating-th

Or maybe it's possible right now but i don't know how?

[aeschli]

This is a proposal on how to represent the classification that a semantic highlighter computes (TokenTypes and TokenModifiers) and how themes and users can associate the style (TokenStyle). I'm still editing this proposal, but feedback and questions are welcome!

Semantic highlight range provider

A semantic highlight range provider takes a document (and potentially other arguments such as the current viewPorts) and returns a list of classified source ranges. The proposal is that each range can be classified with a TokenType and any number of TokenModifiers.

interface SemanticHighlightRangeProvider {
   provideHighlightRanges(doc,...): [ Range, TokenType, TokenModifier[] ][];
}

Note: This proposal focuses just on the classification types. The full SemanticHighlightRangeProvider API with a proper return type and optimizations such as compact token representation or ranges grouped by line will be part of a different proposal.

TokenTypes and TokenModifiers

A TokenType describes the symbol (class, field, variable...) at the location. The TokenModifiers provide additional information on the symbols (static, declaration, write access...).
TokenType and TokenModifiers are identified by a string id.

The purpose of splitting the classification into type and modifier is to simplify the theme definition (described later) and to make it easier for extensions to contribute additional classifications.

Predefined TokenTypes and TokenModifiers

VSCode will define a set of predefined (standardized) TokenTypes and TokenTypeModifiers, but extensions can add more if necessary:

Predefined TokeTypes:

• comments, strings, numbers, keywords, regexp, operators

• namespaces

• types, structs(Type), classes(Type), interfaces(Type), enums(Type), parameterTypes(Type)

• functions, macros(Function)

• variables, constants(Variable), parameters(Variable), properties(Variable),

• labels

Predefined TokeTypes Modifiers:
declaration : For all symbol declarations
documentation: For all occurrences of symbols in documentation
member: for member types, member variables (fields) and member functions (methods)
static for static (member types), member variables and member functions
modification for references where a variable is modified
async for declarations or references for functions that are async
deprecated: for annotating deprecated symbols
abstract: for annotating abstract functions.

TokenTypes and TokenModifiers Defaults

Each TokenType and TokenModifier come with a default style which is used when the actual theme does not define a style for it.

The default style can be specified as either

• a set of textMateScopesToProbe that are ran against the text mate theme rules the current theme defines to find a matching style.
• a predefined (static) style
• a reference to a different token-type to inherit the style of the other token type
• a function of one or more token-types

interface TokenTypeRegistry {
    registerTokenType(id, default: { textMateScopesToProbe: string[], value: TokenStyleValue })
}

type TokenStyleValue: = TokenStyle | TokenType | FunctionOf(TokenStyleValue)

Theme Definition

The active color theme (or user setting) will define colors and styles (TokenStyle) for TokenTypes and TokenModifiers or combination thereof.

interface TokenStyle {
   foreground?: Color | ColorFunction
   style?: bold | italic | underline
}

{
   "tokenStyles": {
       "(typenType|*)[.tokenTypeModifier]*": TokenStyle,
       //...
    }

• The first segment of a rule is the tokenType or '*', followed by modifiers. The order of the modifiers is not relevant.
• A rule matches if the token-type (e.g. const) matches or if the rule starts with * and when all modifiers of the rule are matched.
• The rule specificity is defined by the number of segments matching. That means
  const < *.declaration < *.declaration.exported < const.declaration.exported.
  (still to be decided: order when multiple rules have the same number of segments matching)
• The final style can be the result of multiple rules. For each style property (foreground, underline, bold, italics) the style of the most specific rule is applied. If a rule does not define a style, the rule is ignored for the evaluation of that style.
• The foreground color can be a combination of colors if ColorFunctions ('darker()') are used.

{
        "tokenStyles": {
            // all consts are blue and underlined
            "const": { "foreground": "#0000ff", "underline": true },
            // all declarations (types, functions, vars,....) are bold
            "*.declaration": { "bold": true },
            // all declaration that are also exported (public) are not bold, but underlined
            "*.declaration.exported": { "bold": false, "underline": true },
            // consts declarations that are exported are bold, underlined (inherited) and blue (inherited)
            "const.exported.declaration": { "bold": true },

            // all async decls and refs are rendered with a slightly darker color
            "*.asyncReturn": { "foreground": "darker()" }
        }

	"colors": { ... /* same as before */ },
	"tokenColors": [ 
                // still there for syntax highlighting and for evaluating 
                // default styles for TokenTypes and Modifier
		{
			"scope": [ "storage.type.annotation" ],
			"settings": { "fontStyle": "bold", "foreground": "#AA3731" }
		},
		{
			"scope": [ "entity.name.type", "entity.name.namespace" ],
			"settings": { "fontStyle": "bold", "foreground": "#7A3E9D" }
		}
		// ...
	]
}

TokenTypes and TokenModifiers Contribution

{
   "contributes": {
       "tokenTypes": [ {
           "id": "annotationType",
           "default": {
               "textMateScopes": ["storage.type.annotation"],
               "value": "type" // inherits settings from 'type'
           }
       }],
       "tokenTypeModifier": [ {
           "id": "autoBoxed",
           "default": {
               "value": { "bold": "true" }  // default to be used for '*.autoboxed'
           }
       }]
}

[NTaylorMullen]

Are there any thoughts on how this could work for semantically embedded languages? Aka, HTML with JS/CSS or Razor with HTML/JS/CSS.

For instance in Razor if you do:

<form asp-antiforgery="true">
...
</form>

The true can be C#.

It'd be great if there was a way to say "this range is C#" and then adding additional semantic meaning on top of it would be up to Razor or HTML language servers or their extensions

[AmadeusW]

This might be related to embedded laguages; will we support multiple tokens per location? How about multiple token modifiers per location?
For example, static function or keyword in embedded XML (doc comment) are both used in C#.

[aeschli]

@NTaylorMullen @AmadeusW That needs to be seen when we finalise the SemanticHighlightRangeProvider. Normally with providers we support more than one provider.

[NTaylorMullen]

@NTaylorMullen @AmadeusW That needs to be seen when we finalise the SemanticHighlightRangeProvider. Normally with providers we support more than one provider.

Definitely, but what about registering a "language" with a range semantically?

[aeschli]

@NTaylorMullen This needs to be discussed in the issue where we design the SemanticHighlightRangeProvider. Maybe you can file an issue where you describe the idea?

[NTaylorMullen]

@NTaylorMullen This needs to be discussed in the issue where we design the SemanticHighlightRangeProvider. Maybe you can file an issue where you describe the idea?

Done: #81558

[HighCommander4]

Just wanted to mention that the token type / token modifier design addresses the concerns about having a flat list of highlightings that I described here. Thanks!

One minor comment is that e.g. the function/method and variable/field token types could be folded into one if we have a member token modifier.

A question abut styling: would it make sense to allow including a background color in TokenStyle? For C++ we have been experimenting with using semantic highlighting to grey out inactive code branches (code that is #ifdef'ed out). It's not fully clear whether semantic highlighting is the best fit for this, but if so it would require being able to specify a background color.

[aeschli]

Thanks for the feedback @HighCommander4 . The member modifier suggestion makes sense.
We don't support background colors for tokens in VSCode, that's why it's currently not part of TokenStyle. The request is #3429.

[sharedprophet]

This looks like good progress. However, in order to port semanticolor to VS Code, I need the ability either for the theme to determine color based on the token value (and modifiers), or for a language server to be able to provide actual style information. Would it be possible to add this kind of support?

[aeschli]

@sharedprophet Theming (determine the style based on the token type/modifiers) will be the responsibility of VSCode. I've started working on that. The language server just needs to provide ranges with types and modifiers.

[ryantheleach]

+1 for having support for Token Value. I'd love semantic coloring that could color the accessibility keywords based on visibility.

E.g. bold and loud for public, and quiet and invisible for private.
Or coloring based on whether it's exposed in a super class or interface.

Would help encourage people to consider their visibility more, rather then default it to public.

Would also love to be able to color C# Attributes, so that ASP.net HTTP stateful attributes are red, where as stateless green.

[sharedprophet]

@ryantheleach all the things you mentioned require language specific semantic knowledge and could be done in language servers without giving the token value to the theming layer.

[ryantheleach]

Whilst true, I think there is always going to be a point where someone is going to want something based on the token value, or it's easier to implement it at a level the user can easily customize, like my asp.net stateful/stateless http method attributes. as far as C# is confirmed they are all identical types and modifiers right? I'd foresee for that level of customizability, that as a user I'd need to define overrides for an attribute with a specific value.

[aeschli]

https://github.com/microsoft/vscode/wiki/Semantic-Highlighting-Overview has a summary of the implemented semantic highlight APIs and user settings.

Semantic highlighting for JS in HTML and for TypeScript has been added. Work has started for Java and C++ and a semantic highlight LSP has been proposed.

There's still lot of work to be done, but I close this issue as the Exploration is complete.

[sharedprophet]

It still lacks a way for token theming to be modified based on the text of the token...

[NonLogicalDev]

If I understand correctly this will still not let the identifiers to be consistently coloured—not by their role—but by their value.

It would be great for VSCode to implement something akin to what IntelliJ does.
Just give users a way to specify gradient of colours (let the user define a finite set of colours for semantic colourisation), have language server return hash of the identifier and place identifier somewhere on the gradient based on its hash.

Examples:

• https://www.jetbrains.com/help/idea/configuring-colors-and-fonts.html#semantic-highlighting
• https://github.com/jaxbot/semantic-highlight.vim

Having identifiers coloured by their attributes is not quite as useful as having them coloured by their value.
The reason for that is that having identifiers coloured by their value lets you visually see where variables/functions/methods are used in the code.
It also makes it dead simple to identify typos, (identifier will end up having a wrong colour).

I know it sounds silly, but not having this feature pushes me away from using it, because it helps immensely to navigate code faster, and helps me with my mild case of dyslexia.

I keep coming back to this issue though. Hoping that this will one day get implemented, and I will finally feel compelled to give VSCode a try as an IDE, as opposed to just another nice text editor for a quick simple change or two.

[mattacosta]

@sharedprophet @NonLogicalDev

This is from a quick and dirty example I was doing for another issue but, have a look at the parameters. Is there something I'm missing? It seems to be possible.

[sharedprophet]

@mattacosta that looks good. How did you do it?

[mattacosta]

Basically I had my extension provide a set of default colors for certain modifiers and then had my server add one of those modifiers to each parameter.

[sharedprophet]

@mattacosta but can it work with other language servers? Can your language server act as an intermediary to other language servers so it is lanuguage agnostic?

[mattacosta]

Not sure if I follow exactly, but if you were able to ask those servers for a list of unique symbols in a file, then yes?

[sharedprophet]

I'd like to be able to implement a solution that works for any language without changing the language servers for those languages.

[aeschli]

I'd like to be able to implement a solution that works for any language without changing the language servers for those languages.

There's no need to be part of the language server. Any extension can contribute a semantic token provider, for any language. @alexdima What we still have to work on is to decide what happens when there are multiple providers for a file (which one wins?).

[mattacosta]

@sharedprophet Previous answer still applies. A highly theoretical yes.

| vscode | -- semanticHighlights --> | semanticolor ext | -- documentSymbols --> | other LS |
                                                           <-- SymbolInfo[] --
                                                         -- documentHighlights -->
              <-- highlights[] --    apply coloring logic   <-- Location[] --

I also want to emphasize that just because you can doesn't mean you should... that first run would be hella expensive (if it works at all), and I have no idea what kind of optimizations would be possible/required.

[matklad]

@aeschli the wiki says

Themes and user settings can define rules to assign classifications to styles (foreground, italic, underline, bold)

Is this actually implemented for themes yet?

I've tried adding

{
	"name": "Pale Fire",
	"type": "dark",
	"editor.tokenColorCustomizationsExperimental": {
		"enumMember": "#ff0000"
	},
	"tokenColorCustomizationsExperimental": {
		"enumMember": "#ff0000"
	},
}

to my theme and it doesn't work. Adding this to user settings does work though.

I know I can style things by mapping semantic topics to textmate scopes and then styling the latter as usual, but that's a very roundabout way to do this.

[aeschli]

The property in the theme is called tokenStylingRules. It's not yet documented and might be changed.

[matklad]

Hm, doesn't seem to work:

matklad/pale-fire@01bca94

λ code-insiders --version
1.43.0-insider
b0be0672c210f1fa76109ce675365bb785657e84
x64

[mattacosta]

If a theme has a tokenColors key, it looks like the _loadColorTheme function returns before ever reading tokenStylingRules.

vscode/src/vs/workbench/services/themes/common/colorThemeData.ts

Lines 566 to 580 in fc9543a

	let tokenColors = contentValue.tokenColors;
	if (tokenColors) {
	if (Array.isArray(tokenColors)) {
	result.textMateRules.push(...tokenColors);
	return null;
	} else if (typeof tokenColors === 'string') {
	return _loadSyntaxTokens(extensionResourceLoaderService, resources.joinPath(resources.dirname(themeLocation), tokenColors), result);
	} else {
	return Promise.reject(new Error(nls.localize({ key: 'error.invalidformat.tokenColors', comment: ['{0} will be replaced by a path. Values in quotes should not be translated.'] }, "Problem parsing color theme file: {0}. Property 'tokenColors' should be either an array specifying colors or a path to a TextMate theme file", themeLocation.toString())));
	}
	}
	let tokenStylingRules = contentValue.tokenStylingRules;
	if (tokenStylingRules && typeof tokenStylingRules === 'object') {
	result.stylingRules = readCustomTokenStyleRules(tokenStylingRules, result.stylingRules);
	}

[aeschli]

Ok, sorry, color style customization in themes in not ready yet to be used yet. I'll complete that in the March milestone.