[theme] make token colors user customizable

[aeschli]

Like the workbench colors, it should be possible to also extend (or replace) the current token colors.

This could be part of workbench.colorCustomizations or a separate setting editor.tokenColorsCustomizations.

Internally, coloring of tokens is based on textmate coloring rules. As textmate scopes are an advanced topic (what scopes to use where), we should think of also offering a simple mode where users can set colors to predefined concepts like 'comment, 'strings' 'numbers', 'types'....

[mike-lischke]

I like the idea of separating GUI colors and syntax colors. (GUI) Themes would determine the GUI colors and extensions define syntax colors. Of course there should be a default set that can easily be changed by a user (with the possibility to mark a setting as fixed or overridable). If required there could be syntax themes as well, but in any case syntax color should be made language (document type) dependent.

In order to simplify token definition vscode could provide a default repository for syntaxes files. Authors of these files would then only override/add patterns for special tokens but can otherwise the predefined repository tokens in their rules.

GUI themes could be limited to entries in the workbench.colorCustomizations, while syntax themes only affect code editor colors.

[hoovercj]

@aeschli I like the idea of allowing generic "comment", "string", "number", "types" overrides in addition to specific scopes. That will make it more accessible to users for simple overrides without sacrificing the flexibility and power that I'm looking for.

I want to clarify, though: Unlike the workbench customizations, theme customizations need to be able to do more than just override colors, they should be able to override anything that a normal theme can do. (i.e. fontstyle, foreground, background,etc.).

Given that I would recommend calling it editor.themeCustomizations with a format similar to my initial PR #21981, but altered a bit to match how you implemented workbench customizations (no more arrays). This should have a minimal impact on the complexity and performance of the system. Example shown below:

'editor.themeCustomizations`: {
    /* top level customizations that apply to light or dark themes, override simple */
    "scope.name": {
        fontStyle: 'bold'
    }
    'light': { /* only apply to light themes, override top level, override simple */},
    'dark': { /* only apply to dark themes, override top level, override simple */ }
    'simple': {
        'comments': 'red' | { foreground: 'red', fontStyle: 'italics' }
    }
}

Of course, this can be flattened down to settings like: editor.themeCustomizations.scope.name.foreground = red;

I'd have to look into the code again and play around to see if this also plays well with language specific settings which would play nicely with the simple settings. "i want python comments to be red", for example.

[aeschli]

@hoovercj Following what we already have in 'workbench.colorCustomizations' we should just support one set of colors, no variants for 'light' and dark'. This keeps it simpler. The assumption is that most users stick to either light or dark, most often even to a specific theme.
Once customized, one can use the Create theme from user settings command to create a theme out of the current settings.

editor.themeCustomizations is not as good of a name as there also a lot of other editor colors, which we already have in workspace.colorCustomizations.

I'd suggest the following format:

type TokenSettings  = { foreground: string; fontStyle: string };  // background not supported in VSCode

"editor.tokenColorsCustomizations": {
    "comments": string | TokenSettings;
    "strings": string | TokenSettings;
    "numbers": string | TokenSettings;
    "keywords": string | TokenSettings;
    "types": string | TokenSettings;
    "functions": string | TokenSettings;
    "variables": string | TokenSettings;
    "textMateRules": { "scope": string, "settings": TokenSettings, "name": string }[];
]

[khaosdoctor]

Well, since this idea came out of #11556, as I said in this PR, I got this idea from Atom, I really lied the way you could "hack" the editor using simple skills. In Atom we could only edit a simple less file and it would replace the current color setting which was currently being used, like a CSS override.

Then, I'd suggest something like that... There could be a file like tokens.<extension here> where the user could just edit everything related to the tokens using simple CSS and query selectors. That's one idea.

The other is an extension from @aeschli and @hoovercj's idea, but not limiting only to more generic types (such as variables, functions) but going even deeper (but allowing the user to change the "global" type), for instance, we have a simple variable like this:

let variable = 1

This is a variable in a general scope, but we also have a parameter, which is some sort of variable but inside a function:

function fn (params) { // params should have a color
let variable = 1 // Variable should have another
}

Well, I believe you get the point, but there's a lot more stuff we can break this categories into, like:

• Block comments
• Single line comments

Both comments would have a global type: "comment" which the user would change and affect both subtypes listed above. The same applies bellow

• Scope Variables (Global: Variable)
• Global Variables (Global: Variable)
• Parameters (Global: Variable)
• Anonymous functions (Global: Function)
• Arrow functions (Global: Function)
• Async functions (Global: Function)
• Named functions (Global: Function)
• Generator functions (Global: Function)
• Class methods (or simply, functions inside a class) (Global: Function)
• Constructors (Global: Function)
• Constants (Global: Variable)
• Modules and imports (the alias given to a require'd module) (Global: Variable)
• Spread parameters (Global: Variable)
• Template literals
• Simple strings (strings wraped by '' instead of "")
• Double strings (the other way around)
• Promises (Global: Function)
• ....

The list goes forever, and this is only javascript, there could be "token extenders" which would implement the functionality to other languages so the user could only download the extension they need to change.

The problem with Atom was that it allowed us to change literally everything because we could change the global stylesheet of the editor, but there was no grouping or helper to make it a bit easier.

[aeschli]

@khaosdoctor Sure that would be nice if we could support coloring all these concepts. Problem is that we don't get such detailed tokens from our grammars. We are based on TextMate grammars that we get from the community. Read here to get an idea what the different scopes are that textmate grammars generate. BWT also Atom is based on TextMate grammars, so they have the same limitations. So the simple token types (string, number...) are based on scopes used across all languages, that why I suggested them.

[khaosdoctor]

Yeah, they use it too, but there's a difference between how Atom treats the styling of the tokens compared to Code. Instead of having a single config page with predefined configuration keys (like those in workbench), Atom has a user.less file which is loaded over the main editor's stylesheet, this way the users can literally hack the colors by injecting CSS selectors and colors in that less file.

Maybe we could do something close to it, create a CSS file which would be loaded on top of the editor's current stylesheet and overrides the current configs.

[vytautas-pranskunas-]

Is there any progress on this task? It is most desired feature of mine for now :D

[hoovercj]

It is mine, too :-) I should have time next week to do this.

[vytautas-pranskunas-]

Waiting :)))

[hoovercj]

@khaosdoctor user css css files won't work in VS Code due to optimizations they do with themes. The css classes are generated based on the active theme and therefore are not in a friendly or persistent format. Example output:

<span class="mtk9 mtki">function</span>
<span class="mtk1">&nbsp;</span>
<span class="mtk5">f1</span>
<span class="mtk1">()&nbsp;{</span>

You can read more about that here

[khaosdoctor]

@hoovercj Ok, so that's really complicated 😮

So on another thought, can we edit based on TextMate's description? Like bellow:

We would edit based on thar table on this table:

So we would already have a definition for every type of token and the CSS classes based on them:

...
/* Function name */
.entity.name.function { color: #A6E22E; }
...
/* Class name */
.entity.name.class { color: #A6E22E; text-decoration: underline; }
...

There could be another CSS file to "override" theme's default one. Basically when the user tries to open a new setting window (using <F1> for instance) there could be an option "User token settings" where there would be a default model CSS the user would be able to edit (the same the active theme uses, or at least, the same structure and model). And override the current bit settings for the active tokenizer.

[hoovercj]

@khaosdoctor that's roughly the idea, but not in css.

The idea is that your example from above would look like this in user or workspace settings:

"editor.tokenColorsCustomizations": {
    "textMateRules": [{
        "scope": ".entity.name.function", 
        "settings": {
            "color": "#A6E22E"
         }
    },{
        "scope": ".entity.name.class", 
        "settings": {
            "color": "#A6E22E",
            "text-decoration": "underline"
         }
    }]
}

Note: Text decoration will only work if vscode supports it, which I can't check at the moment.

[aeschli]

@hoovercj thanks for the example. It's correct, except that the scope doesn't start with a dot. "text-decoration" is called 'fontStyle'. There's no need to invent anything new here, it's the textmate theme format that you can see in tmTheme file, except that its json instead of XML.
It gives you the full power to match against the text mate grammar trees.
That CSS is used in our UI is an implementation detail that we want to keep internal in order to be able to evolve our UI and implementation.

[aeschli]

implemented by #29393