Question: Recommendations on how to handle dark mode

[orta]

👋 I'm thinking about how to handle dark and light themes, microsoft/TypeScript-Website#342 eventually I'm going to need to figure out how to make a dark mode which uses css classes because these poor dark mode folks have the same code color schema:

So, I'm wondering what you think might be the best way to do it (and whether it's something I should do upstream back here)

I have two ideas:

• Have a single theme, convert the colors hex to css classes
• Try match two separate themes together

[giovanicascaes]

Does your first idea mean using a single style that looks good both in light and dark mode?

Also, I'd like to help implementing this, is it something lower in complexity?

[orta]

I think of it as being:

• Shiki returns "#ffeeaa" for a token
• My app makes a sha of it: "#ffeeaa" -> "asd12"
• My app sets the class of the token to be "asd12"
• We keep track of all colors which were assigned and spit out a single css file for all those colors
• That CSS file can use prefers-color-scheme: dark to manually set colors for the dark theme

[giovanicascaes]

It is not an elegant solution but code containers could be dark even in light theme. Check Dan Abramov's overreacted.io (https://overreacted.io/goodbye-clean-code/, for example). It looks good for me to read, both in light and dark theme. That seems the easiest possible solution. At least could bring some fresh to the eyes at night until the ideal solution could come up.

[orta]

Yep - that's a reasonable workaround to not have this problem. That's totally doable for people, but I'm not planning on making that design compromise

[giovanicascaes]

Yes, I understand. I'm not proposing defining as design choice, but it is a better solution than the current one. I believe devs feel more comfortable reading dark code during the day than light code at night.

[naiyerasif]

I don't think it is practical enough to support CSS properties-backed dark and light theme, given the size of grammar and sheer number of color tokens. What I'd suggest is to customize some color tokens (like background-color and foreground-color) through CSS properties and toggle them through a feature query. You'll have to find a theme which has got tokens that work nice enough in light and dark mode (which is annoyingly hard). I was able to do that with Prism (an example here), given its very limited set of tokens but gave up on Shiki.

[orta]

@pveyes solved this by switching the output to use css variables:
https://fatihkalifa.com/typescript-twoslash

[pveyes]

Hi, upon closer inspection I found out that we can get token.explanation when using highlighter.codeToThemedTokens. I think we can implement dark/light mode support purely in user space without modification in shiki. CMIIW

I'm not really familiar with how vscode textmate parses grammar and the resulting type so here's my assumption. We can find what scope matches inside token.explanation[0].scopes

type Scope = {
  scopeName: string;
  themeMatches: Array<ThemeMatch>
}

const token = highlighter.codeToThemedTokens(code, lang)

const matchingScope: Scope = token.explanation[0].scopes.find(scope => {
  return scope.themeMatches.length > 0
});

If empty we can use --fallback-color, or other convention

if (!matchingScope) {
  return "--fallback-color"
}

Even though technically we can generate CSS variable using scope.scopeName, I think it's better to use themeMatch to reduce the number of generated CSS variable.

I found that 0 index is the one that's being used if there's more than 1 themeMatches

type Theme = {
  name: string;
  scope: Array<string>;
}

const matchingTheme: Theme = matchingScope.themeMatches[0]

// we can then generate CSS variable from either name or scope
// remove whitespace, convert invalid character, etc
const cssVar = convertToCSSVariable(matchingTheme.name)

// or use scope
// join all scope, compute hash
const cssVar = convertToCSSVariable(matchingTheme.scope);

return cssVar

Then we can generate HTML tag and use CSS variable fallback.

const cssVar = generateCSSVariable(token);
const html += `<span style="color: var(${cssVar}, ${token.color})">${token.content}</span>`

[Gerrit0]

So, dark mode alone wasn't quite sufficient for me. I wanted the color switching with themes that @pveyes built, but without being locked into a specific theme / set of themes. I'd love to have readable class names, but after messing around with scopes as suggested above, was unable to come up with a good solution and have settled for just generating hl-1, hl-2...

Leaving this here as it might be useful to someone - https://gist.github.com/Gerrit0/275a4b8ffee4fa133fd075f5edeb3cda. TypeDoc will use a similar approach in the rebuilt themes.

[octref]

I need to read up dark mode a little bit first, so this won't cut it for 0.2.0.

[octref]

Currently renderers have no access to theme data. I plan to do an API change:

ThemedTokenizer: theme + lang => IThemedTokens[][]
Renderer: IThemedTokens[][] + theme => HTML / SVG / HTML+CSS

This way a renderer has all needed info to output HTML+CSS (with meaningful class names).
We could have HTMLRenderer, SVGRenderer and HTMLCSSRenderer, etc.

HTMLCSSRenderer could generate a CSS mapping such as:

.support-function {
  color: #fff;
}

And give each token its matching scope as class.

Would this work for you?

[Gerrit0]

That would be awesome. For the CSS mapping - how static would these classes be across different themes? For my use case, I want to be able to generate links for some identifiers. Right now, to get highlighting I'm generating a string that looks like TypeScript, getting the tokens from that, and matching the text of tokens against the identifiers I expect. It would be neat to avoid the extra pass to Shiki. https://github.com/TypeStrong/typedoc/blob/library-mode/src/lib/renderer/default-templates.tsx#L539-L591

[octref]

how static would these classes be across different themes

Each token can have multiple scopes, and it's up to the theme to decide which one it would colorize.
In the API, we could also allow multiple themes, and write the matching scopeNames by each theme into each token.
So a token could look like <span class="a b">, and the output CSS would include:

.github-dark {
  .a { }
}
.github-light {
  .a { }
}

Although that would take a larger refactor. ThemedTokenizer really becomes Tokenizer without theming, and there's a ThemeMatcher (in renderer) that assigns color to each token. That would also mean we need to fork vscode-textmate.

I want to be able to generate links for some identifiers.

That's a separate feature, isn't it?

[codepunkt]

I've been doing something like this on my new work-in-progress blog using gatsby-remark-vscode, which is kinda similar, but comes with it's own set of issues - so i've been looking into using shiki instead.

What is done there - and what works really well, is @orta's second idea: Matching two seperate themes together. One is applied in light mode, the other in dark mode.

[Gerrit0]

In the API, we could also allow multiple themes, and write the matching scopeNames by each theme into each token.

This sounds like what I'm looking for.

That's a separate feature, isn't it?

Kind of - I already have it working by using the tokens myself, I'm just looking for a better way of choosing the CSS classes + CSS generation for each token. Really it's just a reason why I will still use tokens, not the codeToHtml, even if codeToHtml supports light/dark mode.

[octref]

https://github.com/anotherglitchinthematrix/monochrome could be a good test.

I could generate 10 different variations and make a slider demo.

[orta]

We are currently inverting and colour hue shifting on the TypeScript website, and I think that's probably enough for us - microsoft/TypeScript-Website#1536

[hipstersmoothie]

@orta's approach is an okay solution but I don't think it works for every theme. Works well on the TS website though. My ideal API would be the following:

const highlighter = await shiki.getHighlighter({
  theme: 'github-light',
  darkTheme: 'github-dark',
  langs: [...BUNDLED_LANGUAGES, ...langs]
})

As a user I want to be able to pick the theme that best fits my website for light/dark mode.

[hipstersmoothie]

I think I've found a pretty good solution. I'm editing rehype-shiki to support a darkTheme option.
That option will make the rehype plugin generate 2 separate code blocks using shiki, one for light mode and one for dark mode with a class that reflects that (ex: syntax-dark).

Then you just add the following css to hide it based on a media query. You could also do class based dark theming, it's up to the user.

.syntax-dark {
  display: none;
}

@media (prefers-color-scheme: dark) {
  .syntax-light {
    display: none;
  }

  .syntax-dark {
    display: block;
  }
}

[orta]

Hah, nice idea

I keep feeling like all of our answers live outside of 'shiki' and in whatever shiki -> x rendering tool you're using

[octref]

I built a playground with dark mode. You can give it a try at https://shiki-play.matsu.io. Source code is at https://github.com/shikijs/shiki-playground

Here are my thoughts on different approaches:

• Outputting two HTML blocks is actually the simplest way as @hipstersmoothie suggested.
• Most themes have a corresponding dark/light theme. I think it's easier for 99% of users to just use the provided counterpart theme, instead of messing with CSS variables and coming up with other colors.

Outputting semantic HTML (meaningful class names) with CSS

Given a source, a grammar and a theme, you can get the matching scope that makes a token a specific color. For example, here the token is #9ecbff because it's string.

So we need a renderer that does this:

• For each token with a matching scope foo.bar that determines its color
• Give it a class foo-bar (convert all scopes to classes would be too verbose)
• Output CSS, matching each scope to a single color.

Note to myself - on the conversion:

If a theme colorizes string and string.quoted differently, I should generate classes like string and string-quoted, but never string quoted. The CSS should use selectors like .string-quoted, not .string.quoted. Textmate themes have different order and specificity than CSS, so I'd want to avoid getting tangled in the conversion.

API wise, something like this:

interface SemanticHTMLCSSRenderer {
  generateSemanticHTML(code: string, lang: Lang): string
  generateCSSFromTheme(theme: Theme): string
}

[orta]

My gut says the simplest API is we allow theme to be either a string or string[], then give each codeblock a css class with the theme name in it. Then people can use the CSS @hipstersmoothie mentioned.

[orta]

I also joined the club of "render many times" with remark-shiki-typescript microsoft/TypeScript-Website#1831

It can render multiple copies of the code, and then it's on the user to write the CSS which removes the specific theme

Light / Dark Modes

If you pass more than one theme into themes then a codeblock will render for each theme into >
your HTML. This means that you can use CSS display: none on the one which shouldn't be seen.

const jsx = await mdx(content, {
  filepath: "file/path/file.mdx",
  remarkPlugins: [[remarkShikiTwoslash, { themes: ["dark-plus", "light-plus"] }]],
})

@media (prefers-color-scheme: light) {
  .shiki.dark-plus {
    display: none;
  }
}

@media (prefers-color-scheme: dark) {
  .shiki.light-plus {
    display: none;
  }
}

[antfu]

For the record, I made markdown-it-shiki using the similar approach of rendering twice:

https://github.com/antfu/markdown-it-shiki#dark-mode

[orta]

Yes, I think we should ship this theme in Shiki - perhaps simply called css-variables.

The CSS will need to go into shiki docs somewhere. I'm not sure where @octref feels WRT a user-facing docs site (e.g. like the shiki-twoslash one, but for now I think this can be put in the docs folder of this repo.

[FredKSchott]

Great, PR added: #212

[Enter-tainer]

#212 is great! Is is possible to get required CSS for specific theme?

[naiyerasif]

@Enter-tainer Take a look at docs here: https://github.com/shikijs/shiki/blob/main/docs/themes.md#theming-with-css-variables

[octref]

Hey @FredKSchott, great work! While I'm late to review the PR, since we are still pre 1.0 I'd like to get the API right. IMO hiding this functionality behind a special theme is not as good as having an explicit API.

I think what you have is a good start. What I also want to see are:

• A standalone API in addition to codeToHtml. Adding option to codeToHtml is not good since theme is uesless in that case. Not sure what's a good name. Maybe codeToParameterizedHtml or codeToHtmlWithCssVariables?
• A way for users to get tokens with groups, similar to codeToThemedTokens but outputting token group instead of color

[orta]

I'm a bit wary about having this as a separate API codepath, interested to see how it turns out. Having a different API means that higher level abstractions like remark-shiki would need to be aware of this and expose new config vars for what is essentially a custom theme output

[octref]

@orta That's a valid concern. However I'm thinking about additional use cases like this. So far what we have are:

code + theme + grammar = themedTokens
themedTokens + renderer = html/svg/etc...

This assumes user want the tokens already colorized, and the only way for them to tweak the coloring pipeline is to write a theme. What we could also have is:

code + grammar = groupedTokens

Essentially what highlight.js does. People who want to write their own colorizer can use this API. It's also not that much – just mapping functions/keywords etc to a color and you are done.

I also feel codeToHtml should output directly embeddable HTML, not "given an option and then you need to fill in some CSS" kind of HTML.

[OskarGroth]

I'm trying out the css-variables theme and I see several issues with this approach:

No clear API, functionality is triggered by a theme name ('css-variables')
Limited to only 12 colors
No control over color names. I'm forced to use colors like shiki-token-punctuation for something semantically different to make use of all 12 colors.

Instead of the hacky and limited css-variables theme, why not just enable the user to supply their own color remap here:

shiki/packages/shiki/src/highlighter.ts

Line 67 in 4478d64

const COLOR_REPLACEMENTS: Record<string, string> = {

and instead of the name check here

shiki/packages/shiki/src/highlighter.ts

Line 99 in 4478d64

if (_theme.name === 'css-variables') {

, add some new option on ShikiTheme like cssVariables: boolean?

Seems to me that by doing this it would unlock full theming via CSS Variables and resolve all dark-mode issues, including my problems listed above. Users would be able to swap out any colors they'd like for a CSS variable instead.

[OskarGroth]

Also the COLOR_REPLACEMENT map even contains errors, there is no #3 entry...

shiki/packages/shiki/src/highlighter.ts

Line 69 in 4478d64

'#000002': 'var(--shiki-color-background)',

[ng-hai]

As an innocent user, I thought I could do this on my-theme.json

{
  "tokenColors": [
    {
      "scope": ["comment"],
      "settings": {
        "foreground": "var(--color-muted)",
        "fontStyle": "italic"
      }
    },
    {
      "scope": ["constant"],
      "settings": {
        "foreground": "var(--color-pine)"
      }
    },
    {
      "scope": [
        "constant.numeric",
        "constant.language",
        "constant.charcter.escape"
      ],
      "settings": {
        "foreground": "var(--color-rose)"
      }
    },
  ]
}

So when Shiki processing, it will inject that value to token style <span style="color: var(--color-muted)">variable</span> 😓

[ng-hai]

I think I've done it, not only dark mode but also multiple themes. Inspired by rehype-pretty-code and Fatih Kalifa’s blog post.

Screen.Recording.2022-05-13.at.19.09.29.mov

[antfu]

I think maybe I found the most balanced way of doing so. You can learn more about the details at antfu/shikiji#5

Basically, it generated inline CSS variables like:

<span style="color:#1976D2;--shiki-dark:#D8DEE9">console</span>

That can be overridden with a short CSS snippet:

@media (prefers-color-scheme: dark) {
  .shiki span {
    color: var(--shiki-dark) !important;
  }
}

live preview

This way we will have a perfectly working light mode output, while being able to switch to dark mode conditionally, without duplicating the content.

Would be happy to port it back to Shiki if you think this approach makes sense.

[orta]

Yeah, I think this answer is the right one - and should probably be canonically the answer

Something I'm not certain about (from a high level) is what might happen if the two themes have different support for coloring source attributes, e.g.

I < 3

might generate something like

  <code>
    <span class="line">
      <span style="color:#1976D2;--shiki-dark:#D8DEE9">I</span>
      <span style="color:#6F42C1;--shiki-dark:#ECEFF4"><</span>
      <span style="color:#6F42C1;--shiki-dark:#88C0D0">3</span>
    </span>
 </code>

Is it possible that there could be inconsistencies in the theme in terms of the tokens <=> colors they support, making something like:

  <code>
    <span class="line">
      <span style="color:#1976D2;--shiki-dark:#D8DEE9">I</span>
      <span style="color:#6F42C1;"><</span>
      <span style="color:#6F42C1;--shiki-dark:#88C0D0">3</span>
    </span>
 </code>

possibly happen?

[Gerrit0]

Yes, it's absolutely possible for that to happen. TypeDoc's implementation of this uses a single class for each color, and overrides what the class does, so I've seen this quite a lot when debugging, even when highlighting with "sister" themes like Light Plus/Dark Plus

[antfu]

@orta I think shikiji handles it correctly. For tokens without colors, inherit will be rendered:

<code>
  <span class="line">
    <span style="color:#1976D2;--shiki-dark:#D8DEE9">I</span>
    <span style="color:#6F42C1;--shiki-dark:inherit"><</span>
    <span style="color:#6F42C1;--shiki-dark:#88C0D0">3</span>
  </span>
</code>

Also it would be cases some theme output larger token, like:

<code>
  <span class="line">
    <span style="color:#1976D2;">I < 3</span>
  </span>
</code>

In that case, shikiji will break those tokens into the common set of two themes.

So far I think it covers most of the edge cases.

[orta]

👍🏻

[Tachi107]

Since nobody mentioned this already: using inline html style attributes is somewhat undesirable, since it requires setting style-src: 'unsafe-inline' Content Security Policy source.

As far as I understand, the two approaches proposed in this thread rely on HTML classes and CSS variables, respectively. Since they both require the use of a separate CSS stylesheet, I'd argue that using HTML classes is better since it doesn't have the aforementioned CSP issue.

Am I missing something? Please let me know!

Thanks for your awesome work on Shiki :D