Optimizing Language Loading

[iamhectorsosa]

Hello, everyone! I'm using shiki to highlight code on the server-side. Here's what I'm doing:

• I create and await single instance of the highlighter by calling getHighlighter: I pass in a theme (from the existing ones), and I pass an array of languages.
• Then I use this instance to highlight all of the code that I need, by calling codeToThemedTokens and
  then renderToHtml.

This works great when we create our project's build. However, I've found that during development, it is incredibly slow. My benchmarks indicate that each page takes around 12 to 15 seconds to load. Each page has around 6 to 12 code blocks to highlight, and each of these code blocks have an HTML and a tsx version.

I ended up opting out of using languages all together to make the DX a bit better. The code comes formatted as usual but without any syntax highlighting.

Any suggestions on how could we could improve this? I would like to provide an alternative to load languages but without having to wait long to work with it during development.

import {
  getHighlighter as getHighlighterFromShiki,
  renderToHtml,
  type Highlighter,
  type Lang,
  type Theme,
} from "shiki";

/** ✅ Config */
const theme: Theme = "github-dark";
const langs: Lang[] = ["html", "tsx"];
const bg: React.CSSProperties["backgroundColor"] = "#011627";

export async function getHighlighter() {
  /** Preload NO languages in development */
  const isDevelopment = process.env.NODE_ENV === "development";

  /* ✅ Create a highlighter instance with a theme */
  return await getHighlighterFromShiki({
    theme,
    langs: isDevelopment ? [] : langs,
  });
}

export async function highlight(
  highlighter: Highlighter,
  code: string,
  lang: Lang = "tsx"
) {
  /** Request NO languages in development */
  const isDevelopment = process.env.NODE_ENV === "development";

  /* ✅ Highlight your code using the right syntax */
  const tokens = highlighter.codeToThemedTokens(
    code,
    isDevelopment ? "" : lang,
    theme
  );
  /* ⚠️ Optional: Custom rendering of code blocks */
  return renderToHtml(tokens, {
    bg,
    elements: {
      pre({ className, style, children }) {
        return `<pre class="${className}" style="${style}">${children}</pre>`;
      },
      line({ children, className, index }) {
        return `<span data-line=${index + 1}
         class=${className}>${children}</span>`;
      },
    },
  });
}

[antfu]

It should be improved a lot with v1.0, #557