Loading All Language Lexers Even When Only One is Needed

[Innei]

Validations

• Follow our Code of Conduct
• Read the Contributing Guide.
• Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.

Describe the bug

Description

I am using Shiki's code highlighting library to implement code splitting and lazy loading for language lexers and themes. However, despite specifying only one language in the configuration, Shiki is loading all the configured language lexers, leading to unnecessary loading of resources.

Steps to Reproduce

Set up the getHighlighterCore with multiple languages configured as lazy-loadable modules.
Use codeHighlighter function specifying only one language (e.g., 'typescript') to highlight.
Observe that all configured language modules are loaded, not just the one required.

async function main() {
  const codeHighlighter = await (async () => {
    const [{ getHighlighterCore }, getWasm] = await Promise.all([
      import('shiki/core'),
      import('shiki/wasm').then((m) => m.default),
    ])

    const shiki = await getHighlighterCore({
      themes: [
        import('shiki/themes/github-light.mjs'),
        import('shiki/themes/github-dark.mjs'),
      ],
      langs: [
        () => import('shiki/langs/javascript.mjs'),
        () => import('shiki/langs/typescript.mjs'),
        () => import('shiki/langs/css.mjs'),
        () => import('shiki/langs/tsx.mjs'),
        () => import('shiki/langs/jsx.mjs'),
        () => import('shiki/langs/json.mjs'),
        () => import('shiki/langs/sql.mjs'),
        () => import('shiki/langs/rust.mjs'),
        () => import('shiki/langs/go.mjs'),
        () => import('shiki/langs/cpp.mjs'),
        () => import('shiki/langs/c.mjs'),
        () => import('shiki/langs/markdown.mjs'),
        () => import('shiki/langs/vue.mjs'),
        () => import('shiki/langs/html.mjs'),
        () => import('shiki/langs/asm.mjs'),
        () => import('shiki/langs/shell.mjs'),
        () => import('shiki/langs/ps.mjs'),
      ],
      loadWasm: getWasm,
    })

    return (o: { lang: string; attrs: string; code: string }) =>
      shiki.codeToHtml(o.code, {
        lang: o.lang,
        theme: 'github-light',
      })
  })()

  document.querySelector<HTMLDivElement>('#app')!.innerHTML = `
  <div>
   ${codeHighlighter({
     lang: 'typescript',
     attrs: '',
     code: `import typescriptLogo from './typescript.svg'`,
   })}
  </div>
`
}
main()

For the code above, I registered many languages, but I only used one.
But it will download all the js files

Expected Behavior

Only the lexer for the specified language should be loaded. For example, if 'typescript' is specified, only the Typescript lexer should be dynamically imported and used.

Or I hope shiki can add automatic language recognition and download lexical js files like prism.

PrismJS/prism#1313 (comment)

Actual Behavior

All lexers specified in the getHighlighterCore setup are being loaded regardless of the language specified in codeHighlighter.

Reproduction

https://github.com/Innei/shiki-bundle-and-import-repro

Contributes

• I am willing to submit a PR to fix this issue
• I am willing to submit a PR with failing tests

[antfu]

Yes because highlighting is sync (required in many integrations), while the loading is async, so it has to be done ahead of time.

If your integration are acceptable for async context, you could do loading yourself

if (!shiki.getLanguages().includes('typescript'))
  await shiki.loadLanguage(import('shiki/langs/typescript.mjs'))

const html = shiki.codeToHtml(code, { lang: 'ts' })

[Innei]

Thank you for your response.
I am using this method now and he is working fine.

  use(
    useMemo(async () => {
      async function loadShikiLanguage(language: string, languageModule: any) {
        const shiki = codeHighlighter?.codeHighlighter
        if (!shiki) return
        if (!shiki.getLoadedLanguages().includes(language)) {
          await shiki.loadLanguage(await languageModule())
        }
      }

      const { bundledLanguages } = await import('shiki/langs')

      if (!language) return
      const importFn = (bundledLanguages as any)[language]
      if (!importFn) return
      return loadShikiLanguage(language || '', importFn)
    }, [codeHighlighter?.codeHighlighter, language]),
  )