Shikiji breaks custom language syntax highlighting

[misterorion]

Astro Info

Astro                    v3.4.0
Node                     v18.18.2
System                   Linux (x64)
Package Manager          npm
Output                   static
Adapter                  none
Integrations             @astrojs/mdx
                         @astrojs/react
                         @astrojs/sitemap

If this issue only occurs in one browser, which browser is a problem?

No response

Describe the Bug

Astro 3.3.0 introduced a change from the internal shiki syntax highlighter to shikiji. After this change, providing custom language grammars in astro.config.mjs via path to a tmLanguage.json file no longer works.

After upgrading Astro to version 3.3.0 and making no other changes to my code, syntax highlighting for the custom language is not applied and no warnings are thrown.

For example, prior to 3.3.0 this would work:

import { defineConfig } from 'astro/config';
import mdx from '@astrojs/mdx';

import sitemap from '@astrojs/sitemap';

export default defineConfig({
  site: 'https://example.com',
  integrations: [mdx(), sitemap()],
  markdown: {
    shikiConfig: {
      theme: 'dracula',
      langs: [
        'go',
        'yaml',
        {
          id: 'caddy',
          scopeName: 'source.Caddyfile',
          path: '../../src/caddyfile.tmLanguage.json', // works with shiki, not with shikiji
        },
      ],
    },
  },
});

In my minimal example (astro 3.4.0, using shikiji), you can see that the syntax highlighting for the custom language in src/pages/index.md does not work in the preview when running npm run dev. Highlighting for the built-in languages does work.

Changing the astro version to an earlier version (e.g. 3.1.0, using shiki) in package.json causes the syntax highlighting to work for both the custom language and built-in languages.

Is it a correct statement to say: with shikiji passing a custom tmLanguage.json file is not possible since shikiji grammars are all pure ESM?

Is there a workaround or way to fix this other than downgrading, perhaps by forcing the internal highlighter to use shiki instead of shikiji?

What's the expected result?

Using custom grammars with the latest Astro version should work with minimal or no code changes.

Link to Minimal Reproducible Example

https://stackblitz.com/edit/withastro-astro-j1xdb7?file=astro.config.mjs

Participation

• I am willing to submit a pull request for this issue.

[bluwy]

I forgot to mention the workaround, but you can manually read the JSON file, parse it, and pass it to shikiConfig.langs instead. I've added the compat code linked above, but it will be removed in Astro's next major.

The solution would be the workaround as it makes the JSON more predictable. If you check the path: '../../src/caddyfile.tmLanguage.json', you'll see that the ../.. part maps to node_modules/shiki which is internal.

PS: I have not tried but import caddyLang from './src/caddyfile.tmLanguage.json' assert 'json' in astro.config.js might also work.