RuntimeError: memory access out of bounds

[sandren]

Validations

• Follow our Code of Conduct
• Read the Contributing Guide.
• Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.

Describe the bug

We're trying to migrate the Waku website to Shiki (v1.0.0-beta.0) and encountering the same RuntimeError: memory access out of bounds issue that others have reported. It's affecting both local builds and Vercel deployments.

RuntimeError: memory access out of bounds
at wasm://wasm/001c7aca:wasm-function[198]:0x24ff1
at OnigScanner._findNextMatchSync (file:///vercel/path0/packages/website/dist/entries.js:39458:32)
at OnigScanner.findNextMatchSync (file:///vercel/path0/packages/website/dist/entries.js:39450:17)
at CompiledRule.findNextMatchSync (file:///vercel/path0/packages/website/dist/entries.js:34607:33)
at matchRule (file:///vercel/path0/packages/website/dist/entries.js:35304:25)
at matchRuleOrInjections (file:///vercel/path0/packages/website/dist/entries.js:35282:23)
at scanNext (file:///vercel/path0/packages/website/dist/entries.js:35159:15)
at _tokenizeString (file:///vercel/path0/packages/website/dist/entries.js:35155:5)
at Grammar._tokenize (file:///vercel/path0/packages/website/dist/entries.js:35643:15)
at Grammar.tokenizeLine2 (file:///vercel/path0/packages/website/dist/entries.js:35609:20)

Reproduction

dai-shi/waku#437

Contributes

• I am willing to submit a PR to fix this issue
• I am willing to submit a PR with failing tests

[fuma-nama]

From your code, I found that you are calling getHighlighter in every render, which is not the best practice as you should put it in the global context instead.

[antfu]

https://github.com/dai-shi/waku/pull/437/files#r1469621783

[joshwcomeau]

I was having this issue too, trying to use this with Next.js in a Server Component. @souporserious shared his solution from MDXTS, and it seems to work great:

import React from 'react';
import { getHighlighter } from 'shiki';

let highlighter: Awaited<ReturnType<typeof getHighlighter>> | null =
  null;

const getCachedHighlighter = React.cache(async () => {
  if (highlighter === null) {
    highlighter = await getHighlighter({
      // options here
    });
  }

  return highlighter;
});

// Call `await getCachedHighlighter()` whenever you need it in your component(s).

[SandroMaglione]

I am having the same issue, but in my case I am using rehype, so the cached highlighter solution probably cannot work.

Someone has an idea on a possible fix?

[souporserious]

Cached highlighter should still work with rehype, assuming you are in control of the instantiation. There's an example here that does this.

[SandroMaglione]

@souporserious the problem is that RehypeShikiOptions does not provide an option to add a custom highlighter:

rehypePlugins: [
  [
    rehypeShiki,
    // 👇 No highlighter
    { theme: "one-dark-pro" } satisfies RehypeShikiOptions,
  ],
]

Do you know if this can be solved?

[fuma-nama]

The official rehype plugin should automatically cache the shiki instance, it's more likely you are processing a large amount of files that your machine can't handle, or a wrong usage (like re-constructing the MDX processor in every file).

You can open another issue and provide sufficient information for debugging.

[joshwcomeau]

Just a quick followup: With the solution I shared above, I still occasionally get the out-of-memory issue. I think it's because I have two separate highlighters (a "light" one that runs in-browser and only has 1 theme and a couple of languages, and a "full" one with many different languages).

It's fine in my case since I'm running Shiki at compile-time, so if 1 in 500 builds fail, that's not a big deal. But if I was doing this with an on-demand SSR strategy, it might give me pause.

It’s likely user error on my part, but wanted to mention this to be fully transparent about how well the solution I shared works.