CDN/Assets-free ESM Approach of loading wasm and languages

[antfu]

Shiki's assets fetching part is always a bit tricky to set up, we introduced setCDN, environment detection, trying to make it work in various environments. At the same time, users who use bundlers might still need to manually handle the asset distribution and the potential misalignment in dev/build environment.

Those are tricky but still solvable if you take some time. However, we hit the wall when trying to deploy shiki to Cloudflare Workers. In CF workers, or other worker runtimes, you usually don't have access to Node.js's API, meaning no file system, etc. And loading WASM can also be a bit challenging.

This is mainly due to the fact that we are resolving the assets at runtime, meaning the bundlers won't know the relationship between your code and assets, making the connection easy to break when moving things around.

Thus, I made an explorational rewrite as https://github.com/antfu/shikiji, trying to improve the situation.

Basically, what it does is to convert the JSON languages and themes into ESM module .mjs and then reference them using dynamic import() (code here). And inline the WASM into Base64 so it can be an ESM module as well. With these, we are building the relationship in pure ESM without any assets and do not require any Node API or network requests. Not only would this make the codebase simpler and the data loading more efficient, but it also would allow bundlers to generate corresponding chunks and ship the data automatically.

The result turns working very well for us, and we are able to run shikiji on Cloudflare worker as an on-demand syntax highlighter running on edge.

Since we are in the process of preparing 1.0, I wonder if you think it would make sense to have this approach as a part of shiki core? /cc @octref @orta

[antfu]

Also experimented with an interesting approach to Dark/Light modes #33 (comment) on antfu/shikiji#5.

Happy to port it back as well.

[orta]

I think this is a good call 👍🏻

[TorstenDittmann]

I am in full support of this 👍🏻

Its kinda blocking me of using shikijs right now, regarding the "patches" I need to implement to be able to use it with Svelte Kit doing SSG.

[gdomingu]

I would like this as well. Shikiji's approach makes it much easier to use in a VS Code extension webview.