ESM support

[masonicboom]

Here's a proof-of-concept of an ES Modules build.

I haven't tested if this achieves any sort of file-size reduction through tree-shaking. I'm just doing this to facilitate use in the Deno runtime (https://deno.com).

To test, with Deno installed:

1. Checkout this code
2. Copy emscripten.sh from ESM build libsodium#1382 to libsodium/dist-build/emscripten.sh
3. make esm
4. deno
5. > import sodium from "./dist/modules/libsodium-esm-wrappers.js"
6. > await sodium.ready
7. > sodium.crypto_secretbox_keygen()

Any feedback on the approach so far? If not, I'll just plan to add Make targets for the sumo builds and get the automated tests running against this (I think there are some—haven't looked deeply).

[jedisct1]

So, the intent would be to have a JS-only build, that would be slow, but would hopefully be sometimes smaller (after tree shaking) that the webassembly module. Did I get that right?

I'm not too familiar with this, but my quick experiments were not very successful, and tree shaking didn't do much with emscripten-generated code. Hopefully you'll be more successful!

Bun/node/deno/etc. support WebAssembly, so a WebAssembly-only build that only contains the required functions (using e.g. https://www.npmjs.com/package/@webassemblyjs/dce) would be the best way to reduce the file size. But does that really matter for server runtimes?

[jedisct1]

With your example (and after having compiled libsodium with -Oz and without WebAssembly):

$ bun build --target bun index.ts --minify --outfile minified.js

  minified.js  546.43 KB

Unfortunately no significant size reduction.

[jedisct1]

With libsodium.esm.js containing only WebAssembly code:

$ bun build --target bun index.ts  --minify --outfile minified.js

  minified.js  317.35 KB

No size reduction, but that's expected.

[jedisct1]

When targeting the browser, this is worse:

WebAssembly build:

$ bun build --target browser index.ts  --minify --outfile minified.js

  minified.js  1016.09 KB

index.ts is just:

import sodium from "libsodium-esm-wrappers"
await sodium.ready

[masonicboom]

I may have confused things by mentioning file size. Some people at #263 seem to be hoping ESM support will reduce file size.

For the moment, I just needed ESM support so I could properly import libsodium.js in Deno. There's a libsodium.js wrapper for Deno at https://github.com/denosaurs/sodium, but when I used that in my tests, it caused stack overflow exceptions, I believe because Deno only likes ES Modules (https://docs.deno.com/runtime/manual/node/migrate#module-imports-and-exports).

So, this first pass was just to get the library built as an ES Module. However, I think I do see how to get some tree-shaking benefit. The wrapper file is dynamically assigning its wrapper functions to the exports object, which I suspect blocks the minifier from trying to strip any unused code. I'll try directly exporting those functions and see what happens.

[masonicboom]

In the latest code, I'm directly exporting the wrapper functions by prefixing each function declaration so it becomes export function. Unfortunately I couldn't do the same for the constants, because they can't be set until after initialization. That means the module signature looks slightly different than before. Here's the test file I'm using (index.ts):

import consts, * as sodium from "./dist/modules/libsodium-esm-wrappers"
await sodium.ready
console.log(consts.crypto_hash_BYTES)
console.log(sodium.crypto_secretbox_keygen())

To get bun build --target browser working I had to do -s ENVIRONMENT=web in emscripten.sh. The full line to build libsodium/libsodium-js/lib/libsodium.esm.js now looks like this:

emccLibsodium "${PREFIX}/lib/libsodium.esm.js" -O3 -s WASM=1 -s ENVIRONMENT=web -s EXPORT_ES6=1 -s EVAL_CTORS=1 -s INITIAL_MEMORY=${WASM_INITIAL_MEMORY}

Here's what I get with the browser build.

> bun build --target browser index.ts  --minify --outfile minified.js

  minified.js  925.38 KB

Targeting bun:

> bun build --target bun index.ts  --minify --outfile minified.js

  minified.js  232.87 KB

[masonicboom]

The file size savings is probably from the bundler eliminating unused wrapper functions. Like you mentioned, really minimizing the size would require eliminating unused functions from the WASM output itself. The ultimate form of that would be something like compiling a separate WASM asset for each symbol, then individually exposing them. That would entail making everything async, which would also allow for loading their WASM as binary rather than embedding it as base64 in JS code.

Clearly that would be very invasive. I won't do any of that here.

But is this current change to expose the lib as an ES Module something you'd like to include in this project, @jedisct1? (Either with or without the latest couple commits directly exporting the wrapper funcs.) I'm happy to do some more tidying up if so.

[masonicboom]

I realized that I was already doing a top-level await, so there was no reason not to just do the module initialization immediately after that, rather than waiting for the consumer to await the ready promise. That's what the latest commit does. This makes it possible to immediately export the constants, so the test index.ts now looks like this:

import * as sodium from "./dist/modules/libsodium-esm-wrappers.js"
console.log(sodium.crypto_hash_BYTES)
console.log(sodium.crypto_secretbox_keygen())

TIL this use case of WASM initialization is one of the motivations for top-level await (https://github.com/tc39/proposal-top-level-await/blob/HEAD/README.md#webassembly-modules).

[cooper667]

Just tried this, nice work. ESM and the top level await are both useful for us