Add ncrypto dependency, implement key handling operations

[jasnell]

Still very much a work in progress.

This is gonna be a big one.

[github-actions]

The generated output of @cloudflare/workers-types matches the snapshot in types/generated-snapshot 🎉

[anonrig]

Amazing work as always @jasnell. There are some TS related comments that I'm not happy about, but we can fix it afterwards. No need to block progress!

[jasnell]

While this is generally ready to go for workerd, the production project uses a different version of boringssl that will require some additional tweaks in ncrypto to get working. This PR will remain in draft until those are done.

[vicb]

Does this unblock some of

workerd/src/node/crypto.ts

Line 250 in 186fa11

// Classes

or is this PR only adding thing that will be used later to implement those?

[jasnell]

@vicb ... it adds implementations for the following:

• createPrivateKey()
• createPublicKey()
• generateKeyPair()
• generateKeyPairSync()
• PublicKeyObject
• PrivateKeyObject

With caveats, of course. Key generation of DSA and DH key pairs do not work, for instance, due to current limitations in boringssl. There will be several followup PRs soon with more implementation detail.

[jasnell]

@panva ... fyi ... just a heads up that this is landing.

[panva]

@jasnell thanks for the heads up. I think it's a move in the right direction for compatibility.

That being said I'm switching jose fully to WebCryptoAPI with a new major and ESM-only as soon as 20.19.0 releases with unflagged require(esm). It will still support KeyObject as inputs (and will use KeyObject.prototype.toCryptoKey to crossover KeyObject to CryptoKey efficiently) and will also use globalThis?.process?.getBuiltinModule?.(id) for small optimizations when available here and there, e.g. new X509Certificate(pem).publicKey).

If you don't support KeyObject.prototype.toCryptoKey yet I'd suggest getting that in.

[jasnell]

Yep, KeyObject.prototype.toCryptoKey(...) will be coming soon (hopefully in the next couple of weeks).

[vicb]

use globalThis?.process?.getBuiltinModule?.(id)

@panva could you expand how you will use that? FYI when using node compatibility with workerd, the return value could be different from an import.

[panva]

@vicb here are all three instances in the v6.x-wip state at the moment

https://github.com/panva/jose/blob/151000e84b94551e7408f988e3d52f8ccf530a64/src/lib/asn1.ts#L253-L266
https://github.com/panva/jose/blob/151000e84b94551e7408f988e3d52f8ccf530a64/src/lib/buffer_utils.ts#L6-L21 (used in other files for base64url handling, this will be replaced with https://github.com/tc39/proposal-arraybuffer-base64 when available)
https://github.com/panva/jose/blob/151000e84b94551e7408f988e3d52f8ccf530a64/src/lib/decrypt.ts#L11-L33

I am specifically not interested in using an import() here because of how bundlers tend to either error out or insert shitty polyfills when they detect node modules in non-node runtime targets.

[vicb]

@vicb here are all three instances in the v6.x-wip state at the moment

https://github.com/panva/jose/blob/151000e84b94551e7408f988e3d52f8ccf530a64/src/lib/asn1.ts#L253-L266 https://github.com/panva/jose/blob/151000e84b94551e7408f988e3d52f8ccf530a64/src/lib/buffer_utils.ts#L6-L21 (used in other files for base64url handling, this will be replaced with https://github.com/tc39/proposal-arraybuffer-base64 when available) https://github.com/panva/jose/blob/151000e84b94551e7408f988e3d52f8ccf530a64/src/lib/decrypt.ts#L11-L33

I am specifically not interested in using an import() here because of how bundlers tend to either error out or insert shitty polyfills when they detect node modules in non-node runtime targets.

Thanks - I'll look more deeply next week.

One thing that could be better for workers is that you have an adapter that could be implemented for different platforms.

i.e.

import type { X509Certificate } from "node:crypto";

interface CryptoAdapter {
  X509Certificate: X509Certificate;
  // ...
}

You can have a default impl based on get getBuiltinModule but it leaves an option for platforms not (fully) supporting it.

For workers getBuiltinModule will return the workerd module which might not implement all the methods. However when you import crypto from 'node:crypto'; we return an hybrid implementation (some APIs are implemented in polyfill).

(Note that we do not polyfill getBuiltinModule to return the polyfills because it would mean that we can not detect the unused polyfills at build time and tree shake them).

[panva]

@vicb how can i enable node-compat in this capnp definition here, that way I could easily test the impact of node compat on jose (which i've seen people struggle with when node compat is enabled for them in wrangler)

I ask because when I add compatibilityFlags = ["nodejs_compat_v2"], then my test script starts failing with

workerd/server/server.c++:4470: debug: [ TEST ] main
A hanging Promise was canceled. This happens when the worker runtime is waiting for a Promise from JavaScript to resolve, but has detected that the Promise cannot possibly ever resolve because all code and events related to the Promise's I/O context have already finished.
workerd/server/server.c++:4478: debug: [ FAIL ] main
Tests failed!

That's certainly not expected just because a compat flag was added so I'm assuming I'm doing something wrong.

Repro with

gh repo clone panva/jose -- --branch v6.x-wip
cd jose
npm i workerd

// either add compat flag in tap/.workerd.sh or leave it be

npm run tap:workerd

[vicb]

compatibilityFlags = ["nodejs_compat_v2"]

That should do.

However the recommended way it to use compatibilityFlags = ["nodejs_compat"] as long as your compatibility date is >= 2024-09-23 (because nodejs_compat will also activate v2 after this date).

Also note that when using wrangler, we add polyfills when the node compatibility is activated (v2 or nodejs_compat with a date >= 2024-09-23).

If you want to test with the polyfill, you would have to use wrangler.

For example, instead of invoking esbuild in your script, you can invoke wrangler deploy --dry-run -outdir=/path/to/dist (You can also pass it the entry point, the compat date, the the compat flags - because the entry point and the flags aren't dynamic, they could also be in your wrangler.{toml,json}). That's similar to this test in the workers-sdk.

[panva]

@vicb i get the same failure regardless of the compat flag version. the test works as expected without a node compact flag

[vicb]

Yep, that's expected because they should be the same :)

To debug that, it's probably easier to create a wrangler.json in this directory, point to a worker.ts that should look something like:

// worker.ts
export default {
  async fetch(request, env, ctx) {
    // call your test from here (i.e. run.ts)
    await test();
    return new Response('Hello World!');
  },
};

Then you can wrangler dev, open the dev tools, add breakpoints or at least break on exception and then open the worker URL in a browser / curl it.

Let me know if you need more info (feel free to email me at the address on my GH profile).