node_compat vs nodejs_compat developer experience

[Cherry]

Node.js compatibility via the nodejs_compat flag is getting better and better with Workers, and the docs at https://developers.cloudflare.com/workers/runtime-apis/nodejs/ are great.

The compat is getting so good in fact that I'm starting to default to using Node.js implementations of things like Buffer and crypto because they're so much more ergonomic to work with.

Unfortunately today, there are still a lot of libraries in the node ecosystem that require('crypto'), or require('tls') and things which this compat flag doesn't solve, so the older node_compat wrangler.toml option often comes into play here. For some libraries, this works really well, such as pg. Here's an anecodtal experience I had recently trying to play with Hyperdrive:

So I have a project that uses Neon for a Postgres DB. My worker currently uses the Neon serverless stuff to connect and works well. I'm also using a bunch of node:buffer, node:crypto, etc. stuffs via nodejs_compat compat flag.

I'd love to use Hyperdrive, but it seems like using pg directly requires enabling the older node_compat, but in doing so, I can no longer use nodejs_compat and any node: stuff in my own code.

This is not an uncommon experience, and people get confused about node_compat vs nodejs_compat regularly.

What is the long-term strategy here? Drop node_compat and work with library authors to ensure things work with nodejs_compat? Offer some kind of interaction with workerd's nodeJsModule type, with polyfills? Find a way for them both to work together? I very much expect folks to start using more and more node.js APIs, especially for things like crypto where the node.js implementations are just so much more ergonomic than WebCrypto, but if in doing so they lock themselves out of using libraries like pg and others, that creates a pretty disjunct developer experience.

This is somewhat similar to cloudflare/workers-sdk#4050.

[irvinebroque]

Offer some kind of interaction with workerd's nodeJsModule type

Yes — this, with Wrangler doing the right set of things when the nodejs_compat compatibility flag is enabled, is the path forward. Working toward making progress here.

[ItsWendell]

I'm running into a very similar issue, my workers depend on async_hooks for logging / instrumentation. I was experimenting with getting a direct PostgreSQL connection to work, instead of using a 'proxy' like Hasura, but indeed pg needs node_compat while asyncvery_hooks need nodejs_compat, and the project won't build. Are here any work arounds for this?

Edit:
@Cherry if your current use-case is using pg, an alternative that I found https://github.com/porsager/postgres, seems to be working just fine in nodejs_compat mode!

[hansottowirtz]

We made this library to compile our workers with esbuild, which keeps compatible node modules (e.g. replaces const util = require('util') with import * as util from 'node:util'), and shims incompatible node modules (e.g. fs is replaced with an empty object). This enables us to use pg without problems.

In my opinion Wrangler should allow a custom esbuild config in the future.