module: add support for node:‑prefixed require(…) calls

[ExE-Boss]

Fixes: #36098 (applies only to commit 1d8c8b7)
Refs: #37178 (applies only to commit b0fbc91)

[aduh95]

You read my mind, I was precisely working on this. I noticed it produces an arguably better error when trying to load internal modules:

$ node -p "require('internal/test/binding')"
Error: Cannot find module 'internal/test/binding'
$ node -p "require('node:internal/test/binding')"
Error: Should not compile internal/test/binding for public use

Can you add tests in test/parallel/test-require-node-prefix.js to make sure requiring internal modules still throws please?

[benjamingr]

cc @nodejs/modules

[bmeck]

this differs from other builtins, I love it, but want to be sure it is intentional.

[ljharb]

this actually seems like an issue - it means someone who replaces a builtin intentionally can’t replace one of these (altho presumably mutating a builtin would be visible in both). That could cause an issue where someone wants to use a package to instrument or lock down fs (and thus, policies are not an ergonomic option, unless I’m misunderstanding how policies work), but accidentally leaves a gaping security hole when an attacker requires node:fs.

[bmeck]

policies should intercept before any of the resolution helpers even get called

node/lib/internal/modules/cjs/helpers.js

Line 52 in 36cc0ee

if (redirects) {

, if any module redirection is in place (including complete shutdown) then it would never call out to module.require and get here without some kind of opt-in to the behavior (like dependencies:true).

[benjamingr]

At the very least this needs documentation and probably a strong indication people using policies + instrumenting fs now need to update the policy.

[ljharb]

My point was that policies aren’t currently required to ensure this for CJS - a package (not the app itself) can do it. Forcing policies to be the mechanism means packages are no longer capable of abstracting this for users.

[ljharb]

Totally understood. Given that policies are the superior mechanism for app developers to lock things down, what's the benefit of adding special, inconsistent, cache-bypassing behavior for require with a node: prefix?

[bmeck]

@ljharb i have no strong opinion on if we should diverge, but cache first behavior of CJS isn't shared by ESM and isn't robust against things like core destructuring the original impls. My like is just that it isn't a confusing situation like with fs/.

[ExE-Boss]

but cache first behavior of CJS isn't shared by ESM and isn't robust against things like core destructuring the original impls.

This is demonstrated by:

const fs = require('fs');

const fakeModule = {};
require.cache.fs = { exports: fakeModule };

assert.strictEqual((await import('fs')).default, fs);

[ljharb]

indeed, but that doesn’t mean it’s a good idea to make CJS inconsistent with itself.

[bmeck]

I don't think CJS is able to be claimed as consistent with itself since in general the natives don't normally populate the cache. I think it was just ad-hoc written together and the cache behavior is evolutionary not intentional or well understood (even by me).

[guybedford]

This bifurcation will have implications for mocking, but I can appreciate the benefit too.

I'd still prefer a more convergent path here eg, to return node:fs for both node:fs and fs inputs, and then to still populate a cache['fs'] entry in the node:fs case as the same object for backwards compatibility. Such a change would hopefully be mostly compatible but would probably need to be a separate major nontheless. Worth thinking about at least.

[ExE-Boss]

Existing mocking tools (such as Jest) provide their own require(…) implementation, since they need to be able to bypass the mocked module using jest.requireActual(…).

[guybedford]

What is the motivation for short-circuiting the cache here, something that has never been done previously for native modules?

I don't see why this scheme should be any different to any other in this regard.

[bmeck]

On this topic, should we throw or defer to cache on unknown builtins?

[aduh95]

LGTM

[benjamingr]

Just making sure #37246 (comment) is addressed before landing :) (namely the caveat) - I'm appreciative of the work put in, thanks!

Also would like more @nodejs/modules and @nodejs/tsc eyes on this.

(The actual changes LGTM)