How do we handle dynamic specifiers (without dynamic import)?

[targos]

ESM cannot load a module "synchronously" with a dynamic specifier. Until we have top-level await, I think we need a supported way to do it. Two use cases:

let env;
if (process.env.NODE_ENV === 'production') {
  env = require('./prod.js');
} else {
  env = require('./dev.js');
}

let canvas, hasCanvas;
try {
  canvas = require('canvas');
  hasCanvas = true;
} catch (e) {
  hasCanvas = false;
}

I can think of three solutions:

1. Explain in the documentation that one has to write the dynamic code in a commonJS module and import it.
2. Add import.meta.require
3. Add a builtin module that provides the require function (npm proposal)

[devsnek]

i would be up for unscoped require from the module module or import.meta.require, but definitely not scoped require from an imported module

[jkrems]

I'm -1 on the builtin module. This seems to go directly against the spirit of TC39 (as I understand it). import.meta is the official way to get module-scoped values and import.meta.require could be one of them.

[MylesBorins]

I'm a huge +1 on import.meta.require

[mcollina]

+1 to import.meta.require.

[devsnek]

I can open a pr for that in a few hours unless someone else wants to take it

[giltayar]

-1 on import.meta.require for this use case (if we decide that this is the way to go for CJS interoperability, then that's OK, but IMHO it's not OK for this use case).

My reasoning is this: ESM is async, and this is not going to change, AFAICT. Developers working with ESM will have to live with that asynchronicity through various means. Having a way to workaround that asynchronicity will mean that when the module they import will move from CJS to ESM, they will have to rewrite the code to deal with that.

Yes, because top-level await is not yet a thing in JS, they have to wrap their dynamic import in an async IIFE, but that's the way it is currently with ESM. Once developers move to ESM land, they need to start thinking ESM-think and not import their CJS-think into it.

So, yes, -1 on adding import.meta.require for this use case. We should not add complexity to an already complex subject (just because we can).

[MylesBorins]

@giltayar there are all sorts of sync apis that exist in node. To me the "async" nature of ESM has to do with the initial module graph, there is 0 guarantee that the execution with make it through the entire graph without hitting a sync api. That being said I recognize that I am conflating "dynamic imports" + "cjs interop"... although it is hard to separate the two when cjs can be used for dynamic moduless.

edit: keep in mind that require does not work for esm

[giltayar]

@MylesBorins - I am not against import.meta.require as a solution for CJS interop, if that is what is decided! I am against getting this feature entering NodeJS through a backdoor as a solution to a use case that for me is not that important.

As for async/sync - it's not the sync-ness of the api that bothers me, but rather it's the fact that we're importing a module in a sync way in an ESM module that should allow only async importing, and which when that imported CJS module will be migrated to ESM will anyway need to be rewritten as an async import (because, as you said, require does not work for ESM).

[MylesBorins]

@giltayar I do think it is worth thinking about the initial graph and subsequent graphs as different beasts. While they may share a cache, when you dynamically import a module you are essentially creating a new graph.

[giltayar]

@MylesBorins - interesting point of view (initial and subsequent graphs)!.

To continue that way of thinking, and if I understand where you're coming from, then from the ESM point of view you are creating a new graph even if you are synchronously importing a CJS module. Synchronicity does not help you with "extending" the existing graph, because, by decree, ESM's initial graph must be statically analyzable. And dynamic import, whether sync or async, is not.

So we're not gaining anything with synchronous import.meta.require, except for a comfortable and known way of importing modules. At least for this use case!

[weswigham]

If I, as a user, can write a global require function implementation in commonjs that is in the global scope (and I'm fairly certain I could, by inspecting the stack of an error I make to determine the dirname and filename of the calling file) that is then usable in a module, I don't see why node shouldn't offer that functionality in it's core, the "spirit of the tc39" not withstanding. I dislike both import.meta and a module that brings in require because both require varying degrees of refactoring to migrate to modules... It's very much not meeting users where they are (that is to say, using a seemingly global synchronous require for most imports).

[TimothyGu]

@weswigham A global require approach has far reaching drawbacks. What should it do, e.g., when a vm.Script created without a file name invokes it? eval? Would that break CJS expectations that require() be always per-module? We have not put that option into consideration, for some very good reasons.

[weswigham]

@TimothyGu I'm not sure that the technical considerations for the vm module should override the ergonomic considerations of migration.

[weswigham]

Especially, since, as I said, I could write a package that does this today, and offer a better experience with an import "shim-global-require"; or something. I don't think allowing a shim package to offer a better migration experience than node proper is correct. And by owning it in core, you can make sure it behaves correctly with respect to loaders and require hooks.

[MylesBorins]

I've kicked off #10 to discuss interop... this issue should likely stick to the initial thread of dynamic imports, although I recognize they are very very intertwined

[TimothyGu]

I'm not sure that the technical considerations for the vm module should override the ergonomic considerations of migration.

vm module was just an example. And I would argue for the opposite.

I don't think allowing a shim package to offer a better migration experience than node proper is correct.

Something worth considering is that if we were to add such a feature to core, it will probably never be removed because of userland dependency on it. For examples of such, see util.is* methods which are deprecated since forever but still there for compatibility.

On the other hand, a userland module would have this flexibility. And we (Node.js core) wouldn’t have to deal with the issue of tricky edge cases with eval or new Function that may result from doing that.

[TimothyGu]

@MylesBorins Oops, just saw your comment. Will move this over in my next comment, if any.

[bmeck]

I'm a big fan of punting this if top level await is currently being specified, adding import.meta.require is a topic that carries more weight than just this issue and would mean we have to keep it around forever. import.meta.require is also a poison that means that modules immediately are opted out of browser compatibility if they use it. We should avoid encouraging that if it is a means of gaining tech debt before it even ships.

[jkrems]

import.meta.require is also a poison that means that modules immediately are opted out of browser compatibility if they use it.

I would say that any module loading a CommonJS script is opting out of browser compatibility. import.meta.require just makes it immediately visible, including to static analysis (for the most part). For me being able to easily track modules that aren't (yet) compatible is one of the upsides of import.meta.require.

[bmeck]

@jkrems even with being able to tell that for a single type of import you won't be able to tell that for other types that might not work cross env, like HTML modules. CJS should not be treated as unique when we have other module types in the same category of removing cross env compatibility.

[WebReflection]

what @jkrems says + import.meta has been left extensible not by accident in the specifications so that browsers might opt in through 3rd parts polyfill (.e.g a loader that sets import.meta.require as getter and assign the current file/folder when invoked once the module is injected)

We should avoid encouraging that if it is a means of gaining tech debt before it even ships.

I also agree if that's the case.

[benjamingr]

I'm not convinced we should solve this use case. I'm not sure why "without dynamic import" is a requirement here.

We should just tell people they can do:

(async () => {
  if (process.env.NODE_ENV === 'production') {
    env = await import('./prod.js');
  } else {
    env = await import('./dev.js');
  }
})();

And then drop the iife once top level await lands. In any case, I don't think this should block shipping modules.

[targos]

IIFE only works well for the main module. I don't think this should block anything but I wanted to know what other people thought about this. I'm not going to push hard on this because there are "solutions" (IIFE and point 1 in the OP), but to me they are more workarounds than first class features. I guess it's fine if we are sure that top-level await eventually makes it to the spec.

[WebReflection]

IIFE only works well for the main module.

Isn't the main module where 99% of the time you decide dev VS prod dependencies, though?

In any case even this syntax doesn't look so bad to me:

import(`./${process.env.NODE_ENV || 'dev'}.js`).then(module => {
  const stuff = await module.default || module;
  stuff.toDo();
});

But the main issue here is that whatever module you used to load dynamically via require is a CJS module already. Maybe a transpiled one, but it is inevitably CJS (because it's targeting a CJS require) so import.meta.require(...) there would solve the issue synchronously, if really needed.

[benjamingr]

IIFE only works well for the main module.

That's true (and unfortunate) - like all async code it's contagious.

but to me they are more workarounds than first class features

I agree, it's not that optimal (or great) but in all honesty I'd expect things like conditional requires to simply stop existing when people start using ESM - since the pattern won't be available to them people will stop using it - and I'm fine with sacrificing a tiny bit of expressiveness for more static module code.

I guess it's fine if we are sure that top-level await eventually makes it to the spec.

No one can promise that - but I think that we should at least try

[giltayar]

@WebReflection - yes, import.meta.require would solve the problem of async, but as I said above, it would just delay the problem till the module is converted to ESM (which we MUST assume is the end goal of all modules, at one time or another).

So, yes, it would alleviate the pain, but only temporarily. And that's, I believe, the feeling here—import.meta.require is a nice to have for this scenario, but it's not a must have.

(and as I will stress again, I am not against import.meta.require. I am only against adding it just for this reason)

[benjamingr]

Just to make sure I'm on the same page since I haven't been following discussion nearly as long as some of you - import.meta.require as in allowing require in ESM through import.meta right @giltayar ?

[MylesBorins]

Import.meta.require would only support loading cjs

…

On Wed, Feb 7, 2018, 9:57 AM Benjamin Gruenbaum ***@***.***> wrote: Just to make sure I'm on the same page since I haven't been following discussion nearly as long as some of you - import.meta.require as in allowing require in ESM through import.meta right @giltayar <https://github.com/giltayar> ? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#7 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAecV5zuMWju5ED4JKbC8KLHHI-78SVEks5tSbm7gaJpZM4R5xqU> .

[benjamingr]

Yeah, in that case I'm pretty sure I think we should give top-level await a chance and solve this later.

To clarify - I do think there should definitely be a way of loading cjs from mjs. I'm specifically referring to dynamic config.

[ljharb]

There is no facility in ESM to do conditional static imports; while I'd love to see that added to the language, I don't think it's node's place to address that - and thus I don't think it should block shipping modules.

[benjamingr]

@targos this has been an interesting discussion, given the consensus so far - is there anything further to discuss here in your opinion?

[targos]

@benjamingr I don't think so. Thank you all for your input.

[x5engine]

+1 to import.meta.require.