Node compatibility

[ry]

We've resisted considering Node compatibility as Deno was under initial development so that those considerations didn't influence the design. But now Deno has a relatively established API and the URL module system is deeply baked in. It is unlikely, at this point, that considering compatibility with existing Node modules will compromise Deno's design goals.

It's unlikely we can make Node code work 100% of the time. But we can do a lot to make porting modules over easier. Let's discuss how make Deno work with existing Node code. Here are some ideas:

• One direction could be a transpilation program that takes a CommonJS node file, and translates it into ES modules.

• Node has core APIs like file system and networking which are different (sometimes subtly so) in Deno. For example, require('fs').readFileSync() in Node supports encoding to strings, Deno.readFileSync() does not. I suggest we implement a compatibility layer in deno_std/node to match the semantics. (There has been some effort to do this already in https://github.com/denolib/node, but progress is slow, I think it makes sense to do it in deno_std.)

• We should attempt a require() implementation. It should probably be in deno_std rather than deno itself. It might make sense to base this on top of dynamic import Support dynamic import #1789 (which will land very soon - see Dynamic import #2516).

Existing work

• https://github.com/denolib/node-to-deno
• https://github.com/denolib/node

[zekth]

I've started to work on https://github.com/denolib/node with @kevinkassimo but this feature was not having so much following untill now so not so much effort has been involved it. If you want to make it part of deno-std i can address the port of core APIs.

[kitsonk]

@ry my suggestion is to scope what the expectations around require() are. It easily becomes a complicated thing, because of the rather complex module resolution logic (as I know you are aware), but it is resolution logic that lots of Node.js code relies upon. There are also the decorations on require() which a decent amount of code uses to hack around with.

Another couple thoughts about compatibility is:

1. do you need to build it into Deno? Is there some preprocessing, potentially on the fly, that could be offloaded to an intelligent CDN? https://www.pika.dev/cdn takes ESM modules out of npm and bundles them for being loaded in the browser, resolving dependencies, etc. Something like that for Deno might be an easier path than building all the heavy lifting in the runtime engine.
2. Is there some way the TypeScript compiler do the heavy lifting. While it wouldn't re-write module specifiers, it can convert JavaScript CommonJS modules to ES Modules. If we implemented whatever module resolution logic we wanted to support in Rust land and used the compiler to simply rewrite the modules to ES, we wouldn't have to implement require. I just can't remember what happens when you have a require() not at the top level. 🤔

[fusionstrings]

This is great to have. I have been following JSPM since beginning of the project. In its latest iteration JSPM2 @guybedford has managed a great feat in this area and result is in terms of https://github.com/jspm/jspm-resolve. The overall scope of JSPM is big but a solution to module resolution is at the core.
I guess this is where deno might join hands with JSPM. A problem which has already been identified, researched and solved at great extend from all possible perspectives and keeping in mind limitations of the environment and infrastructure (network and web) by Guy. What’s left is to align it with this context.

I have (and I see several others) been using modules from https://jspm.io/ (which is same as using jspm resolver) in deno and it works out great.

If deno includes jspm resolver in std, it will be huge for deno and JSPM both because I assume both platforms want to align with web and tc39 standards as possible.

[guybedford]

Thanks @fusionstrings for pinging me on this issue. An https://deno.jspm.io service could certainly be an interesting prospect. Since jspm.io already does the heavy-lifting of converting CommonJS to ES modules conversion while supporting Node.js semantics and strict mode conversions, the only part of the problem left to solve to get this support would exactly be that compatibility wrapper for Node.js core libraries as is being worked on in https://github.com/denolib/node.

I think it would actually be quite easy to throw such a server together given enough coverage of the core APIs through the compatibility layer. I will continue to track the repo there, and happy to discuss further too.

[guybedford]

Also, if anyone is interested in working on this problem separately to jspm, the CJS -> ESM conversion layer that is used is open sourced here - https://github.com/jspm/babel-plugin-transform-cjs-dew.

[thisconnect]

One direction could be a transpilation program that takes a CommonJS node file, and translates it into ES modules.

library authors are already moving to shipping more and more native es modules

Node has core APIs like file system and networking which are different (sometimes subtly so) in Deno. For example, require('fs').readFileSync()

Node.js itself is moving to native esm i.e. import fs, { readFileSync } from 'fs';

https://nodejs.org/api/esm.html

The future will be without require nor any other custom non-standard

[szhu]

I'd love to see a npm library that implements the Deno api using the Node stdlib.

This library would:

• Allow people familiar with Node to start to enjoy some parts of Deno. This is good for people who can't immediately switch to Deno right away for some reason. This will also make it easier to convince users that Deno will be easy to work with if they do switch.

• Make it possible for simpler projects to support both Node and Deno without needing to transpile code. Also it makes the experience more pleasant -- if I need to write such code, I'd prefer writing Deno code and supporting Node as an afterthought using this library rather than writing Node code and supporting Deno as an afterthought using deno_std/node.

[kevinkassimo]

I'd love to see a npm library that implements the Deno api using the Node stdlib.

Here is one: https://github.com/keroxp/denode

[yorkie]

Proposals in this issue are to handle the "API" compatibility, but I don't think it makes too much sense on Node.js, because the API changes are always frequent and the compatible works are very huge and segmented.

How about patching to Node.js core to add native APIs to import/require Node.js packages(NPM partially) and run and evaluate functions, therefore any other runtime like deno could embed the Node.js inside, and use its ecosystem in native way, The advantage of this is that the workload is really small, but corresponding disadvantage is that we have to tolerate the existence of two runtimes.

This idea is inspired by CPython C-API, If you don’t think it matters, just ignore it :)

[hayd]

I'd love to see a npm library that implements the Deno api using the Node stdlib.

That's also what the npm package with the coveted name "deno" attempts to do...

[waldyrious]

the npm package with the coveted name "deno"

Links for reference: https://www.npmjs.com/package/deno, https://github.com/YounGoat/nodejs.deno.

[Soremwar]

@allnulled Jeez, calm down

What you even mean?

[allnulled]

@allnulled Jeez, calm down

What you even mean?

Sorry, I am sorry. I thought it was better to divide the code in a superlanguage to make more environments and languages compatible, instead of creating a new one.

But you are the geniuses, and in you we trust.

[Soremwar]

@allnulled Technically speaking, it's Node the one with the divisory problem

Deno supports JavaScript as in the browser, and emulates it's environment the best it can with a couple upgrades to enhance it's reach in to the BackEnd area

Node in the other hand uses CommonJS, a variation of JS that isn't compatible with anything else in the whole environment. Doesn't use modules, has a lacky asynchronous support, and changes the behavior of JavaScript to it's pleasure

In that context, it's pretty clear to me what the incompatible technology is

[allnulled]

@Soremwar Yes, you are right. Good luck with it. And sorry for taking any of your time, it was not my intention.

[brettz9]

Node does use ESM modules now (and CJS predated them): https://nodejs.org/api/esm.html .

I guess you mean a hacky async support, but if you mean Promises, those are increasingly supported, e.g., https://nodejs.org/api/fs.html#fs_fs_promises_api .

I'm not sure what you mean about changing the behavior of JavaScript.

Where it seems to me, in my limited experience, that Deno shines, is with its sandboxing of permissions (I'm personally interested in strict linting procedures to effectively bring something similar to Node, but that's quite a tall order).

While I hope the language can now be toned down, I do think there can be what seems like hubris that dismisses the HUGE amount of work already in play with Node (akin to humanity failing to appreciate what enormous benefits there would be with an official second language!), even while I understand an originally experimental project would want to reinvent some aspects which in retrospect may only make more headaches for compatibility. Thankfully, it seems there are some efforts in both directions to bring that compatibility back. Node itself in the above cited page refers to accommodating "Deno" as an export type: https://nodejs.org/api/esm.html#esm_conditional_exports . And others here have mentioned projects on the Deno end.

I think often experimentation tends to be followed by consolidation anyways. But I'd personally agree it would admittedly be nice if the great pioneers (and I don't mean that at all facetiously) would give more consideration to compatibility in future projects (and in this current one).

[allnulled]

I really wish you the best, I know you are putting great efforts, and it would be awesome to unify the JavaScript communities and environments in 1 language that can stand any environment. I appreciate your work, and I am sure Deno will be the new thing we all want for the JavaScript programming interface.

I am just sad that all the Node.js ecosystem gets wasted. But I give thanks it was done by the same creator of it, and not another.

Thank you guys, I will give a chance to Deno, despite how much used I got to Node.

[Soremwar]

@brettz9 I like your attitude 😄

Just like you said, they are there, they are just really hacky.

Ex:

JavaScript

await someAsyncFunction();

Node (written same as before, but Node converts it to this)

(async function(){
  await someAsyncFunction();
}());

I see this examples under the following light: The more JS changes and grows, the further Node will deviate from it. That's why I don't see as much of a benefit creating a compatibility layer with it.

@allnulled But let's be fair, Node brought us some really cool stuff. Many libraries written in JavaScript can be ported directly to Deno and that is really good. Node pushed forward a language that people thought was only for adding cool effects to a website to a whole new level. JS would probably be dead by now if it weren't for Node.

Just hope more people will make their code runtime agnostic and less compatibility-layer dependant.

[brettz9]

--harmony-top-level-await in Node 13.3+ avoids the need for the top-level async. It seems to me that Node if anything has been becoming ever more in harmony with JS (and polyglot support with browser APIs--e.g., adding support for the URL and URLSearchParams globals). However, I can understand that for someone starting off with Deno which had the advantage of hindsight and ability to make a clean break, that it may seem like some aspects may be outdated.

But yeah, hear-hear for more runtime agnosticism.

[allnulled]

@Soremwar I do not want to polute this Github issue talking about other things that are not "Node compatibility", but saying that "JS would probably be dead by now if it weren't for Node", that is crazy, the whole web is driven by JS, Node is just one of the ecosystems out of it that works, but we could be using Rhino otherwise, or others, NodeJS is great but JavaScript came before.

I worked on NaturalScript years ago, a language that aimed to synthetize JavaScript and Natural Language in a mix of both, and I realized that languages are more driven by companies and fame and economical interests, than by rationality and understandability.

It was a simple transpiler that could do perfectly all these things you said, a very simple PEGjs-based parser. You are still fighting for doing things in a low level, instead of working on better interfaces that are usable by the people. And I understand, programming languages have to take care of the low-level, but in that sense, we could be using ASM language and compilers for it, and done. But, in real world, we are sacrificing a lot of efforts in optimization or branding, just because some rich people are playing to "win the battle" using talent for their selfish purposes, that is what is happening in real world.

So, again, you are the geniuses, do what you think it is more convenient, I am out of the market even, because companies do not like me, or whatever, I have to still struggle, for some reason, to find a simple job (this is just to set it clear: I am not the genius here, for sure).

It is cool Deno, go ahead, it would be great that informaticians understand the importance of accessibility a little more, the importance of having 1 tool instead of thousands. But anyway, if you think it deserves to do it (or even if you just like to do it), why not, it seems cool to me too.

[JohnnyUrosevic]

https://www.pika.dev/cdn

Seems like all the tools are there to convert npm modules to deno automatically, I think it would be very nice to have some standard functionality in the deno ecosystem to do this conversion for us.

[eloff]

At this risk of adding noise to this thread, I want to mention I spent a day checking Deno out now that it's at 1.0, and I would have used it for a new project, except for the fact that it's incompatible with npm modules.

I did the investigation as to what dependencies I can convert with pika.dev, which I have to fork and modify, and which would be better to remove all-together. It was looking like a big task, but still worth it for me.

But my end users would need to do the same for their dependencies, and that's a show stopper. If it's not already obvious, npm compatibility will make a big difference to Deno adoption. I've subscribed to this issue and will reconsider moving to Deno when it's resolved. I will also help out with the effort as time and energy permit.

[balupton]

I'm doing an experiment for a few more days on making a project that will:

1. provide an executable that will make a typescript npm package compatible with deno

2. have a cdn proxy for direct access.

Will post back with my results at the end of may.

Intent is to automatically make all the existing bevry packages available for deno.

[hcldan]

@balupton what were your results?

[balupton]

@hcldan accomplished everything except for the cdn, which seems superfluous for now:

I've created make-deno-edition to make npm packages written in typescript compatible with deno - is working on badges - usage proof of this here https://repl.it/@balupton/badges-deno - has been used now to make 32 node packages compatible with deno - you can use project to automatically generate the readme instructions for the deno edition - and can use boundation to automatically scaffold your projects to automate the entire process - start-of-week is an example where different entries are used for node, deno, and web browsers

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[ealib]

Is Node compatibility layer supposed to include compatibility with Node native binary modules? #12424 #11152

[kitsonk]

Not at this stage.

[bartlomieju]

Basic Node compatibility is available starting with Deno 1.15: https://deno.com/blog/v1.15#improving-node-compatibility

Work is tracked in #12295 and #12577