Devise a migration plan for native addons

[mmarchini]

Native addons will be eventually affected by this change, therefore we need to devise a migration plan, including expected timelines and preferably migration tooling.

[mmarchini]

I'll throw a wild idea here, but could we move to something like https://github.com/cmake-js/cmake-js for addons before migrating Node.js to the next thing? Having the native addons build toolchain decoupled from core has the advantage that we can replace things in core if necessary without breaking the ecosystem.

[richardlau]

Sure, at least we could recommend so -- the challenge will be getting buy in from the ecosystem.

Also if someone has time to work on nodejs/node-gyp#1118 we can break node-gyp's dependence on core and allow migrating core and addons to occur on independent timeframes.

[richardlau]

There's absolutely nothing stopping module authors today from using something other than node-gyp (e.g. https://github.com/cmake-js/cmake-js) to build their addons -- the reason to pick node-gyp is that's what (almost) every other addon is using and npm (and yarn?) bundle node-gyp and autotmatically run it.

Maybe a plan for addons could be:

1. evaluate https://github.com/cmake-js/cmake-js (and alternatives?)
2. update the addons docs to use the alternative(/alternatives if more than one) for new modules
3. engage with package managers to streamline the user experience with the alternatives

We'd still need some sort of strategy for the vast number of existing modules out there that use node-gyp, some of which are no longer maintained. Unfortunately not being maintained doesn't mean they're not being used.

[mmarchini]

the challenge will be getting buy in from the ecosystem

Right, we would need to deprecate gyp and provide tooling for folks to migrate to new shiny thing. It won't be painless (no migration will), but it's a start.

and npm (and yarn?) bundle node-gyp and autotmatically run it

I thought Node.js bundled it and npm/yarn would just use the Node.js gyp package. I defenitely will need to learn more on how this happens.

We'd still need some sort of strategy for the vast number of existing modules out there that use node-gyp, some of which are no longer maintained. Unfortunately not being maintained doesn't mean they're not being used.

Right, having gyp out of tree might help with that as it'll be possible for consumers to install it via npm.

[richardlau]

I thought Node.js bundled it and npm/yarn would just use the Node.js gyp package. I defenitely will need to learn more on how this happens.

No, for addons npm bundles node-gyp (which bundles its own copy of gyp) and we vendor-in npm (I don't know what yarn currently does but at one point it was attempting to locate the node-gyp that was bundled with npm). Addons don't see the copy of gyp we vendor into tools to build core. The whole gyp-next (i.e. the new upstream for gyp) came about because the versions of gyp in core and in node-gyp got out of step and we found it difficult syncing changes for things like the Python 3 enablement and fixes for Catalina.

I'll reiterate for anyone not familiar that we have node-gyp, which is a JavaScript/Node.js wrapper around gyp/gyp-next (which it bundles) and gyp-next, which is the fork of (the unmaintained) gyp project written in Python.

[mmarchini]

So any attempt to migrate out of gyp will need to be coordinated with npm and possibly other package managers 🤔

[nschonni]

Here is the high-level current hook for the NPM side https://docs.npmjs.com/misc/scripts#default-values
Potentially, the migration would for NPM to look for a CMakeList.txt and run cmake-js instead in this existing logic

[nschonni]

Oh, and it would be nice to sketch out some things like nodejs/node#30139

[mmarchini]

Thanks for the link. The idea is to deprecate and eventually discontinue gyp altogether, so we still would need to work with npm (and any other package manager that bundles gyp) to remove gyp from there.

Oh, and it would be nice to sketch out some things like nodejs/node#30139

IMO this is out of scope for this initiative, but if someone has the time and energy to work on it I won't object.

[mhdawson]

It depends on what we do in terms of toolchain for Node.js core, but given that cmake is already an option for native addons at least for those using N-API, cmake would be a good option if that aligns with what core does.

[DeeDeeG]

I think cmake-js sounds good. (If only because I don't know of a Linux/macOS/Windows option other than node-gyp and cmake-js, and the point is to move away from gyp. I don't think I saw a serious alternative raised to cmake-js yet.)

I think folks can start using that right away, although cmake-js integration with npm/Yarn/pnpm/etc. like @nschonni mentioned (#4 (comment)), on par with the current node-gyp integrations to said package managers, would be GREAT.

I don't think that whatever is used to build Node has any impact on what addon authors need to use. (So this issue can be thought of as a totally separate question from "How to build Node," as far as I can tell.)

I see this issue as being a "get the word out"/documentation-only issue, short of the integration with package managers, and perhaps coordinating when/how to responsibly deprecate node-gyp, which are all really nice things to do as stewardship of the ecosystem, but technically optional. (Technically, cmake-js existing is the Minimum Viable Product/proof of concept for providing a path away from node-gyp, and with it all use of gyp on the "native addons" side of things.)

I foresee node-gyp hanging around for a long time for building older, unmaintained packages, as they have bindings written as binding.gyp files, and they require node-gyp to build. I think if a binding.gyp could be seamlessly converted to the equivalent CMakeLists.txt for cmake-js, or if the ecosystem moves on from those umaintained packages, then node-gyp could be dropped once and for all. (And to reiterate: I do think dropping node-gyp means fully dropping gyp on the "native addons" side of things, i.e. dropping node-gyp means "mission accomplished" for this issue.)

P.S. as a matter of smoothing out/reducing user pain points switching to cmake-js, it occurs to me that cmake-js expects cmake on the PATH, whereas node-gyp bundles its own copy of gyp. Perhaps cmake-js should bundle cmake binaries?

[richardlau]

node-gyp is able to bundle gyp as gyp is written in Python and doesn't need to be compiled for different platforms/architectures. You do, however, need Python installed.

[DeeDeeG]

Okay, bundling cmake binaries in cmake-js might not work so great. (I suppose bundling a separate cmake binary for each supported platform would be the cost of bundling it in cmake-js, without even getting into the minutiae of shipping binaries that work on "this Linux vs that Linux", "this macOS vs that macOS"...)

Back to the main topic at hand: Most of my comment above was restating things from earlier in the thread, but I hoped to highlight which things are actionable. (Basically honing in on "how to make using cmake-js as nice as using node-gyp is now." And messaging/communicating/documenting the transition away from node-gyp, and maybe deprecating the node-gyp package in a responsible way.)

[DeeDeeG]

For helping native addons stop depending on gyp (weaning off of node-gyp), here's an interesting comment: nodejs/node#21410 (comment)

Also apropos cmake, I'm reasonably confident I can teach node-gyp to convert a binding.gyp to CMakeLists.txt on the fly and effectively phase out GYP.

[vweevers]

Many popular native addons ship prebuilt binaries, which can help migration because it means a smaller percentage of end users (that e.g. use a platform for which no prebuilt binaries are available) actually need a build toolchain.

Two years ago I collected data from the npm registry (to see which tools are used, how many native addons ship prebuilds, etc). When I get a chance I will dust that off and share updated data here.

[mmarchini]

I think the next step is to talk with npm about decoupling node-gyp from there (I opened #10 with one option but I believe there are other strategies we could adopt). I'll see if we can get some npm engagement here.

I see this issue as being a "get the word out"/documentation-only issue

FWIW I disagree with this statement, even thought node-gyp is bundled by npm, it's maintained by us today. And it doens't matter to users if node-gyp is bundled by npm or us, they get it whenever Node.js is installed, and it's even in our API docs. Future-proofing work as well as migration tooling will be necessary as part of this effort.

[vweevers]

Many popular native addons ship prebuilt binaries, which can help migration because it means a smaller percentage of end users (that e.g. use a platform for which no prebuilt binaries are available) actually need a build toolchain.

Correction: doesn't help much because the majority of packages does not use prebuilt binaries. Take the following with a grain of salt (needs a second pass to cleanup data) but of 991 npm packages:

• 623 (62.9%) use node-gyp implicitly or explicitly, without prebuilt binaries (combined downloads: 189m)
• 81 (8.2%) use prebuild-install (combined downloads: 11m)
• 73 (7.4%) use node-gyp-build (combined downloads: 28m)
• 68 (6.9%) use @mapbox/node-pre-gyp (combined downloads: 8.8m)
• 63 (6.4%) use a custom install script (combined downloads: 24m)
• 61 (6.2%) use node-pre-gyp (combined downloads: 3.3m)
• 10 (1.0%) use cmake-js (combined downloads: 50k)
• 6 (0.6%) use cargo-cp-artifact (combined downloads: 146k)
• 6 (0.6%) use neon-cli (combined downloads: 12k)

For more, see data.md. Note, this data can't be used to determine popularity of e.g. cmake-js. For example, if a package depends on cmake-js but its install script runs node-gyp then I count it as node-gyp.

[nschonni]

Just because they use node-gyp doesn't mean there is no prebuilt. node-sass had to roll our own because it predated many of those packages. Caching of those binaries is a related issue that only some of those build tools have tackled nodejs/node#30139

Edit: I see you do separate out the custom versions