Should we bundle the toolchain to build native modules as part of Node.js?

[mmarchini]

forking discussion from #4

Today node-gyp is bundled with npm (which is bundled with Node.js), therefore we don't provide a build toolchain for native addons on our own. I wonder if as part of the node-gyp deprecation and migration to something else, we should start bundling that something else with Node.js. Building a native module is more coupled to the Node.js version than it is to the package manager, so from a design perspective it makes sense. It also ensures that deprecations and changes are properly propagated across package managers.

Just bundling would be easy, but I think we can go one step further and provide APIs that allow package managers to identify build files (so that npm install can still run the build toolchain when those are present) as well as provide APIs that abstract the necessary build parameters (defines, includes, etc) so that community can safely write their own build toolchains (not limiting folks to whatever we decide to use). Depending on how well we can write those APIs, maybe we don't even need to bundle anything (bold idea, I know).

As for what a plan would look like, I think the first step would be to move node-gyp to be bundled with Node.js instead of package managers, that'll give us more control over the deprecation moving forward.

P.S.: I haven't looked into how yarn and pnpm handle node-gyp yet (specifically, if they bundle it themselves or not).

[richardlau]

W.r.t. API’s there was an old stalled out PR that tried to do away with node-gyp and provide API’s to invoke the compile and link steps: nodejs/node#7440

[mmarchini]

fascinating, although I was thinking something more abstract than that: the API only provides the information needed to build addons and we leave the "invoke and link" to the packages. And if we decide to bundle anything we can use this API to implement whatever we provide.

[mmarchini]

The API can even come before deprecating gyp, and we can use it on node-gyp to both validate and remove the dependency surface on node-gyp. If we are able to decouple node-gyp from package managers, and decouple build configs from node-gyp, swapping it for another toolchain will be straightforward (implementation wise), and providing two toolchains will be possible with less code duplication.

[mhdawson]

On the surface decoupling node-gyp from npm and packaging instead with Node.js sounds like a good first step. I'd be interested in what @rvagg thinks about that.

[rvagg]

I think: gyp is kind of garbage, but it's the best garbage we have and our ecosystem has standardised around it. It's already "official" so where it gets bundled doesn't really seem to matter since npm is bundled anyway. What should concern people is the maintenance of gyp and node-gyp and its viability as something that not many people care deeply enough about to keep updated. Really it's just something we keep on life support because we know it's important but don't have the drive to do anything better (mind you, there are some folks that seem to want to contribute more radical improvements, but it's such an important piece of software that shepherding major changes in is a huge undertaking in itself cause there's so so much potential for breakage!).

Aside from that, no strong opinions. If there's energy for something like this, maybe you should be considering alternatives that might offer better user experience and future proofing? Investing in getting ninja.js going (stalled, but we nearly got to releasing an alpha), or pulling in cmake.js (there seems to be a lot of interest in it in the ecosystem) might be worthwhile. But if you just want consistency then gyp+node-gyp is fine, it just needs maintenance and love all the way down and we around here are all that are doing that.

[mmarchini]

@rvagg for context, the idea is to move node-gyp to Node.js as a first step into moving to something else (cmake-js or whatever we choose). By moving it to Node.js now we have more control over the deprecation process (instead of having npm and other package managers as the middle man). Short term this will allow us to experiment faster. Long term we can provide a more seamless migration experience for the ecosystem.

[targos]

pnpm depends on node-gyp:
https://github.com/pnpm/pnpm/blob/21cd0f260191729a71115e1ae16654d2b4faf32d/packages/pnpm/package.json#L18-L20
https://github.com/pnpm/pnpm/blob/21cd0f260191729a71115e1ae16654d2b4faf32d/packages/pnpm/package.json#L164-L171

I'm not sure about yarn, but it seems to require modules or users to install node-gyp themselves.
/cc @arcanis

[arcanis]

I'm not sure about yarn, but it seems to require modules or users to install node-gyp themselves.

Yarn 1 used whatever node-gyp is available (either explicitly, or by accident due to hoisting, or in magic locations), or transparently installs it as a global package if it isn't there to be found. It's not a good strategy, since it encourages packages to omit depending on node-gyp, thus bypassing the lockfile and making installs less predictable.

Modern releases now require packages depending on node-gyp to either list it in their dependencies, or to be installed in a project where node-gyp is explicitly declared as a dependency/devDependency of the top-level, thus guaranteeing that the version used is consistent now and in the future (using the wrong node-gyp version used to break installs which previously worked fine).

---

Having a native build mechanism tied to Node doesn't strike me as a bad idea, my main worry would be that unless given special consideration it'd prevent retroactively applying build pipeline improvements to older Node releases (vs a package, which can be upgraded as any other dependency). For instance, say you later want to provide a utility in this pipeline to easily build WASM modules, would only new Node releases be able to use it?

In my opinion it really depends how much you plan to maintain this build pipeline: if it frequently receives new features, then a separate package seem to make more sense. If it's expected to frequently receive breaking changes, then making it builtin would be safer. Or perhaps the best would be a mix of both, with a low-level Node API that would rarely change (similar to napi in idea) and a companion package whose interface would regularly improve and which could be used with any Node version supporting this low-level API.

[vweevers]

It's not a good strategy, since it encourages packages to omit depending on node-gyp, thus bypassing the lockfile and making installs less predictable.

It's not that clear-cut, from a wider perspective than Yarn. Omitting node-gyp gives the user a choice (e.g. if you have Python 2 installed: use old node-gyp, if Python 3 then latest). It saves bytes on disk as node-gyp is a relatively big package (2 MB unpacked). If an addon ships prebuilt binaries, then node-gyp is only a fallback for lesser-used platforms (for which the addon maintainer likely isn't giving guarantees or support) or an opt-in tool for wanting to build from source.

[mmarchini]

Omitting altogether is far in the future, because today node-gyp is heavily dependent on the Node.js version it's shipped with (another reason to bundle node-gyp within Node.js and not npm).

[vweevers]

If the intent of bundling is to gradually replace node-gyp, with a Node.js API that would wrap node-gyp and eventually something else, how do we prevent that API from becoming the next node-gyp or cmake-js?

It has to deal with a lot of tedious details that aren't easily abstracted away.

[mmarchini]

The idea is not to wrap node-gyp. The API I'm thinking would me more high level to simplify the build process from a user perspective (and that could be reused by package managers to implement their build methods). The API would also provide any Node.js-specific flags and header paths that need to be used when building an addon, but how to use those flags and paths would be responsibility of the build toolchains (like cmake-js). Note that
this is an early proposal, nothing is defined yet.

[vweevers]

Note that this is an early proposal, nothing is defined yet.

Roger. It's too abstract for me atm, but I'm interested (mainly as co-maintainer in the prebuild org) to see where it goes.

[mhdawson]

@vweevers good to know you are keeping an eye on it. It's important that we consider all of the aspects like prebuild etc.