Compatibility with bundling tools like webpack and rollup

[raviqqe]

A way to find binaries by contents of package.json files seems to be needed for compatibility with JS bundlers.

As this code in gRPC's repo, we need to pass paths to package.json to the find(package_json_path, opts) function in lib/pre-binding.js currently.
However, when the paths to package.json are hard-coded in source files like the one of gRPC, bundling tools, such as webpack and rollup, cannot detect the dynamic loading of external files and end up creating bundled files which don't work.

So, we need some function like findByPackageInfo(package_json_content, opts). Then, we can change the code of gRPC like:

var binary = require('node-pre-gyp/lib/pre-binding');
var path = require('path');
var binding_path = binary.findByPackageInfo(require('../../../package.json'));
var binding = require(binding_path);

Any thoughts?

References

• The similar issue of gRPC
• webpack | Module Resolution

[dperetti]

Is node-pre-gyp supposed to work with Webpack... or not ? I've been spending hours on getting a third party library to work, and it seems to boil down to this "package.json does not exist" issue.

[pronebird]

I actually experience the following error with webpack:

Error: Can't resolve 'aws-sdk'

Coming from ./node_modules/node-pre-gyp/lib/unpublish.js

I think it's suggested to wrap require calls for devDependencies in try+catch but I may be wrong.

[dperetti]

I have eventually sorted it out, as far as I'm concerned.
Please read my comment here.

[whisper-bye]

+1

./node_modules/fsevents/node_modules/node-pre-gyp/lib/info.js
Module not found: Can't resolve 'aws-sdk' in '/tmp/demo/node_modules/fsevents/node_modules/node-pre-gyp/lib'

[springmeyer]

Is node-pre-gyp supposed to work with Webpack... or not ?

No. I designed node-pre-gyp and I've never used webpack nor do I understand what it is. So, its definitely not supposed to work. That said if it is feasible to get it working, I'd review a PR with tests. Until then I'll close this issue to avoid confusion/the assumption that things should work.

[rajivshah3]

If anyone comes here looking for a solution for Webpack, this fixed the problem for us:

I solved it by adding

externals: { 'sqlite3':'commonjs sqlite3', }

to the webpack config file

Simply replace sqlite3 with the the dependency that uses node-pre-gyp (in our case, argon2)

[dmarcs]

This pull request #622 solves the main problem that prevents @mapbox/node-pre-gyp from working with webpack.

webpack needs to know where package.json and the .node file are located during the build process. Right now both locations are determined at run-time instead of during the build process.

Using externals as described above is a valid workaround, but it requires that the package is available at run-time. This means that after you build with webpack, you need to run npm install package. It would be better if we could avoid npm install package since it adds an extra step.

@springmeyer a couple reasons webpack is useful include

1. Say you have a node app written in Typescript. You normally have to convert the Typescript (server.ts) to Javascript (server.js), and then run node server.js. If you make a change to server.ts you now need to recompile the Typescript to Javascript, stop server.js, and re-run server.js. With webpack you can make a change to server.ts, and you don't need to recompile, stop the server, and re-run the server. It does all of those steps for you. You could use nodemon and typescript watch instead of webpack for this though.

2. If you have a node project with many relative imports, for example import '../../../../src/file.js' the code can become unreadable. With webpack we can get rid of every ../ and write code that looks cleaner, import 'src/file.js' in this example.

You could argue that webpack isn't necessary (it's much more useful for front-end development than back-end development), but you might be stuck using a node framework that depends on webpack.

[inmyth]

To add why we use webpack, it's for deployment in lambda.
EDIT: if you like me are having this issue due to argon2, there is another library that is webpack-able called argon2-browser. The catch is it's much slower.

[Rush]

@springmeyer any chance you can merge #622 ?

[sibelius]

any workaround for this ?

[Bessonov]

Ran into this issue as well. @springmeyer @dmarcs, is there any reason why #622 was closed instead of being merged?

[Bessonov]

Oh gosh, I ran into different issues, but I finally found a workaround by hijacking the binary through Webpack. It might be helpful to others. Feel free to optimize it and share it with the community.

Some Issues I Encountered:

• Can not import .node file in esm nodejs/node#40541: My codebase is ESM-only. Using a bcrypt.cjs file with module.exports = require('bcrypt') didn’t work because Webpack seems to convert it to ESM.
• The instance I obtained with __non_webpack_require__ didn’t behave as documented. For example, non-sync methods should return a promise if no callback is passed, but instead, it throws a TypeError: 3 arguments expected.

The Workaround:

// webpack.config.mjs

import { dirname, resolve } from 'path'
import { fileURLToPath } from 'url'

const context = dirname(fileURLToPath(import.meta.url))

export default {
	// [...]
	target: 'node',
	entry: {
		bcrypt: {
			import: resolve(context, 'node_modules/bcrypt/lib/binding/napi-v3/bcrypt_lib.node'),
		},
		// [...]
	},
	module: {
		rules: [
			{
				test: /\.node$/,
				type: 'asset/resource', // Possibly works with node-loader as well
				generator: {
					filename: pathData => `${pathData.runtime}.node`,
				},
			},
		],
	},
}

// server.mts
import { promisify } from 'util'

const bcrypt = (() => {
	// @ts-expect-error
	const bcrypt = __non_webpack_require__('./bcrypt.node')
	return {
		compare: promisify(bcrypt.compare) as (data: string | Buffer, encrypted: string) => Promise<boolean>,
		// [...]
	}
})()

After building, my dist directory contains:

-rw-r--r-- 1 dev dev 102128 Dec  2 12:26 bcrypt.node
-rw-r--r-- 1 dev dev 409272 Dec  2 12:26 server.mjs
[...]

[Bessonov]

Not drop-in replacement, but alternative to __non_webpack_require__:

import { createRequire } from 'module'

const bcrypt = (() => {
	const bcrypt = createRequire(import.meta.url)('../dist/bcrypt.node') // import.meta.url is [...]/build/server.mjs
	[...]