(lambda-nodejs): Unable to use some packages since move to esbuild

[adam-nielsen]

This is a continuation of AWS support case 8536830561, where I was advised to open the issue here as it appears to be a bug introduced in a recent CDK change.

Since the move to esbuild, stacks that previously deployed without issue will no longer build. This seems to affect any Lambda function that uses a module that requires a compilation step.

Reproduction Steps

npm install https://github.com/adam-nielsen/netcdf4

Add to a Lambda:

const netcdf4 = require('netcdf4');
let cdf = new netcdf4.File('dummy.cdf', 'r');

Try to build it:

cdk synth
> asset-input/test.js:1:24: error: No loader is configured for ".node" files: asset-input/node_modules/netcdf4/build/Release/netcdf4.node
    1 │ const netcdf4 = require('netcdf4');

Note the module being installed is my own fork of an upstream one modified to compile in a more recent version of Node. I am happy to further modify it to get it to work with esbuild, if that's an option.

There is a minimal working example here: http://uq-ads-aws-support-public.s3-website-ap-southeast-2.amazonaws.com/cdk-bundle-test.zip

It can be run like so:

npm install
cd lambda
npm install
cd ..
cdk synth

What did you expect to happen?

I expected it to build and deploy as it did with CDK 1.73.0.

What actually happened?

It failed with an error:

> asset-input/test.js:1:24: error: No loader is configured for ".node" files: asset-input/node_modules/netcdf4/build/Release/netcdf4.node
    1 │ const netcdf4 = require('netcdf4');

Environment

• CDK CLI Version : 1.109.0
• Framework Version: 1.110.1
• Node.js Version: v16.2.0
• OS : Arch Linux
• Language (Version): Javascript

Other

There appears to be some discussion upstream about this issue however I am not sure whether that proposed fix can be put into CDK.

---

This is 🐛 Bug Report

[BenChaimberg]

@adam-nielsen I've tried to take a stab at reproducing this error but have run into trouble getting libnetcdf4 to install. Would you be able to try to set up a smaller reproduction that doesn't depend on a large external library so that someone from the team can easily set it up? I would also appreciate some more information on ".node" files as I haven't been able to find any information on them.

[adam-nielsen]

Many thanks for having a look at this! I have since found the talib package has the same problem and doesn't appear to have as many external dependencies.

Essentially you can pick any CDK stack and try to add talib as a dependency and it will no longer build. Here is a smaller reproduction demonstrating the issue: cdk-bundle-test.zip

npm install
cd lambda
npm install
cd ..
cdk synth

I don't know much about the .node files but it looks like when you compile a NodeJS plugin (i.e. C++ code that hooks into the Javascript engine) it produces a native code library (like a .dll or .so file) but stores it with a .node extension. It seems that if you point your package entry point to this .node file instead of a .js file, then this library gets loaded into the JS engine. It seems these .node files are produced by the node-gyp tool, which is the official way of building native modules on various operating systems.

That's about the extent of what I could find out from my research, as I've never written a module like this, only consumed them like any other NodeJS module.

[privogpynes]

I had this issue a while back deploying lambda@edge with the sharp module as it also has native dependencies. You're spot on about the compiled native outputs, they're binaries which is why you can't actually bundle them into your js the way esbuild wants.

I took a stab at solving your issue using the sample you provided. When I tried to download that talib package you had though, even trying to just npm install it locally gave me a bunch of install/compile errors, same with your netcdf4 package. So I replaced them with sharp and took a look at your lib/cdk-bundle-test-stack.js and noticed you didn't have the nodeModules property inside the bundling options for your NodejsFunction. I popped it in there and also turned on the forceDockerBundling option so that the native libs get built using the the proper lambda os environment instead of my computer's os and everything works fine. I don't think you need the forceDockerBundling since esbuild wasn't in your dev deps, but tossed in regardless.

Here's the version I got working. Not sure if this helps you out or not
cdk-bundle-test.zip

[nija-at]

Thanks @privogpynes for looking and helping out on this.

@adam-nielsen - can you check if this helps you?

[adam-nielsen]

My apologies I meant to look at this sooner! Yes, it does appear to have fixed the problem, although I am not entirely sure how.

Part of the issue was that I didn't have my bundling options inside the bundling property, which I think was causing nodeModules to be ignored (and thus it tried to bundle the modules that can't be bundled).

I also had to adjust some git settings inside the bundling container as it was trying to use SSH to clone a public module from git, but there are no host keys or private keys in the container so it was failing. By forcing it to the HTTPS instead, it can clone without needing any credentials.

There also seems to be a bug in CDK that requires you to specify ALL the commandHooks, you can't specify just one.

For the record, this is the bundling section that now works with the netcdf4 module:

bundling: {
	commandHooks: {
		beforeInstall: (inputDir, outputDir) => ([
			// Force using HTTPS instead of SSH to connect to GitHub, to avoid
			// errors about missing SSH keys inside the bundling container.
			'git config --global url."https://github.com/".insteadOf ssh://git@github.com',
		]),

		// We don't need these but the CDK CLI fails with errors if they are missing.
		beforeBundling: () => {},
		afterBundling: () => {},
	},

	// We have to do the packaging in a custom Docker container, as certain
	// C libraries must be available or installing the netcdf4 module will
	// fail.  This container mirrors the one normally used by CDK for this,
	// but with the addition of these C libraries already installed.
	dockerImage: cdk.BundlingDockerImage.fromAsset(
		__dirname + '/../netcdf-docker/'
	),
	forceDockerBundling: true,

	// Don't bundle these modules, just include them in node_modules.
	nodeModules: [
		'netcdf4',
		'pg',
	],

	// Completely ignore this module, and don't complain that it is unavailable.  This is
	// because the code in the pg module checks to see if the pg-native module exists
	// and falls back to other functions if not, but the check itself causes esbuild to try
	// to bundle in the module, even though it doesn't exist and we don't want to use it.
	externalModules: [
		'pg-native',
	],
}

In the netcdf-docker folder referenced in the dockerImage above, is a Dockerfile that is identical to the upstream one with a couple of extra commands on the end to install the various netcdf4 C libraries required to compile the NodeJS module.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.