Cannot find module error for Node-provided package

[benmccann]

- Do you want to request a feature or report a bug? bug

- What is the current behavior? build fails with error Cannot find module 'stream/package.json'

- If the current behavior is a bug, please provide the steps to reproduce.

• Create a new SvelteKit project with npm init svelte@next
• Install the Netlify adapter: https://github.com/sveltejs/kit/tree/master/packages/adapter-netlify#installation
• Create an .nvmrc with the contents 16 to use Node 16
• Deploy to Netlify

- What is the expected behavior?

There should be no need to install a stream package because it is provided by Node: https://nodejs.org/api/stream.html#stream_stream

I'm not sure why it's trying to find stream/package.json and failing since this is a built-in package

- Please mention your node.js, and operating system version.

Node 16. Linux

[netlify-team-account-1]

Hi @benmccann, thank you for reporting this issue! I have an idea of what this may be caused by. Will look into it more tomorrow.

[netlify-team-account-1]

I was able to fix this by upgrading netlify-cli to 6.12.6. Can you check if that works for you, too?

[marovargovcik]

I face the same issue. I put together a repository where behavior can be reproduced: https://github.com/marovargovcik/netlify-functions-demo Function is located in src/functions directory and is transpilled to ES5 with rollup by running npm run build and outputted to dist/functions. Here is the deploy log https://gist.github.com/marovargovcik/ff6989665cc347ce9cf9cba36cbb5598.

I am trying to have Netlify function written in modern JavaScript (?? operator, ?. property accessing) and run by Netlify without complaining about syntax errors (? unexpected character) or having a problem with import statements etc.

[benmccann]

I was able to fix this by upgrading netlify-cli to 6.12.6. Can you check if that works for you, too?

I'm not using netlify-cli at all. I'm just doing a git push and Netlify is building the git repo, so if there's some Netlify tooling that's outdated it would be on the Netlify side where that upgrade would need to take place

[netlify-team-account-1]

Did some more digging. This error happens when you're using zip-it-and-ship-it's dependency resolution mechanism, which uses the Node's builtin process.binding("natives") to get a list of which dependencies are native (like fs or path or stream/web) and which are not (like react or is-even). Relying on process.binding("natives") means that it depends on whatever Node version is used to do the bundling. Since stream/web, which is used by your dependency fetch-blob, is a Node v16 feature, it rightfully fails (aws lambda, and by extension netlify functions, don't yet support Node v16). What zip-it-and-ship-it doesn't detect: fetch-blob accounted for that to happen, and has a fallback strategy. So it fails wrongfully.

You can unblock yourselves by using the esbuild bundler (which I see you already found out via sveltejs/kit#2605). In your case @marovargovcik this has the additional upside of replacing the rollup bundling stage altogether, since esbuild does all the things rollup does for you.

[benmccann]

it rightfully fails (aws lambda, and by extension netlify functions, don't yet support Node v16)

Ah, this was entirely unclear to me. I had checked the docs and they make no mention of Node 16 being unsupported. I had put 16 in my .nvmrc file and Netlify didn't seem to complain about it. It would be nice if it just bombed out immediately and told you Node 14 was the latest supported version if you tried to use something newer

[benmccann]

As a potential fix to this, what if we changed fetch-blob to import node:stream/web instead of stream/web (this is a supported syntax for core modules). Then zip-it-and-ship-it could just ignore any imports starting with node:. I don't know if it does that already or not, but if it doesn't perhaps it could be made to? I've sent a PR to make that change in fetch-blob and they've approved it

[benmccann]

I also wonder why zip-it-and-ship-it needs to scan imports at all. I've provided a package.json. Why don't you just install what's in the dependencies? Obviously you couldn't just make that change as it'd be a breaking change, but perhaps it could be an opt-in?

[Xenonym]

Ah, this was entirely unclear to me. I had checked the docs and they make no mention of Node 16 being unsupported. I had put 16 in my .nvmrc file and Netlify didn't seem to complain about it.

@benmccann some confusion here I think: .nvmrc specifies the version of Node used in the build environment, which is seperate from the version of Node in the AWS Lambda runtime environment.

As per the docs, the build environment supports any released version of Node, while the AWS Lambda runtime environment (specified by AWS_LAMBDA_JS_RUNTIME) supports only those versions AWS Lambda supports, which is up to Node.js 14 at the moment.

I also wonder why zip-it-and-ship-it needs to scan imports at all. I've provided a package.json. Why don't you just install what's in the dependencies?

Netlify functions don't actually need to be their own package, and the functions directory does not need to have a package.json. The recommended workflow is to install function dependencies in the same package.json as the project base:

To optimize build performance, we recommend specifying function dependencies in the top-level package.json file in the site’s base directory.

ZISI then zips only those modules require()ed by functions for deployment, and skips over those used for building the site.

[benmccann]

ZISI then zips only those modules require()ed by functions for deployment, and skips over those used for building the site.

Perhaps there should be an option to continue if the package cannot be found? Or alternatively, or in addition, we could use the node: prefix option suggested above

[netlify-team-account-1]

@Xenonym I couldn't have said it better! Some more context about why we need to keep function size down:

AWS Lambda constrains total bundle size to 50mb. Because your package.json likely contains more than Function dependencies, like some build framework or SSG, we need to be smart about what to include in the Lambda bundle.

The observed issue, where zip-it-and-ship-it fails at bundling a native module, is a defect on our end. #746 deals with it. This issue won't occur if you set node_bundler = "esbuild", which is most likely the best solution to this. (we're thinking about making esbuild our new default bundler)

[benmccann]

precinct was upgraded here: #773

However, I'm still getting deployment failures: netlify/cli#3533 (comment)

[benmccann]

This is now working. Existing projects will need to "clear cache and deploy"

Thanks for all the help!!

[vldmrkl]

@benmccann I'm still experiencing this issue on my existing project even after clearing the cache and re-deploying.

@netlify-team-account-1 do you know what could go wrong?

I had to change the node_bundler to esbuild to make it work.

[netlify-team-account-1]

Hey @vldmrkl! I'm not aware of any changes that could've lead to this. We introduced a related test case when this issue first occurred, and that's still green.

Could you open a new issue and provide me with a reproduction case? That'd make it a lot easier for me to help you with this :)