Functions not being built and packaged?

[jimniels]

I've been trying to get a simple set of functions setup and running and it keeps failing and I can't figure out why.

My netlify.toml looks like this:

[build]
  command = "npm run build"
  functions = "dist"

And my npm run build command is just netlify-lambda build src

If I run npm run build locally, everything works just fine.

> npm run build

> api@1.0.0 build /Users/jimnielsen/Sites/jim-nielsen.com/api
> netlify-lambda build src

netlify-lambda: Building functions
Hash: 8faa90c77ae883f3a8c3
Version: webpack 4.30.0
Time: 1117ms
Built at: 04/16/2019 4:04:00 PM
              Asset      Size  Chunks             Chunk Names
           hello.js  1.03 KiB       0  [emitted]  hello
icg-slack-deploy.js  18.5 KiB       1  [emitted]  icg-slack-deploy
Entrypoint hello = hello.js
Entrypoint icg-slack-deploy = icg-slack-deploy.js
[0] external "stream" 42 bytes {1} [built]
[1] external "zlib" 42 bytes {1} [built]
[2] external "url" 42 bytes {1} [built]
[3] external "http" 42 bytes {1} [built]
[4] external "https" 42 bytes {1} [built]
[5] ./hello.js 123 bytes {0} [built]
[6] ./icg-slack-deploy.js 2.19 KiB {1} [built]
[7] ../node_modules/node-fetch/lib/index.mjs 39.6 KiB {1} [built]

But as soon as I push, the build fails:

3:55:55 PM: Executing user command: npm run build
3:55:56 PM: > api@1.0.0 build /opt/build/repo
3:55:56 PM: > netlify-lambda build src
3:55:56 PM: netlify-lambda: Building functions
3:55:58 PM: Hash: a6669134aaaa082f8a2e
3:55:58 PM: Version: webpack 4.30.0
3:55:58 PM: Time: 1634ms
3:55:58 PM: Built at: 04/16/2019 9:55:58 PM
3:55:58 PM:               Asset      Size  Chunks             Chunk Names
3:55:58 PM:            hello.js  1.03 KiB       0  [emitted]  hello
3:55:58 PM: icg-slack-deploy.js  19.5 KiB       1  [emitted]  icg-slack-deploy
3:55:58 PM: Entrypoint hello = hello.js
3:55:58 PM: Entrypoint icg-slack-deploy = icg-slack-deploy.js
3:55:58 PM:  [0] external "stream" 42 bytes {1} [built]
3:55:58 PM:  [1] external "zlib" 42 bytes {1} [built]
3:55:58 PM:  [2] external "url" 42 bytes {1} [built]
3:55:58 PM:  [3] external "http" 42 bytes {1} [built]
3:55:58 PM:  [4] external "https" 42 bytes {1} [built]
3:55:58 PM:  [5] ./hello.js 123 bytes {0} [built]
3:55:58 PM:  [6] ./icg-slack-deploy.js 2.19 KiB {1} [built]
3:55:58 PM:  [7] ../node_modules/dotenv/lib/main.js 2.87 KiB {1} [built]
3:55:58 PM:  [8] external "fs" 42 bytes {1} [built]
3:55:58 PM:  [9] external "path" 42 bytes {1} [built]
3:55:58 PM: [10] ../node_modules/node-fetch/lib/index.mjs 39.6 KiB {1} [built]
3:55:58 PM: Build script success
3:55:58 PM: Starting to prepare functions from '/'
3:55:58 PM: Zipping functions from /opt/build/repo/dist to /tmp/zisi-940663858
3:56:04 PM: Error: Could not find "encoding" module in file: /repo/dist/icg-slack-deploy.js. 
          
Please ensure "encoding" is installed in the project.
3:56:04 PM: Failing build: Failed to prepare functions for deployment
3:56:04 PM: failed during stage 'preparing functions for deployment': exit status 1
3:56:05 PM: Finished processing build request in 23.159125539s
3:56:05 PM: Shutting down logging, 0 messages pending

This is the interesting line to me: "Could not find "encoding" module in file: /repo/dist/icg-slack-deploy.js."

What's interesting is that I am only requireing one dependency in that file:

const fetch = require('node-fetch');

That line of code is making everything fail. If I comment that file out, Netlify builds everything as expected (but then the function doesn't work, because I can't use fetch).

Am I missing something here?

[bcomnes]

We recently added support for unbundled function deployment to the buildbot and CLI (via https://github.com/netlify/zip-it-and-ship-it if you are curious). That means, you can deploy js functions with dependencies without bundling them first with a tool like netlify-lambda. Your deployed functions folder can contain function files that require modules in a node_modules in our outside the deployed functions folder. On deployment, we walk the dependency tree and grab all the dependencies the functions need. In general, it should work for bundled and unbundled functions. For some reason, its finding node-fetch requires an encoding module that appears to be consumed as a peer dependency (e.g. node-fetch requires it, but doesn't ship a reference in its package.json).

The solution now is to install encoding in the nearest package.json. We need to look into if this is an issue with node-fetch that needs correcting, or if our dependency algorithm is just missing stuff, but there is an action you can take now.

[bcomnes]

I'll open a tracking issue for encoding + node-fetch and reference this.

[swyxio]

hell is underspecified dependencies.

[jimniels]

We recently added support for unbundled function deployment to the buildbot and CLI...That means, you can deploy js functions with dependencies without bundling them first with a tool like netlify-lambda

That's actually pretty neat. I haven't messed with functions and netlify-lambda since a 0.x release of some sort, so it's cool to see the new stuff. Plus this model fits what I'm trying to do. My functions are simple and I just need their node_modules deps brought in.

So, in this case, I decided that rather than troubleshoot netlify-lamdba + encoding, I would just switch over to this "new" way of doing things.

Unfortunately, I didn't get very far.

First, I installed the CLI then ran netlify dev functions:create hello

> netlify functions:create hello
? Pick a template js-node-fetch
◈ Creating function hello
◈ Created src/hello/node-fetch.js
◈ Created src/hello/package.json
✔ installed dependencies for hello

Which resulted in this:

I removed the build command from my netlify.toml file and had it only pointing at my folder of functions:

[build]
  functions = "./src"

As stated in the docs (and as you alluded to):

all the functions created by netlify functions:create require no build step

So It would seem, at this point, that I have the most basic setup you could have for deploying functions: I created a new function with netlify dev functions:create, I point at that folder in my netlify.toml, and it's still not working. It works locally when I run netlify dev, I can go to http://localhost:8888/.netlify/functions/hello and I get the expected response. But when I push to Netlify, the build fails:

11:39:34 PM: Found netlify.toml. Overriding site configuration
11:39:34 PM: Different functions path detected, going to use the one specified in the toml file: './src' versus '' in the site
11:39:34 PM: No build command found, continuing to publishing
11:39:34 PM: Starting to prepare functions from '/'
11:39:34 PM: Zipping functions from /opt/build/repo/src to /tmp/zisi-624832131
11:39:35 PM: Error: Could not find "node-fetch" module in file: /hello/hello.js. 
          
Please ensure "node-fetch" is installed in the project.
11:39:35 PM: Failing build: Failed to prepare functions for deployment
11:39:35 PM: failed during stage 'preparing functions for deployment': exit status 1
11:39:35 PM: Finished processing build request in 5.468181506s
11:39:35 PM: Shutting down logging, 0 messages pending

Am I missing something here?

[swyxio]

yea you caught us at the early stage where we still havent implemented package installing per function folder i think (@bcomnes knows, i dont work on that part). so basically right now the only way to do it is build locally and then netlify deploy —prod. but ofc we will be making sure those node modules install eventually.

[jimniels]

Haha ok well, so I stopped using this "new" method and instead went back to the "old" way of doing things, which funnily enough is where I started when I first posted this issue.

So to resolve the first issue I had, I added encoding to my package.json (as recommended) and that finally made the build succeed in netlify when pushed to GitHub.

However, now my simple hello.js function doesn't actually work. After some troubleshooting, I found once again it seems to stem from use of node-fetch. When I don't use node-fetch at all, my functions deploy and they actually work. Example of hello.js:

exports.handler = (event, context, callback) => {
  callback(null, {
    statusCode: 200,
    body: "Hello, World!"
  });
}

But as soon as I try to bring in a fetch call, it fails.

const fetch = require("node-fetch");
exports.handler = (event, context, callback) => {
  callback(null, {
    statusCode: 200,
    body: "Hello, World!"
  });

  fetch(
    "https://icanhazdadjoke.com/",
    { method: "GET", headers: { Accept: "application/json" } }
  )
    .then(res => res.json())
    .then(json => { console.log(json) })
}

That code results in Netlify functions logging this:

8:39:33 AM: hello invoked
8:39:33 AM: TypeError: o is not a function
    at t.handler (/var/task/hello.js:1:18480)
RequestId: 249caac6-9e6b-4c76-86a8-9b45d70d44a5 Process exited before completing request

When I run netlify-lamdba serve locally to test this, I get something similar when I hit http://localhost:9000/.netlify/functions/hello

Lambda server is listening on 9000
Request from ::1: GET /.netlify/functions/hello
Response with status 200 in 10 ms.
TypeError: o is not a function
    at Object.t.handler (/Users/jimnielsen/Sites/jim-nielsen.com/api/build/hello.js:1:18480)
    at /Users/jimnielsen/Sites/jim-nielsen.com/api/node_modules/netlify-lambda/lib/serve.js:133:27
    at Layer.handle [as handle_request] (/Users/jimnielsen/Sites/jim-nielsen.com/api/node_modules/express/lib/router/layer.js:95:5)
    at next (/Users/jimnielsen/Sites/jim-nielsen.com/api/node_modules/express/lib/router/route.js:137:13)
    at next (/Users/jimnielsen/Sites/jim-nielsen.com/api/node_modules/express/lib/router/route.js:131:14)
    at next (/Users/jimnielsen/Sites/jim-nielsen.com/api/node_modules/express/lib/router/route.js:131:14)
    at next (/Users/jimnielsen/Sites/jim-nielsen.com/api/node_modules/express/lib/router/route.js:131:14)
    at next (/Users/jimnielsen/Sites/jim-nielsen.com/api/node_modules/express/lib/router/route.js:131:14)
    at next (/Users/jimnielsen/Sites/jim-nielsen.com/api/node_modules/express/lib/router/route.js:131:14)
    at next (/Users/jimnielsen/Sites/jim-nielsen.com/api/node_modules/express/lib/router/route.js:131:14)
    at Route.dispatch (/Users/jimnielsen/Sites/jim-nielsen.com/api/node_modules/express/lib/router/route.js:112:3)
    at Layer.handle [as handle_request] (/Users/jimnielsen/Sites/jim-nielsen.com/api/node_modules/express/lib/router/layer.js:95:5)
    at /Users/jimnielsen/Sites/jim-nielsen.com/api/node_modules/express/lib/router/index.js:281:22
    at param (/Users/jimnielsen/Sites/jim-nielsen.com/api/node_modules/express/lib/router/index.js:354:14)
    at param (/Users/jimnielsen/Sites/jim-nielsen.com/api/node_modules/express/lib/router/index.js:365:14)
    at Function.process_params (/Users/jimnielsen/Sites/jim-nielsen.com/api/node_modules/express/lib/router/index.js:410:3)

In fact, if I change that my hello.js file while the server is running locally, netlify-lambda seems to choke on something and just dies:

Hash: 80255c95f9b2a80ce027
Version: webpack 4.30.0
Time: 93ms
Built at: 04/17/2019 8:44:35 AM
              Asset      Size  Chunks             Chunk Names
           hello.js  18.1 KiB       0  [emitted]  hello
icg-slack-deploy.js  19.5 KiB       1  [emitted]  icg-slack-deploy
Entrypoint hello = hello.js
Entrypoint icg-slack-deploy = icg-slack-deploy.js
 [0] external "stream" 42 bytes {0} {1} [built]
 [1] external "zlib" 42 bytes {0} {1} [built]
 [2] external "url" 42 bytes {0} {1} [built]
 [3] external "http" 42 bytes {0} {1} [built]
 [4] external "https" 42 bytes {0} {1} [built]
 [5] ../node_modules/node-fetch/lib/index.mjs 39.6 KiB {0} {1} [built]
 [6] ./hello.js 349 bytes {0} [built]
 [7] ./icg-slack-deploy.js 2.19 KiB {1} [built]
 [8] ../node_modules/dotenv/lib/main.js 2.87 KiB {1} [built]
 [9] external "fs" 42 bytes {1} [built]
[10] external "path" 42 bytes {1} [built]
events.js:174
      throw er; // Unhandled 'error' event
      ^

Error: listen EADDRINUSE: address already in use :::9000
    at Server.setupListenHandle [as _listen2] (net.js:1270:14)
    at listenInCluster (net.js:1318:12)
    at Server.listen (net.js:1405:7)
    at Function.listen (/Users/jimnielsen/Sites/jim-nielsen.com/api/node_modules/express/lib/application.js:618:24)
    at Object.exports.listen (/Users/jimnielsen/Sites/jim-nielsen.com/api/node_modules/netlify-lambda/lib/serve.js:179:7)
    at Watching.handler (/Users/jimnielsen/Sites/jim-nielsen.com/api/node_modules/netlify-lambda/bin/cmd.js:38:26)
    at compiler.hooks.done.callAsync (/Users/jimnielsen/Sites/jim-nielsen.com/api/node_modules/webpack/lib/Watching.js:98:9)
    at AsyncSeriesHook.eval [as callAsync] (eval at create (/Users/jimnielsen/Sites/jim-nielsen.com/api/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:6:1)
    at Watching._done (/Users/jimnielsen/Sites/jim-nielsen.com/api/node_modules/webpack/lib/Watching.js:97:28)
    at compiler.emitRecords.err (/Users/jimnielsen/Sites/jim-nielsen.com/api/node_modules/webpack/lib/Watching.js:72:19)
    at Compiler.emitRecords (/Users/jimnielsen/Sites/jim-nielsen.com/api/node_modules/webpack/lib/Compiler.js:449:39)
    at compiler.emitAssets.err (/Users/jimnielsen/Sites/jim-nielsen.com/api/node_modules/webpack/lib/Watching.js:53:20)
    at hooks.afterEmit.callAsync.err (/Users/jimnielsen/Sites/jim-nielsen.com/api/node_modules/webpack/lib/Compiler.js:435:14)
    at AsyncSeriesHook.eval [as callAsync] (eval at create (/Users/jimnielsen/Sites/jim-nielsen.com/api/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:6:1)
    at asyncLib.forEachLimit.err (/Users/jimnielsen/Sites/jim-nielsen.com/api/node_modules/webpack/lib/Compiler.js:432:27)
    at /Users/jimnielsen/Sites/jim-nielsen.com/api/node_modules/neo-async/async.js:2813:7
Emitted 'error' event at:
    at emitErrorNT (net.js:1297:8)
    at process._tickCallback (internal/process/next_tick.js:63:19)
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! jimniels-api@1.0.0 start: `netlify-lambda serve src`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the jimniels-api@1.0.0 start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/jimnielsen/.npm/_logs/2019-04-17T14_44_35_788Z-debug.log

netlify dev seems to have a similar issue. If I run netlify dev and then hit http://localhost:8888/.netlify/functions/hello, it gives me a similar o is not a function (but netlify dev doesn't seem to choke when the server is running locally and I change the file, the server stays up and running with my updated file changes, but it still chokes on the fetch call)

Connected!

   ┌─────────────────────────────────────────────────┐
   │                                                 │
   │   ◈ Server now ready on http://localhost:8888   │
   │                                                 │
   └─────────────────────────────────────────────────┘

Request from ::ffff:127.0.0.1: GET /.netlify/functions/hello
Response with status 200 in 8 ms.
TypeError: o is not a function
    at Object.t.handler (/Users/jimnielsen/Sites/jim-nielsen.com/api/build/hello.js:1:18480)
    at /usr/local/lib/node_modules/netlify-cli/node_modules/netlify-dev-plugin/src/utils/serveFunctions.js:158:29
    at Layer.handle [as handle_request] (/usr/local/lib/node_modules/netlify-cli/node_modules/express/lib/router/layer.js:95:5)
    at next (/usr/local/lib/node_modules/netlify-cli/node_modules/express/lib/router/route.js:137:13)
    at next (/usr/local/lib/node_modules/netlify-cli/node_modules/express/lib/router/route.js:131:14)
    at next (/usr/local/lib/node_modules/netlify-cli/node_modules/express/lib/router/route.js:131:14)
    at next (/usr/local/lib/node_modules/netlify-cli/node_modules/express/lib/router/route.js:131:14)
    at next (/usr/local/lib/node_modules/netlify-cli/node_modules/express/lib/router/route.js:131:14)
    at next (/usr/local/lib/node_modules/netlify-cli/node_modules/express/lib/router/route.js:131:14)
    at next (/usr/local/lib/node_modules/netlify-cli/node_modules/express/lib/router/route.js:131:14)

[sergio-toro]

In fact, if I change that my hello.js file while the server is running locally, netlify-lambda seems to choke on something and just dies:

Hash: 80255c95f9b2a80ce027
Version: webpack 4.30.0
Time: 93ms
Built at: 04/17/2019 8:44:35 AM
              Asset      Size  Chunks             Chunk Names
           hello.js  18.1 KiB       0  [emitted]  hello
icg-slack-deploy.js  19.5 KiB       1  [emitted]  icg-slack-deploy
Entrypoint hello = hello.js
Entrypoint icg-slack-deploy = icg-slack-deploy.js
 [0] external "stream" 42 bytes {0} {1} [built]
 [1] external "zlib" 42 bytes {0} {1} [built]
 [2] external "url" 42 bytes {0} {1} [built]
 [3] external "http" 42 bytes {0} {1} [built]
 [4] external "https" 42 bytes {0} {1} [built]
 [5] ../node_modules/node-fetch/lib/index.mjs 39.6 KiB {0} {1} [built]
 [6] ./hello.js 349 bytes {0} [built]
 [7] ./icg-slack-deploy.js 2.19 KiB {1} [built]
 [8] ../node_modules/dotenv/lib/main.js 2.87 KiB {1} [built]
 [9] external "fs" 42 bytes {1} [built]
[10] external "path" 42 bytes {1} [built]
events.js:174
      throw er; // Unhandled 'error' event
      ^

Error: listen EADDRINUSE: address already in use :::9000
    at Server.setupListenHandle [as _listen2] (net.js:1270:14)
    at listenInCluster (net.js:1318:12)
    at Server.listen (net.js:1405:7)
    at Function.listen (/Users/jimnielsen/Sites/jim-nielsen.com/api/node_modules/express/lib/application.js:618:24)
    at Object.exports.listen (/Users/jimnielsen/Sites/jim-nielsen.com/api/node_modules/netlify-lambda/lib/serve.js:179:7)
    at Watching.handler (/Users/jimnielsen/Sites/jim-nielsen.com/api/node_modules/netlify-lambda/bin/cmd.js:38:26)
    at compiler.hooks.done.callAsync (/Users/jimnielsen/Sites/jim-nielsen.com/api/node_modules/webpack/lib/Watching.js:98:9)
    at AsyncSeriesHook.eval [as callAsync] (eval at create (/Users/jimnielsen/Sites/jim-nielsen.com/api/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:6:1)
    at Watching._done (/Users/jimnielsen/Sites/jim-nielsen.com/api/node_modules/webpack/lib/Watching.js:97:28)
    at compiler.emitRecords.err (/Users/jimnielsen/Sites/jim-nielsen.com/api/node_modules/webpack/lib/Watching.js:72:19)
    at Compiler.emitRecords (/Users/jimnielsen/Sites/jim-nielsen.com/api/node_modules/webpack/lib/Compiler.js:449:39)
    at compiler.emitAssets.err (/Users/jimnielsen/Sites/jim-nielsen.com/api/node_modules/webpack/lib/Watching.js:53:20)
    at hooks.afterEmit.callAsync.err (/Users/jimnielsen/Sites/jim-nielsen.com/api/node_modules/webpack/lib/Compiler.js:435:14)
    at AsyncSeriesHook.eval [as callAsync] (eval at create (/Users/jimnielsen/Sites/jim-nielsen.com/api/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:6:1)
    at asyncLib.forEachLimit.err (/Users/jimnielsen/Sites/jim-nielsen.com/api/node_modules/webpack/lib/Compiler.js:432:27)
    at /Users/jimnielsen/Sites/jim-nielsen.com/api/node_modules/neo-async/async.js:2813:7
Emitted 'error' event at:
    at emitErrorNT (net.js:1297:8)
    at process._tickCallback (internal/process/next_tick.js:63:19)
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! jimniels-api@1.0.0 start: `netlify-lambda serve src`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the jimniels-api@1.0.0 start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/jimnielsen/.npm/_logs/2019-04-17T14_44_35_788Z-debug.log

I'm experiencing the same EADDRINUSE error. Where you able to fix it?

[jimniels]

No I haven’t been able to get this working. I basically gave up. Figured I’d try again later, or when somebody has some insight into the issue.

[suits-at]

We recently added support for unbundled function deployment to the buildbot and CLI (via https://github.com/netlify/zip-it-and-ship-it if you are curious). That means, you can deploy js functions with dependencies without bundling them first with a tool like netlify-lambda. Your deployed functions folder can contain function files that require modules in a node_modules in our outside the deployed functions folder. On deployment, we walk the dependency tree and grab all the dependencies the functions need. In general, it should work for bundled and unbundled functions. For some reason, its finding node-fetch requires an encoding module that appears to be consumed as a peer dependency (e.g. node-fetch requires it, but doesn't ship a reference in its package.json).

The solution now is to install encoding in the nearest package.json. We need to look into if this is an issue with node-fetch that needs correcting, or if our dependency algorithm is just missing stuff, but there is an action you can take now.

I just had the same problem after using node-fetch inside a function, but for me installing encoding solved the problem!

[swyxio]

thanks everyone for the feedback and sorry about these early troubles. i’m going to recommend people use axios for the time being. but also would love to fix the underlying problem

[bcomnes]

(This API requires an optional dependency on npm package encoding, which you need to install manually. webpack users may see a warning message due to this optional dependency.) -- https://github.com/bitinn/node-fetch/blob/master/package.json

Just to get the info out there, node-fetch uses an undeclared peer dependency in some cases. Our bundling tree shaker sometimes fails the build when it can't find this dependency. Failing the build due to missing dependencies is generally the desired behavior (vs a successful deploy that will fail during runtime due to missing deps), however in some cases, when modules do weird things, it's sometimes not what you want. We are currently investigating available options to handle this edge case in a cleaner manner. In the meantime, just install the encoding peer dependency. Track here: netlify/zip-it-and-ship-it#30