Allow to run node_modules binaries from makefile builder

[HaaLeo]

Description:

When I try to execute scripts from locally npm packages the makefile builder fails

Steps to reproduce:

1. create a template containing a simple lambda with a makefile builder:

   MyLambda:
     Type: AWS::Serverless::Function
     Properties:
       CodeUri: ./
       Handler: index.lambdaHandler
       Runtime: nodejs12.x
       MemorySize: 128
       Timeout: 60
       Description: my lambda.
       Policies:
         - AWSLambdaBasicExecutionRole
     Metadata:
       BuildMethod: makefile 

2. A package.json with dependencies, for example typescript:

   {
       "scripts": {
           "build": "tsc -p tsconfig.json"
       },
       "devDependencies": {
           "typescript": "^4.1.3"
       }
   }

3. A Makefile containing:

   build-MyLambda:
       npm run build #or node node_modules/.bin/tsc -p tsconfig.json
   .PHONY: build
   build:
       npm run build

4. A tsconfig.json file:

   {
       "compilerOptions": {
           "module": "commonjs",
           "target": "es2019",
           "noImplicitAny": true,
           "preserveConstEnums": true,
           "outDir": "./dist",
           "sourceMap": true,
           "moduleResolution": "node"
       },
       "include": [
           "src/**/*"
       ],
       "exclude": [
           "node_modules",
           "**/*.spec.ts"
       ]
   }

Run make build to confirm the setup is working.
Now run sam build --debug to reproduce the error

Observed result:

sam build --debug errors with the following message:

2021-01-29 08:56:16,376 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2021-01-29 08:56:16,479 | 'build' command is called
2021-01-29 08:56:16,485 | Collected default values for parameters: {}
2021-01-29 08:56:16,507 | 1 resources found in the template
2021-01-29 08:56:16,507 | Found Serverless function with name='MyLambda' and CodeUri='./'
2021-01-29 08:56:16,507 | Collected default values for parameters: {}
2021-01-29 08:56:16,528 | Instantiating build definitions
2021-01-29 08:56:16,530 | Unique function build definition found, adding as new (Function Build Definition: BuildDefinition(nodejs12.x, ../, Zip, , ae285d5a-7ad0-4790-b62a-82c2d9b32503, {'BuildMethod': 'makefile'}, []), Function: Function(name='InvalidateCloudFrontCache', functionname='InvalidateCloudFrontCache', runtime='nodejs12.x', memory=128, timeout=60, handler='index.lambdaHandler', imageuri=None, packagetype='Zip', imageconfig=None, codeuri='../', environment=None, rolearn=None, layers=[], events=None, metadata={'BuildMethod': 'makefile'}, codesign_config_arn=None))
2021-01-29 08:56:16,531 | Building codeuri: ../ runtime: nodejs12.x metadata: {'BuildMethod': 'makefile'} functions: ['InvalidateCloudFrontCache']
2021-01-29 08:56:16,596 | Building to following folder /Somewhere/my-lambda/.aws-sam/build/InvalidateCloudFrontCache
2021-01-29 08:56:16,597 | Loading workflow module 'aws_lambda_builders.workflows'
2021-01-29 08:56:16,600 | Registering workflow 'PythonPipBuilder' with capability 'Capability(language='python', dependency_manager='pip', application_framework=None)'
2021-01-29 08:56:16,602 | Registering workflow 'NodejsNpmBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm', application_framework=None)'
2021-01-29 08:56:16,604 | Registering workflow 'RubyBundlerBuilder' with capability 'Capability(language='ruby', dependency_manager='bundler', application_framework=None)'
2021-01-29 08:56:16,605 | Registering workflow 'GoDepBuilder' with capability 'Capability(language='go', dependency_manager='dep', application_framework=None)'
2021-01-29 08:56:16,607 | Registering workflow 'GoModulesBuilder' with capability 'Capability(language='go', dependency_manager='modules', application_framework=None)'
2021-01-29 08:56:16,609 | Registering workflow 'JavaGradleWorkflow' with capability 'Capability(language='java', dependency_manager='gradle', application_framework=None)'
2021-01-29 08:56:16,622 | Registering workflow 'JavaMavenWorkflow' with capability 'Capability(language='java', dependency_manager='maven', application_framework=None)'
2021-01-29 08:56:16,624 | Registering workflow 'DotnetCliPackageBuilder' with capability 'Capability(language='dotnet', dependency_manager='cli-package', application_framework=None)'
2021-01-29 08:56:16,718 | Registering workflow 'CustomMakeBuilder' with capability 'Capability(language='provided', dependency_manager=None, application_framework=None)'
2021-01-29 08:56:16,718 | Found workflow 'CustomMakeBuilder' to support capabilities 'Capability(language='provided', dependency_manager=None, application_framework=None)'
2021-01-29 08:56:16,719 | Running workflow 'CustomMakeBuilder'
2021-01-29 08:56:16,719 | Running CustomMakeBuilder:CopySource
2021-01-29 08:56:20,847 | CustomMakeBuilder:CopySource succeeded
2021-01-29 08:56:20,847 | Running CustomMakeBuilder:MakeBuild
2021-01-29 08:56:20,847 | Current Artifacts Directory : /Somewhere/my-lambda/.aws-sam/build/InvalidateCloudFrontCache
2021-01-29 08:56:20,847 | executing Make: ['make', '--makefile', '/Somewhere/my-lambda/Makefile', 'build-InvalidateCloudFrontCache']
2021-01-29 08:56:21,400 | CustomMakeBuilder:MakeBuild failed
Traceback (most recent call last):
  File "/usr/local/Cellar/aws-sam-cli/1.15.0/libexec/lib/python3.8/site-packages/aws_lambda_builders/workflows/custom_make/actions.py", line 87, in execute
    self.subprocess_make.run(
  File "/usr/local/Cellar/aws-sam-cli/1.15.0/libexec/lib/python3.8/site-packages/aws_lambda_builders/workflows/custom_make/make.py", line 88, in run
    raise MakeExecutionError(message=err.decode("utf8").strip())
aws_lambda_builders.workflows.custom_make.make.MakeExecutionError: Make Failed: internal/modules/cjs/loader.js:818
  throw err;
  ^

Error: Cannot find module '../lib/tsc.js'
Require stack:
- /private/var/folders/_h/33xwt_j11sjd5p50drcl2kfr0000gn/T/tmp_degkr1o/node_modules/.bin/tsc
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:815:15)
    at Function.Module._load (internal/modules/cjs/loader.js:667:27)
    at Module.require (internal/modules/cjs/loader.js:887:19)
    at require (internal/modules/cjs/helpers.js:74:18)
    at Object.<anonymous> (/private/var/folders/_h/33xwt_j11sjd5p50drcl2kfr0000gn/T/tmp_degkr1o/node_modules/.bin/tsc:2:1)
    at Module._compile (internal/modules/cjs/loader.js:999:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)
    at Module.load (internal/modules/cjs/loader.js:863:32)
    at Function.Module._load (internal/modules/cjs/loader.js:708:14)
    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:60:12) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/private/var/folders/_h/33xwt_j11sjd5p50drcl2kfr0000gn/T/tmp_degkr1o/node_modules/.bin/tsc'
  ]
}
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! invalidate-cf-cache@0.0.1 build: `tsc -p tsconfig.json`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the invalidate-cf-cache@0.0.1 build script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/z0049e1t/.npm/_logs/2021-01-29T07_56_21_371Z-debug.log
make: *** [build-InvalidateCloudFrontCache] Error 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/Cellar/aws-sam-cli/1.15.0/libexec/lib/python3.8/site-packages/aws_lambda_builders/workflow.py", line 269, in run
    action.execute()
  File "/usr/local/Cellar/aws-sam-cli/1.15.0/libexec/lib/python3.8/site-packages/aws_lambda_builders/workflows/custom_make/actions.py", line 97, in execute
    raise ActionFailedError(str(ex))
aws_lambda_builders.actions.ActionFailedError: Make Failed: internal/modules/cjs/loader.js:818
  throw err;
  ^

Error: Cannot find module '../lib/tsc.js'
Require stack:
- /private/var/folders/_h/33xwt_j11sjd5p50drcl2kfr0000gn/T/tmp_degkr1o/node_modules/.bin/tsc
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:815:15)
    at Function.Module._load (internal/modules/cjs/loader.js:667:27)
    at Module.require (internal/modules/cjs/loader.js:887:19)
    at require (internal/modules/cjs/helpers.js:74:18)
    at Object.<anonymous> (/private/var/folders/_h/33xwt_j11sjd5p50drcl2kfr0000gn/T/tmp_degkr1o/node_modules/.bin/tsc:2:1)
    at Module._compile (internal/modules/cjs/loader.js:999:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)
    at Module.load (internal/modules/cjs/loader.js:863:32)
    at Function.Module._load (internal/modules/cjs/loader.js:708:14)
    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:60:12) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/private/var/folders/_h/33xwt_j11sjd5p50drcl2kfr0000gn/T/tmp_degkr1o/node_modules/.bin/tsc'
  ]
}
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! invalidate-cf-cache@0.0.1 build: `tsc -p tsconfig.json`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the invalidate-cf-cache@0.0.1 build script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/z0049e1t/.npm/_logs/2021-01-29T07_56_21_371Z-debug.log
make: *** [build-InvalidateCloudFrontCache] Error 1

Build Failed
2021-01-29 08:56:22,223 | Sending Telemetry: {'metrics': [{'commandRun': {'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam build', 'duration': 5846, 'exitReason': 'WorkflowFailedError', 'exitCode': 1, 'requestId': 'ed59d879-ed40-4e0e-ad2d-2a0f678c8597', 'installationId': '17adb356-4bd3-44f0-8b2c-2e783726a1f8', 'sessionId': '62216092-cb80-44e7-8292-83b33897c81f', 'executionEnvironment': 'CLI', 'pyversion': '3.8.7', 'samcliVersion': '1.15.0'}}]}
2021-01-29 08:56:22,927 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)
Error: CustomMakeBuilder:MakeBuild - Make Failed: internal/modules/cjs/loader.js:818
  throw err;
  ^

Error: Cannot find module '../lib/tsc.js'
Require stack:
- /private/var/folders/_h/33xwt_j11sjd5p50drcl2kfr0000gn/T/tmp_degkr1o/node_modules/.bin/tsc
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:815:15)
    at Function.Module._load (internal/modules/cjs/loader.js:667:27)
    at Module.require (internal/modules/cjs/loader.js:887:19)
    at require (internal/modules/cjs/helpers.js:74:18)
    at Object.<anonymous> (/private/var/folders/_h/33xwt_j11sjd5p50drcl2kfr0000gn/T/tmp_degkr1o/node_modules/.bin/tsc:2:1)
    at Module._compile (internal/modules/cjs/loader.js:999:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)
    at Module.load (internal/modules/cjs/loader.js:863:32)
    at Function.Module._load (internal/modules/cjs/loader.js:708:14)
    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:60:12) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/private/var/folders/_h/33xwt_j11sjd5p50drcl2kfr0000gn/T/tmp_degkr1o/node_modules/.bin/tsc'
  ]
}
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! invalidate-cf-cache@0.0.1 build: `tsc -p tsconfig.json`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the invalidate-cf-cache@0.0.1 build script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/z0049e1t/.npm/_logs/2021-01-29T07_56_21_371Z-debug.log
make: *** [build-InvalidateCloudFrontCache] Error 1

Expected result:

The build does not error.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

Interestingly the build does not error if you install typescript globally (npm i -g typescript) and call it in the makefile directly.
However, I want all the dependencies of my project installed locally, not globally to build my app.

1. OS: macOS Catalina 10.15.7
2. sam --version: 1.15.0

[mgrandis]

Hi @HaaLeo,
Thanks for reporting this.
We have an ongoing PR to package local NPM dependencies: aws/aws-lambda-builders#215
Would this help?

[HaaLeo]

The PR you mentioned seems to be at least related. I also found aws/aws-lambda-builders#106 which seems to be closer related to this issue.
However, after a brief look at those PRs I am unsure whether those consider the makefile builder as well.

[mgrandis]

Could reproduce using your example (devdependencies needs to be renamed to devDependencies)
However, the make build only worked after I ran npm install.

I tried adding npm install to the Makefile before the npm build but the error persisted.

Based on the last comment here jupyter/nbdime#201 I also tried adding rm -rf node_modules before npm install, this time the execution took longer and ended on a different error.

[hung3102]

It takes much of time on npm install when removing node_modules in each function build.
Remove node_modules/.bin and reinstall typescript works for me:

build-lambda-common:
        npm install
        rm -rf node_modules/.bin
        npm i typescript

[MattLish]

For anyone else that comes across this, it seems that with an npm install the node_modules/.bin seems to get broken.
If you replace npm install in your Makefile with npm ci, it seems to solve the problem (although unfortunately, a little slower).

e.g.

build-MyLambda:
    npm ci
    npm run build

[heitormon]

use the absolute path in node_modules/.bin