sam build does not recognise local nodejs projects outside CodeUri

[rfrad]

Description

I have 2 Node.js projects: CommonT and Profile. Profile has a local dependency on CommonT.

When running the sam build -t template.yaml --debug command, it fails with the following error:

npm ERR! code ENOLOCAL
npm ERR! Could not install from "../CommonT" as it does not contain a package.json file.

Here is the project structure:

├── Backend
│   ├── CommonT
│   │   ├── package.json
│   │   └── src
│   └── Profile
│       ├── package.json
│       └── src
└── template.yaml

Here is the Profile/package.json:

{
  "main": "dist/src/index.js",
  ...
  "scripts": {
    "test": "ts-node ./node_modules/jasmine/bin/jasmine",
    "build": "tsc",
    "deploy": "npm run build && sls deploy"
  },
  "dependencies": {
    "@myteam/common": "file:../CommonT",
	...
  },
  "devDependencies": {
    "@types/aws-lambda": "^8.10.46",
    "@types/node": "^12.12.32",
    "aws-sdk": "^2.508.0",
    "ts-node": "^8.10.2",
    "typescript": "^3.8.3",
	...
  }
}

Here is the CommonT/package.json:

{
  "name": "@myteam/common",
  ...
  "main": "dist/src/index",
  "typings": "dist/src/index",
  "private": true,
  "scripts": {
    "test": "ts-node ./node_modules/jasmine/bin/jasmine",
    "build": "tsc",
    "deploy": "npm run build && sls deploy"
  },
  "dependencies": {
    ...
  },
  "devDependencies": {
    "@types/aws-lambda": "^8.10.46",
    "@types/node": "^12.12.32",
    "aws-sdk": "^2.508.0",
    "ts-node": "^8.10.2",
    "typescript": "^3.8.3",
    ...
  }
}

Here is my template.yaml:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  My APIs
 
Resources:
  Profile:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: Backend/Profile
      Handler: dist/src/index.handler
      Runtime: nodejs12.x
      Events:
        GetOneUser:
          Type: Api
          Properties:
            ...
  ...

This is an error I was facing with sam --version: 0.11.0 that disapeared when upgrading to sam --version: 0.52.0 using homebrew. I am currently moving CICD tool and I installed the same sam --version: 0.52.0 using pip3 as I cannot install homebrew on my CICD platform. So here it is:

• npm build run manually: works fine
• sam build (v 0.11.0) - Homebrew install: ERR! code ENOLOCAL
• sam build (v 0.52.0) - Homebrew install: works fine
• sam build (v 0.52.0) - pip3 install: ERR! code ENOLOCAL

Steps to reproduce

cd __project_root__

npm install --prefix Backend/CommonT
npm run build --prefix Backend/CommonT
npm run test --prefix Backend/CommonT

npm install --prefix Backend/Profile
npm run build --prefix Backend/Profile
npm run test --prefix Backend/Profile

# I tried both with and without this line, it doesn't make a difference
# npm install -q --no-audit --no-save --production --unsafe-perm --prefix %ROOT%/CommonT

# This is the line that fails
sam build -t template.yaml --debug 

Observed result

Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
'build' command is called
Collected default values for parameters: {}
8 resources found in the template
Found Serverless function with name='Profile' and CodeUri='Backend/Profile'
Found Serverless function with name='###Something else###' and CodeUri='###'
Collected default values for parameters: {}
Building function 'Profile'
Loading workflow module 'aws_lambda_builders.workflows'
Registering workflow 'PythonPipBuilder' with capability 'Capability(language='python', dependency_manager='pip', application_framework=None)'
Registering workflow 'NodejsNpmBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm', application_framework=None)'
Registering workflow 'RubyBundlerBuilder' with capability 'Capability(language='ruby', dependency_manager='bundler', application_framework=None)'
Registering workflow 'GoDepBuilder' with capability 'Capability(language='go', dependency_manager='dep', application_framework=None)'
Registering workflow 'GoModulesBuilder' with capability 'Capability(language='go', dependency_manager='modules', application_framework=None)'
Registering workflow 'JavaGradleWorkflow' with capability 'Capability(language='java', dependency_manager='gradle', application_framework=None)'
Registering workflow 'JavaMavenWorkflow' with capability 'Capability(language='java', dependency_manager='maven', application_framework=None)'
Registering workflow 'DotnetCliPackageBuilder' with capability 'Capability(language='dotnet', dependency_manager='cli-package', application_framework=None)'
Registering workflow 'CustomMakeBuilder' with capability 'Capability(language='provided', dependency_manager=None, application_framework=None)'
Found workflow 'NodejsNpmBuilder' to support capabilities 'Capability(language='nodejs', dependency_manager='npm', application_framework=None)'
Running workflow 'NodejsNpmBuilder'
Running NodejsNpmBuilder:NpmPack
NODEJS packaging file:/my/directory/job/Backend/Profile to /my/directory/temp/buildTmp/XXXXX
executing NPM: ['npm', 'pack', '-q', 'file:/my/directory/job/Backend/Profile']
NODEJS packed to canaree-profile-1.0.0.tgz
NODEJS extracting to /my/directory/temp/buildTmp/XXXXX/unpacked
NodejsNpmBuilder:NpmPack succeeded
Running NodejsNpmBuilder:CopyNpmrc
.npmrc copying in: /my/directory/temp/buildTmp/XXXXX/unpacked/package
NodejsNpmBuilder:CopyNpmrc succeeded
Running NodejsNpmBuilder:CopySource
NodejsNpmBuilder:CopySource succeeded
Running NodejsNpmBuilder:NpmInstall
NODEJS installing in: /my/directory/job/.aws-sam/build/Profile
executing NPM: ['npm', 'install', '-q', '--no-audit', '--no-save', '--production', '--unsafe-perm']
NodejsNpmBuilder:NpmInstall failed
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/aws_lambda_builders/workflows/nodejs_npm/actions.py", line 108, in execute
    ["install", "-q", "--no-audit", "--no-save", "--production", "--unsafe-perm"], cwd=self.artifacts_dir
  File "/usr/local/lib/python3.6/dist-packages/aws_lambda_builders/workflows/nodejs_npm/npm.py", line 85, in run
    raise NpmExecutionError(message=err.decode("utf8").strip())
aws_lambda_builders.workflows.nodejs_npm.npm.NpmExecutionError: NPM Failed: npm WARN deprecated urix@0.1.0: Please see https://github.com/lydell/urix#deprecated
npm WARN deprecated resolve-url@0.2.1: https://github.com/lydell/resolve-url#deprecated
npm ERR! code ENOLOCAL
npm ERR! Could not install from "../CommonT" as it does not contain a package.json file.

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2020-06-22T12_03_11_067Z-debug.log

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/aws_lambda_builders/workflow.py", line 269, in run
    action.execute()
  File "/usr/local/lib/python3.6/dist-packages/aws_lambda_builders/workflows/nodejs_npm/actions.py", line 112, in execute
    raise ActionFailedError(str(ex))
aws_lambda_builders.actions.ActionFailedError: NPM Failed: npm WARN deprecated urix@0.1.0: Please see https://github.com/lydell/urix#deprecated
npm WARN deprecated resolve-url@0.2.1: https://github.com/lydell/resolve-url#deprecated
npm ERR! code ENOLOCAL
npm ERR! Could not install from "../CommonT" as it does not contain a package.json file.

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/XXXXXXXX-debug.log

Build Failed

npm ERR!     /root/.npm/_logs/XXXXXXXX-debug.log
Sending Telemetry: {'metrics': [{'commandRun': {'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam build', 'duration': 2034, 'exitReason': 'WorkflowFailedError', 'exitCode': 1, 'requestId': '42b13729-2136-4df8-8442-a25c0ae2ce19', 'installationId': 'acac1f1a-some-long-id-b854a9c82c0b', 'sessionId': 'fb6bba5f-some-other-long-id-ed535022cafb', 'executionEnvironment': 'CLI', 'pyversion': '3.6.9', 'samcliVersion': '0.52.0'}}]}
HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)
Error: NodejsNpmBuilder:NpmInstall - NPM Failed: npm WARN deprecated resolve-url@0.2.1: https://github.com/lydell/resolve-url#deprecated
npm WARN deprecated urix@0.1.0: Please see https://github.com/lydell/urix#deprecated
npm ERR! code ENOLOCAL
npm ERR! Could not install from "../CommonT" as it does not contain a package.json file.

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/XXXXXXXX-debug.log

Expected result

When manually running the builds/tests with npm, everything compiles and run properly. I expect the same behavour to happen with sam build.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: Ubuntu 18.04.4 LTS
2. sam --version: 0.52.0

[rfrad]

@sriram-mv , sorry, I'm a bit new to github. Could you please explain what the added "type/feature" means? Does that mean this bug is being looked at?
Thanks a lot

[rfrad]

Here is a work around:

I did manage to "force" the creation of the CommonT package by adding a resource with the CommonT name in my template.yaml.

Here is my new file:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  My APIs
 
Resources:
  CommonT: # <= This NEEDS to have the same name as the CommonT Typescript project!
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: Backend/CommonT
      Handler: dist/src/index.handler
      Runtime: nodejs12.x
      Events:
        GetCommonData:
          Type: Api
          Properties:
            RestApiId: !Ref RestAPI
            Path: /some/config/data/for/example
            Method: get

  Profile:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: Backend/Profile
      Handler: dist/src/index.handler
      Runtime: nodejs12.x
      Events:
        GetOneUser:
          Type: Api
          Properties:
            ...
  ...

Then, when sam build creates the .aws-sam package, it starts by creating the folder CommonT for the resource with the same name. And it is then used as the common project as well.

It is dirty... but it allows to deploy for now.

[EladHeller]

This work around work only when the name of the directory is alphanumeric.

[dsandor]

Here is my work around. It is not pretty but it works.

First, use npm link or yarn link on your local node modules.

Next, set the dependencies section in your lambda's package.json to a path that is relative to the Lambda Function Code Folder under .aws-sam.

My folders look like this:

   |--.aws-sam
   |----build
   |------zipcodeFunction
   |--------node_modules
   |--local-modules
   |----data-layer
   |--src
   |----zipcodes

So I start by going into the local-modules/data-layer directory and running yarn link.
Next, go into the src/zipcodes (my Lambda function) directory and run yarn link data-layer.

Now a sam build will work and all your development tooling works also because the yarn link in the lambda directory adds the file in node modules. When you do a sam build it will bundle the local node module for you as you intend.

[bingtimren]

Here's my work around. Simply put - copy the dependency project into the current root.

My project has three parts - frontend, backend (SAM) and "proj-common", common artifacts for both front and backend.

I first put the dependency in package.json "dependencies" as "file:../proj-commons", pointing to the local folder. This works for the IDE, but sam build fails. The reason is SAM build in a container. In that container the mapped dir cannot access proj-commons. Check .aws-sam/build dir for the files actually available in SAM build.

I tried to put "../proj-commons" in "files" of "package.json" and it does not work. I tried to symbol link "../proj-commons" to a folder under the root, and refer to the folder in both "dependencies" and "files", it does not work.

So the final work around is - in my build script, first copy "../proj-commons" into a directory under the SAM project root, and also put it inside "files" in "project.json".

Here's how my package.json looks like. Note the build script.

    "dependencies": {
        "aws-sdk": "^2.437.0",
        "proj-commons": "file:./.dependency_by_copy/proj-commons"
    },
    "files": [
        "built/",
        ".dependency_by_copy/"
    ],
....
    "scripts": {
        "build": ". ./env.sh; rm -fr ./.dependency_by_copy; mkdir .dependency_by_copy; cp -r ../proj-commons ./.dependency_by_copy; tsc; sam build; announce \"Project Built\"",

[fbn-roussel]

I am trying to add some shared code to a layer but I am getting this error as well.

The yarn link did not worked for me. But @bingtimren workaround did, thank you !

Before :

{
  ...
  "scripts": {
    "build": "yarn install"
  },
  "dependencies": {
    "shared": "file:./../shared"
  }
}

After :

{
  ...
  "scripts": {
    "build": "mkdir -p dist && rsync -avr ../../shared dist --exclude node_modules --exclude src && yarn install"
  },
  "dependencies": {
    "shared": "file:./dist/shared"
  },
  "files": [
    "dist"
  ]
}

You can find the full example here : https://github.com/fbn-roussel/aws-sam-typescript-webpack-backend/blob/main/layers/application-layer/package.json

Native support of local path dependency would be greatly appreciated :)

[mgrandis]

We are currently reviewing a PR that should address this issue:
aws/aws-lambda-builders#215

[kellerkind87]

hi,
are there any news reagrding that topic? we are highly interested in that beeing solved.

[eladg]

@cBiscuitSurprise thank you for the hard work building this critical functionality! 🙏

@mgrandis I and probably the entire AWS SAM community are wondering if there is any progress on this PR since Feb 26? are there any better workarounds we should be aware of? Is there any way we can help test this so it will move forward?

Thanks!

[bingtimren]

If you are willing to try CDK, NodejsFunction with esbuild works for me pretty well.

See

https://docs.aws.amazon.com/cdk/api/latest/docs/aws-lambda-nodejs-readme.html#local-bundling

https://docs.aws.amazon.com/cdk/api/latest/docs/aws-lambda-nodejs-readme.html#configuring-esbuild