TypeScript+esbuild and sam build with --manifest don't recognize the package.json

[oieduardorabelo]

Description:

After reading through #4427 , I noticed the suggested solution #4427 (comment) doesn't work for TypeScript projects (it does work for plain Node.js).

Steps to reproduce:

1. sam init filled out options to get a typescript nodejs hello world project
2. cd sam-app
3. mv hello-world/package.json .
4. sam build --manifest ./package.json

sam-build-manifest-typescript.mov

Observed result:

2023-03-08 12:07:12,668 | Config file location: /oooo/sam-app/samconfig.toml
2023-03-08 12:07:12,668 | Loading configuration values from [default.['build'].parameters] (env.command_name.section) in config file at '/oooo/sam-app/samconfig.toml'...
2023-03-08 12:07:12,671 | Configuration values successfully loaded.
2023-03-08 12:07:12,671 | Configuration values are: {'stack_name': 'sam-app', 'cached': True, 'parallel': True}
2023-03-08 12:07:12,672 | Using SAM Template at /oooo/sam-app/template.yaml
2023-03-08 12:07:12,697 | Using config file: samconfig.toml, config environment: default
2023-03-08 12:07:12,697 | Expand command line arguments to:
2023-03-08 12:07:12,697 | --template_file=/oooo/sam-app/template.yaml --manifest=package.json --parallel --build_dir=.aws-sam/build --cache_dir=.aws-sam/cache --cached 
2023-03-08 12:07:12,815 | 'build' command is called
2023-03-08 12:07:12,815 | Starting Build use cache
2023-03-08 12:07:12,816 | No Parameters detected in the template
2023-03-08 12:07:12,823 | There is no customer defined id or cdk path defined for resource HelloWorldFunction, so we will use the resource logical id as the resource id
2023-03-08 12:07:12,823 | There is no customer defined id or cdk path defined for resource ServerlessRestApi, so we will use the resource logical id as the resource id
2023-03-08 12:07:12,823 | 0 stacks found in the template
2023-03-08 12:07:12,823 | No Parameters detected in the template
2023-03-08 12:07:12,828 | There is no customer defined id or cdk path defined for resource HelloWorldFunction, so we will use the resource logical id as the resource id
2023-03-08 12:07:12,828 | There is no customer defined id or cdk path defined for resource ServerlessRestApi, so we will use the resource logical id as the resource id
2023-03-08 12:07:12,828 | 2 resources found in the stack 
2023-03-08 12:07:12,828 | Found Serverless function with name='HelloWorldFunction' and CodeUri='hello-world/'
2023-03-08 12:07:12,828 | --base-dir is not presented, adjusting uri hello-world/ relative to /oooo/sam-app/template.yaml
2023-03-08 12:07:12,830 | 2 resources found in the stack 
2023-03-08 12:07:12,830 | Found Serverless function with name='HelloWorldFunction' and CodeUri='hello-world/'
2023-03-08 12:07:12,830 | Found Serverless function with name='HelloWorldFunction' and CodeUri='hello-world/'
2023-03-08 12:07:12,830 | --base-dir is not presented, adjusting uri hello-world/ relative to /oooo/sam-app/template.yaml
2023-03-08 12:07:12,830 | Instantiating build definitions
2023-03-08 12:07:12,832 | Same function build definition found, adding function (Previous: BuildDefinition(nodejs18.x, /oooo/sam-app/hello-world, Zip, , e34abb10-3487-4a79-994f-c50649ea5154, {'BuildMethod': 'esbuild', 'BuildProperties': {'Minify': True, 'Target': 'es2020', 'Sourcemap': True, 'EntryPoints': ['app.ts']}}, {}, x86_64, []), Current: BuildDefinition(nodejs18.x, /oooo/sam-app/hello-world, Zip, , f3c293be-ea33-4d9f-b638-33fdcdb69c8e, {'BuildMethod': 'esbuild', 'BuildProperties': {'Minify': True, 'Target': 'es2020', 'Sourcemap': True, 'EntryPoints': ['app.ts']}}, {}, x86_64, []), Function: Function(function_id='HelloWorldFunction', name='HelloWorldFunction', functionname='HelloWorldFunction', runtime='nodejs18.x', memory=None, timeout=3, handler='app.lambdaHandler', imageuri=None, packagetype='Zip', imageconfig=None, codeuri='/oooo/sam-app/hello-world', environment=None, rolearn=None, layers=[], events={'HelloWorld': {'Type': 'Api', 'Properties': {'Path': '/hello', 'Method': 'get', 'RestApiId': 'ServerlessRestApi'}}}, metadata={'BuildMethod': 'esbuild', 'BuildProperties': {'Minify': True, 'Target': 'es2020', 'Sourcemap': True, 'EntryPoints': ['app.ts']}, 'SamResourceId': 'HelloWorldFunction'}, inlinecode=None, codesign_config_arn=None, architectures=['x86_64'], function_url_config=None, stack_path='', runtime_management_config=None))
2023-03-08 12:07:12,832 | Async execution started
2023-03-08 12:07:12,832 | Invoking function functools.partial(<bound method ParallelBuildStrategy.build_single_function_definition of <samcli.lib.build.build_strategy.ParallelBuildStrategy object at 0x107b0a690>>, <samcli.lib.build.build_graph.FunctionBuildDefinition object at 0x107b66510>)
2023-03-08 12:07:12,832 | Running incremental build for runtime nodejs18.x for following resources (HelloWorldFunction)
2023-03-08 12:07:12,833 | Waiting for async results
2023-03-08 12:07:12,833 | Manifest file is changed (new hash: 71b4a527ad81470e0879517c484f4ff7) or dependency folder (.aws-sam/deps/e34abb10-3487-4a79-994f-c50649ea5154) is missing for (HelloWorldFunction), downloading dependencies and copying/building source
2023-03-08 12:07:12,833 | Building codeuri: /oooo/sam-app/hello-world runtime: nodejs18.x metadata: {'BuildMethod': 'esbuild', 'BuildProperties': {'Minify': True, 'Target': 'es2020', 'Sourcemap': True, 'EntryPoints': ['app.ts']}} architecture: x86_64 functions: HelloWorldFunction
2023-03-08 12:07:12,833 | Building to following folder /oooo/sam-app/.aws-sam/build/HelloWorldFunction
2023-03-08 12:07:12,833 | Loading workflow module 'aws_lambda_builders.workflows'
2023-03-08 12:07:12,834 | Registering workflow 'CustomMakeBuilder' with capability 'Capability(language='provided', dependency_manager=None, application_framework=None)'
2023-03-08 12:07:12,834 | Registering workflow 'DotnetCliPackageBuilder' with capability 'Capability(language='dotnet', dependency_manager='cli-package', application_framework=None)'
2023-03-08 12:07:12,835 | Registering workflow 'GoModulesBuilder' with capability 'Capability(language='go', dependency_manager='modules', application_framework=None)'
2023-03-08 12:07:12,836 | Registering workflow 'JavaGradleWorkflow' with capability 'Capability(language='java', dependency_manager='gradle', application_framework=None)'
2023-03-08 12:07:12,836 | Registering workflow 'JavaMavenWorkflow' with capability 'Capability(language='java', dependency_manager='maven', application_framework=None)'
2023-03-08 12:07:12,837 | Registering workflow 'NodejsNpmBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm', application_framework=None)'
2023-03-08 12:07:12,837 | Registering workflow 'NodejsNpmEsbuildBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm-esbuild', application_framework=None)'
2023-03-08 12:07:12,838 | Registering workflow 'PythonPipBuilder' with capability 'Capability(language='python', dependency_manager='pip', application_framework=None)'
2023-03-08 12:07:12,838 | Registering workflow 'RubyBundlerBuilder' with capability 'Capability(language='ruby', dependency_manager='bundler', application_framework=None)'
2023-03-08 12:07:12,839 | Registering workflow 'RustCargoLambdaBuilder' with capability 'Capability(language='rust', dependency_manager='cargo', application_framework=None)'
2023-03-08 12:07:12,839 | Found workflow 'NodejsNpmEsbuildBuilder' to support capabilities 'Capability(language='nodejs', dependency_manager='npm-esbuild', application_framework=None)'
2023-03-08 12:07:12,839 | executing NPM: ['npm', 'root']
2023-03-08 12:07:13,006 | Lambda Builders found the following esbuild properties:
{"minify": true, "target": "es2020", "sourcemap": true, "entry_points": ["app.ts"]}
2023-03-08 12:07:13,007 | Running workflow 'NodejsNpmEsbuildBuilder'
2023-03-08 12:07:13,007 | Running NodejsNpmEsbuildBuilder:CopySource
2023-03-08 12:07:13,007 | Copying source file (/oooo/sam-app/hello-world/.npmignore) to destination (/var/folders/nf/nqq4l4_s0_n16m276ljfflv40000gn/T/tmpmr38yepq/.npmignore)
2023-03-08 12:07:13,007 | Copying source file (/oooo/sam-app/hello-world/app.ts) to destination (/var/folders/nf/nqq4l4_s0_n16m276ljfflv40000gn/T/tmpmr38yepq/app.ts)
2023-03-08 12:07:13,007 | Creating target folders at /var/folders/nf/nqq4l4_s0_n16m276ljfflv40000gn/T/tmpmr38yepq/tests
2023-03-08 12:07:13,007 | Copying directory metadata from source (/oooo/sam-app/hello-world/tests) to destination (/var/folders/nf/nqq4l4_s0_n16m276ljfflv40000gn/T/tmpmr38yepq/tests)
2023-03-08 12:07:13,007 | Creating target folders at /var/folders/nf/nqq4l4_s0_n16m276ljfflv40000gn/T/tmpmr38yepq/tests/unit
2023-03-08 12:07:13,007 | Copying directory metadata from source (/oooo/sam-app/hello-world/tests/unit) to destination (/var/folders/nf/nqq4l4_s0_n16m276ljfflv40000gn/T/tmpmr38yepq/tests/unit)
2023-03-08 12:07:13,008 | Copying source file (/oooo/sam-app/hello-world/tests/unit/test-handler.test.ts) to destination (/var/folders/nf/nqq4l4_s0_n16m276ljfflv40000gn/T/tmpmr38yepq/tests/unit/test-handler.test.ts)
2023-03-08 12:07:13,008 | Copying source file (/oooo/sam-app/hello-world/.prettierrc.js) to destination (/var/folders/nf/nqq4l4_s0_n16m276ljfflv40000gn/T/tmpmr38yepq/.prettierrc.js)
2023-03-08 12:07:13,008 | Copying source file (/oooo/sam-app/hello-world/.eslintrc.js) to destination (/var/folders/nf/nqq4l4_s0_n16m276ljfflv40000gn/T/tmpmr38yepq/.eslintrc.js)
2023-03-08 12:07:13,008 | Copying source file (/oooo/sam-app/hello-world/tsconfig.json) to destination (/var/folders/nf/nqq4l4_s0_n16m276ljfflv40000gn/T/tmpmr38yepq/tsconfig.json)
2023-03-08 12:07:13,008 | Copying source file (/oooo/sam-app/hello-world/.eslintignore) to destination (/var/folders/nf/nqq4l4_s0_n16m276ljfflv40000gn/T/tmpmr38yepq/.eslintignore)
2023-03-08 12:07:13,008 | Copying source file (/oooo/sam-app/hello-world/jest.config.ts) to destination (/var/folders/nf/nqq4l4_s0_n16m276ljfflv40000gn/T/tmpmr38yepq/jest.config.ts)
2023-03-08 12:07:13,008 | NodejsNpmEsbuildBuilder:CopySource succeeded
2023-03-08 12:07:13,008 | Running NodejsNpmEsbuildBuilder:NpmInstall
2023-03-08 12:07:13,008 | NODEJS installing in: /var/folders/nf/nqq4l4_s0_n16m276ljfflv40000gn/T/tmpmr38yepq
2023-03-08 12:07:13,008 | executing NPM: ['npm', 'install', '-q', '--no-audit', '--no-save', '--unsafe-perm', '--production']
2023-03-08 12:07:13,186 | NodejsNpmEsbuildBuilder:NpmInstall failed
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/aws-sam-cli/1.76.0/libexec/lib/python3.11/site-packages/aws_lambda_builders/workflows/nodejs_npm/actions.py", line 110, in execute
    self.subprocess_npm.run(
  File "/opt/homebrew/Cellar/aws-sam-cli/1.76.0/libexec/lib/python3.11/site-packages/aws_lambda_builders/workflows/nodejs_npm/npm.py", line 85, in run
    raise NpmExecutionError(message=err.decode("utf8").strip())
aws_lambda_builders.workflows.nodejs_npm.npm.NpmExecutionError: NPM Failed: npm WARN config production Use `--omit=dev` instead.
npm ERR! code ENOENT
npm ERR! syscall open
npm ERR! path /private/var/folders/nf/nqq4l4_s0_n16m276ljfflv40000gn/T/tmpmr38yepq/package.json
npm ERR! errno -2
npm ERR! enoent ENOENT: no such file or directory, open '/private/var/folders/nf/nqq4l4_s0_n16m276ljfflv40000gn/T/tmpmr38yepq/package.json'
npm ERR! enoent This is related to npm not being able to find a file.
npm ERR! enoent 

npm ERR! A complete log of this run can be found in:
npm ERR!     /oooo/.npm/_logs/2023-03-07T23_07_13_158Z-debug-0.log

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/homebrew/Cellar/aws-sam-cli/1.76.0/libexec/lib/python3.11/site-packages/aws_lambda_builders/workflow.py", line 370, in run
    action.execute()
  File "/opt/homebrew/Cellar/aws-sam-cli/1.76.0/libexec/lib/python3.11/site-packages/aws_lambda_builders/workflows/nodejs_npm/actions.py", line 115, in execute
    raise ActionFailedError(str(ex))
aws_lambda_builders.actions.ActionFailedError: NPM Failed: npm WARN config production Use `--omit=dev` instead.
npm ERR! code ENOENT
npm ERR! syscall open
npm ERR! path /private/var/folders/nf/nqq4l4_s0_n16m276ljfflv40000gn/T/tmpmr38yepq/package.json
npm ERR! errno -2
npm ERR! enoent ENOENT: no such file or directory, open '/private/var/folders/nf/nqq4l4_s0_n16m276ljfflv40000gn/T/tmpmr38yepq/package.json'
npm ERR! enoent This is related to npm not being able to find a file.
npm ERR! enoent 

npm ERR! A complete log of this run can be found in:
npm ERR!     /oooo/.npm/_logs/2023-03-07T23_07_13_158Z-debug-0.log
2023-03-08 12:07:13,187 | Exception raised during the execution

Build Failed
2023-03-08 12:07:13,188 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2023-03-08 12:07:13,189 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2023-03-08 12:07:13,189 | Unable to find Click Context for getting session_id.
Error: NodejsNpmEsbuildBuilder:NpmInstall - NPM Failed: npm WARN config production Use `--omit=dev` instead.
npm ERR! code ENOENT
npm ERR! syscall open
npm ERR! path /private/var/folders/nf/nqq4l4_s0_n16m276ljfflv40000gn/T/tmpmr38yepq/package.json
npm ERR! errno -2
npm ERR! enoent ENOENT: no such file or directory, open '/private/var/folders/nf/nqq4l4_s0_n16m276ljfflv40000gn/T/tmpmr38yepq/package.json'
npm ERR! enoent This is related to npm not being able to find a file.
npm ERR! enoent 

npm ERR! A complete log of this run can be found in:
npm ERR!     /oooo/.npm/_logs/2023-03-07T23_07_13_158Z-debug-0.log

Expected result:

It should build the project properly

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: macOS Ventura 13.1
2. sam --version: 1.76.0
3. AWS region: us-east-1

{
  "version": "1.76.0",
  "system": {
    "python": "3.11.2",
    "os": "macOS-13.1-arm64-arm-64bit"
  },
  "additional_dependencies": {
    "docker_engine": "Not available",
    "aws_cdk": "2.67.0 (build b6f7f39)",
    "terraform": "Not available"
  }
}

[mndeveci]

The example that was given in the other issue is probably regular nodejs18 hello world application (not the typescript one as you selected above). I do confirm the build fails, if I init an application template with Typescript after moving package.json from its original location and run sam build --manifest option.

But my question would be, why do you need to run sam build with manifest option? Usually, manifest is at the root level of the folder of the function which is denoted by CodeUri property of the function itself. If you place your package.json file there, sam build should work as expected, without needing to provide the --manifest option.

[oieduardorabelo]

Yes, the example in issue #4427 is a regular nodejs project:

doesn't work for TypeScript projects (it does work for plain Node.js).

As described in SAM CLI docs:

-m, --manifest PATH

The path to a custom dependency manifest file (for example, package.json) to use instead of the default.

https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-cli-command-reference-sam-build.html

I was searching for a way to share a package.json across all my lambda handlers instead of placing one package.json per lambda. That's how I arrived at #4427.

Usually, manifest is at the root level of the folder of the function which is denoted by CodeUri property of the function itself. If you place your package.json file there, sam build should work as expected, without needing to provide the --manifest option.

Correct. One workaround is to change the CodeUri in the sam init from hello-world/ to ./ and update the entry path from app.ts to hello-world/app.ts, everything works as expected. I don't need --manifest in this case.

For example, with sam init, the template.yml looks like:

Resources:
  HelloWorldFunction:
    #...
    Properties:
      CodeUri: hello-world/
      # ...
    Metadata:
      BuildProperties:
        # ...
        EntryPoints: 
        - app.ts

If I change it to:

Resources:
  HelloWorldFunction:
    #...
    Properties:
      CodeUri: ./
      # ...
    Metadata:
      BuildProperties:
        # ...
        EntryPoints: 
        - hello-world/app.ts

It works correctly and I can share the single package.json across all my handlers and I don't need --manifest.

But after reading what --manifest should do, my understanding is that it should respect/use the path I suggested where the package.json is @mndeveci

[mndeveci]

Correct. One workaround is to change the CodeUri in the sam init from hello-world/ to ./ and update the entry path from app.ts to hello-world/app.ts, everything works as expected. I don't need --manifest in this case.

If you want to go into that direction, I would recommend putting lambda code in particular folder, and place your package.json file there. Setting CodeUri to root folder of the stack might cause other issues with other commands. Something like this would be better for your use case;

├── functions
│   ├── package.json
│   ├── function1
│   │   └── app.js
│   └── function2
│       └── app.js
└── template.yaml

And you need to set CodeUri to functions folder.

But after reading what --manifest should do, my understanding is that it should respect/use the path I suggested where the package.json is @mndeveci

I think this might be related to how we build functions with esbuild and that is why this is not seen on regular nodejs functions but it is seen in the ones which is built with esbuild. I will take this to the team to discuss further.

[oieduardorabelo]

Thanks for the support @mndeveci 👍

[oieduardorabelo]

After trying your suggestion locally @mndeveci , I pushed an example to https://github.com/oieduardorabelo/aws-sam-typescript-shared-package-json

Because I use eslint / typescript for other files in my project, I need it at the root level. The suggestion of having a package.json inside functions folders adds some extra manual steps and requires developer attention too.

It would be great to declare the dependencies key in the root package.json (see the example repository).

In other projects, I'm still using CodeUri: ./ to workaround this issue, and I haven't faced any problems so far. It is fair to say I haven't used sam local or local debugger with VSCode. So, I'm unsure if CodeUri: ./ introduces problems with other commands.

I can confirm sam build, sam package, sam sync, sam deploy works correctly with CodeUri: ./ and I can use a single package.json across my TypeScript project.

[oieduardorabelo]

Hey @mndeveci I created a working example of my previous comment on:

https://github.com/oieduardorabelo/aws-sam-nested-applications-single-package-json

[mndeveci]

Thanks @oieduardorabelo

As far as I understand, this example repository that you linked works as expected for your use case. If that is the case I am glad that you are unblocked now.

For the issue with running sam build --manifest ./package.json, I will be taking it to the team to discuss it further.