sam build fail with "No such file or directory" for node build

[Cubid]

Description

When building with
sam build --template sam-template.yaml
the build fail due to non existent temporary folder

Steps to reproduce

run sam build --template sam-template.yaml with js source in a scub package

Observed result

Running workflow 'NodejsNpmBuilder'
Running NodejsNpmBuilder:NpmPack
NODEJS packaging file:/authorizer to /tmp/tmpmn3v_qa7
executing NPM: ['npm', 'pack', '-q', 'file:/authorizer']
NODEJS packed to > backlinks-authorizer@1.0.0 prepare /authorizer

npm run build

backlinks-authorizer@1.0.0 build /authorizer
tsc --outDir build

backlinks-authorizer-1.0.0.tgz
NODEJS extracting to /tmp/tmpmn3v_qa7/unpacked
NodejsNpmBuilder:NpmPack raised unhandled exception
Traceback (most recent call last):
File "/home/xxx/.local/lib/python3.6/site-packages/aws_lambda_builders/workflow.py", line 248, in run
action.execute()
File "/home/xxx/.local/lib/python3.6/site-packages/aws_lambda_builders/workflows/nodejs_npm/actions.py", line 67, in execute
self.osutils.extract_tarfile(tarfile_path, self.artifacts_dir)
File "/home/xxx/.local/lib/python3.6/site-packages/aws_lambda_builders/workflows/nodejs_npm/utils.py", line 23, in extract_tarfile
with tarfile.open(tarfile_path, 'r:*') as tar:
File "/usr/lib/python3.6/tarfile.py", line 1571, in open
return func(name, "r", fileobj, **kwargs)
File "/usr/lib/python3.6/tarfile.py", line 1636, in gzopen
fileobj = gzip.GzipFile(name, mode + "b", compresslevel, fileobj)
File "/usr/lib/python3.6/gzip.py", line 163, in init
fileobj = self.myfileobj = builtins.open(filename, mode or 'rb')
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpmn3v_qa7/> backlinks-authorizer@1.0.0 prepare /authorizer\n> npm run build\n\n\n> backlinks-authorizer@1.0.0 build //authorizer\n> tsc --outDir build\n\nbacklinks-authorizer-1.0.0.tgz'

Build Failed

Expected result

The temp directory (and sub dirs) should be created for the script to continue

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: Ubuntu 18.04.3
2. sam --version: SAM CLI, version 0.22.0

Add --debug flag to command you are running

[Cubid]

Seems like filename is the whole output of npm build command. Mine is:

> backlinks-authorizer@1.0.0 prepare <build-dir>/authorizer
> npm run build


> backlinks-authorizer@1.0.0 build <build-dir>/authorizer
> tsc --outDir build

backlinks-authorizer-1.0.0.tgz

it seems it is a issue in npm silent option

# npm pack -q file:<build-dir>/authorizer

> backlinks-authorizer@1.0.0 prepare <build-dir>/authorizer
> tsc --outDir build

backlinks-authorizer-1.0.0.tgz

$ npm pack -s file:<build-dir>/authorizer
backlinks-authorizer-1.0.0.tgz

it seems that changing the flag from -q to -s in aws_lambda_builders/workflows/nodejs_npm/actions.py solve the issue

[awood45]

Can you expand a bit on the steps to reproduce? If the fix is simple I'd want to fast track it, but I'm not sure I understand the conditions under which I can reproduce this to confirm the fix is correct.

[Cubid]

I implemented a small project based on the default node sam app:
https://github.com/Cubid/aws-sam-cli-1426
And I made the breaking change here:
Cubid/aws-sam-cli-1426@61d4818

On my side I use TypeScript compiler, but it is the same for any command added in the build.

sam build --template template.yaml is crashing on this project

[seriojaignat]

I also work with TypeScript and my sam build fails because the dist folder is not copied to .aws-sam folder. The reason is that dist is listed in .gitignore and thereby ignored by the sam build command (why?). The ugly solution is to remove dist or any output folder from .gitignore.
It is normal for dist or any output directory to be ignored by the code management tool, but it is necessary for those folders to be part of the package.
How can I configure sam build to not ignore the dist folder or any other output folder used by my compiler or transpiler?

[awood45]

@Cubid Thanks for the example, I'll take a look at that now.

@seriojaignat Our JS builder is specific for Node, and runs npm pack - are you compiling from TS to JS before you run sam build?

[awood45]

@Cubid it looks like what you're trying to do is override the prepack step - I know what you gave me is a reproducing example, which it does reproduce, but is the idea here that you're trying to shim in TypeScript support?

[awood45]

Also, what are you actually trying to put in the prepack step?

[awood45]

Ok, here's a bit of background to help out. https://github.com/awslabs/aws-lambda-builders/blob/develop/aws_lambda_builders/workflows/nodejs_npm/DESIGN.md contains the design of the Node builder that's run by sam build.

@seriojaignat we run npm pack which I understand ignores files in your .gitignore by default, so you may want to add an .npmignore file that has a narrower scope and see if that includes the files you want built.

@Cubid What version of node are you running? I'm doing some research and it does look like prepack can have issues with pack depending on version (though I'm not a node expert myself so I'm researching in parallel here).

[Cubid]

@awood45 In the script section of my package.json, I have the following line:
"prepare": "tsc --outDir build"
It's not link to prepack, but to any script that will output something
Since the whole output is taken as file name, the file name is not correct.

[awood45]

Understood - we have some features in our backlog that should make this easier in general, but I'm looking in to what to recommend for this use case.