sam build includes absolute paths in Golang binaries

[suroh1994]

Description:

When you run sam build and sam package in two instances of the same projects with different absolute paths and upload to the same S3 bucket, the compiled binaries will be uploaded both times, even though the binaries are identical.
According to #2110 this has been fixed, but I can still reproduce the issue on Windows.

Steps to reproduce:

1. Clone this repository into a directory on your local Windows machine.
2. cd into the directory.
3. Run sam build.
4. Run sam package --s3-bucket YOUR_BUCKETNAME
5. Clone the same repository into a second directory on your local windows machine.
6. Copy the second directory into a third directory using xcopy seconddirectory thirddirectory /s /e.
7. cd into the second directory.
8. Run sam build.
9. Run sam package --s3-bucket YOUR_BUCKETNAME
10. cd into the third directory.
11. Run sam build.
12. Run sam package --s3-bucket YOUR_BUCKETNAME

Observed result:

The calculated hashes change, when the parent directory changes. This results in the compiled binaries being re-uploaded for every directory, although no files have been modified.

Expected result:

Hashes should not depend on the parent directory. Compiled binaries should only be uploaded, when they have been modified.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: Windows 10 Pro Version 20H2 (Build 19042.1165)
2. sam --version: 1.31.0
3. AWS region: eu-central-1

Further notes:

I can imagine this is also the cause for the issue #2872, where running sam package in an ephemeral container results in the files being uploaded every time.

[suroh1994]

I've played around with the source code and it seems the hashes are correct, but instead the compiled binaries are not identical. But I have not yet identified, why they differ.

[suroh1994]

I believe I have identified the issue. When I build my binaries using sam build, the resulting binary is identical to go build. But when I use go build -trimpath the binary is different, yet the binaries in both directories are now identical. I'm looking into what other metadata is stored in the binaries, that could let to the binaries differing between two CodeBuild runs.

[suroh1994]

I solved the issue successfully for both local builds and builds generated by CodeBuild. As my last post suggested, the issue came down to go build embedding absolute paths into the binary. So to resolve the issue I did the following:

To have sam build use the -trimpath flag, I setup makefiles for every Lambda function and added these lines to each function:

    Metadata:
      BuildMethod: makefile

Then I added a Makefile file to each directory listed as CodeUri with the following content:

build-Logical-Resource-Id:
	go build -trimpath -o "$(ARTIFACTS_DIR)/your-handler-name"

For more information on how to use a makefile, see the documentation.
A complete example can be found in the resolved branch of my repo.

Going forward, it might be interesting to consider using the -trimpath by default. I do not know how important it is to keep the information in the binaries and which variant more commonly would lead to issues.

[suroh1994]

Updated title to reflect the found issue.

[aahung]

@suroh1994 Thanks for the info, it is very valuable. We will consider adding -trimpath by default if that avoid such issue

[jaglade]

Any update on if trimpath can be included by default? My group is running into this issue atm.

[sriram-mv]

@jaglade This will be a welcome contribution to the aws-lambda-builders golang build workflow and we can make this feature opt-in (atleast at this point, I have not considered the ramifications of the change in behavior causing breaks).