SAM build still fails for go with `--use-container`

[michal-sa]

Description:

Related to: #5280 building a template containing a SAM go function still doesn't seem to work.

Steps to reproduce:

Boostrap handler

Function resource following the format from: https://aws.amazon.com/blogs/compute/migrating-aws-lambda-functions-from-the-go1-x-runtime-to-the-custom-runtime-on-amazon-linux-2/:

  Function:
    Type: AWS::Serverless::Function
    Properties:
      FunctionName: MyFunction
      CodeUri: cmd/func
      Timeout: 30
      Handler: bootstrap
      Runtime: provided.al2023
      Architectures:
        - x86_64
    Metadata:
      BuildMethod: go1.x

Still gives the error message:
aws_lambda_builders.exceptions.WorkflowFailedError: GoModulesBuilder:Build - Builder Failed: go: go.mod file not found in current directory or any parent directory; see 'go help modules'

Standard handler

Changing the template resource to:

  Function:
    Type: AWS::Serverless::Function
    Properties:
      FunctionName: MyFunction
      CodeUri: .
      Timeout: 30
      Handler: cmd/func
      Runtime: provided.al2023
      Architectures:
        - x86_64
    Metadata:
      BuildMethod: go1.x

Instead gives a lot of errors about git not being present in path, for example:
git init --bare in /root/go/pkg/mod/cache/vcs/0a897d51bfc9ae666ee1454be13381a5c8df2e3bb2a5bd1dba78528978aa3b56: exec: "git": executable file not found in $PATH

Expected result:

Sam build manages to build the go inside a container without any issues

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: Ubuntu 20.04.6 LTS
2. sam --version: SAM CLI, version 1.108.0
3. AWS region: eu-west-1

Paste the output of sam --info here

{
  "version": "1.108.0",
  "system": {
    "python": "3.9.18",
    "os": "Linux-5.14.0-1055-oem-x86_64-with-glibc2.31"
  },
  "additional_dependencies": {
    "docker_engine": "24.0.7",
    "aws_cdk": "2.93.0 (build 724bd01)",
    "terraform": "1.3.7"
  },
  "available_beta_feature_env_vars": [
    "SAM_CLI_BETA_FEATURES",
    "SAM_CLI_BETA_BUILD_PERFORMANCE",
    "SAM_CLI_BETA_TERRAFORM_SUPPORT",
    "SAM_CLI_BETA_RUST_CARGO_LAMBDA"
  ]
}

[mndeveci]

Hi there,

I assume you have more than one lambda function inside cmd folder, since this works with single Go lambda functions. I will tag this as a bug and we will work on this.

In the meantime, you can workaround issue with following steps;

• Update your template to point to the folder where you have go.mod file.
• Setup custom build using Makefile to build function inside the container;

build-HelloWorldFunction:
	go build -o bootstrap cmd/helloWorld/main.go
	cp bootstrap $(ARTIFACTS_DIR)

Here was my folder structure and template;

.
├── hello-world
│   ├── Makefile
│   ├── cmd
│   │   └── helloWorld
│   │       └── main.go
│   ├── go.mod
│   └── go.sum
└── template.yaml

Resources:
  HelloWorldFunction:
    Type: AWS::Serverless::Function
    Metadata:
      BuildMethod: makefile
    Properties:
      CodeUri: hello-world
      Handler: cmd/helloWorld
      Runtime: provided.al2

[michal-sa]

Thanks for fast respose @mndeveci. I actually only have one function inside the cmd folder where I'm running the sam build --use-container command. My folder structure looks like this:

├── func
│   ├── cmd
│   │   └── func
│   │       └── main.go
│   ├── go.mod
│   ├── go.sum
|   └── template.yaml

I tried to use the same as in your hello world example (without the Makefile), but that didn't work for me either with the same errors. Do you have a working template/structure that works inside the container?

[mndeveci]

@michal-sa if you have single lambda function, then you can move your main.go file under func and it should work in that case. Your folder structure will look like;

├── func
│   ├── main.go
│   ├── go.mod
│   ├── go.sum
└── template.yaml

and your function resource should look like this;

Resources:
  HelloWorldFunction:
    Type: AWS::Serverless::Function
    Metadata:
      BuildMethod: go1.x
    Properties:
      CodeUri: func/
      Handler: bootstrap
      Runtime: provided.al2

Please let us know if this does work for your case

[michal-sa]

Thanks for detailed example @mndeveci.

TL;DR

I managed to get the build inside of container to work (with some workarounds especially for the provided.al2023 runtime). Will need to perform some more tests the next week to see if the binaries work during runtime as well.

---

I have now tested the structure you suggested using sam init (to make the setup and reproduction faster) and here are my current findings:

hello-world with provided.al2

1. sam init --runtime provided.al2 --app-template hello-world --name func (and select go in runtime the prompt)
   Produces template:

   Metadata:
     BuildMethod: go1.x
   Properties:
     CodeUri: hello-world/
     Handler: bootstrap
     Runtime: provided.al2
     Architectures:
       - x86_64

   Tree:

   func
   ├── hello-world
   │   ├── go.mod
   │   ├── go.sum
   │   ├── main.go   
   ├── samconfig.toml
   └── template.yaml
   

2. sam build --use-container works fine!

hello-world with provided.al2023

1. sam init --runtime provided.al2023 --app-template hello-world --name func (and select go in runtime the prompt)
   Produces template:

   Metadata:
    BuildMethod: go1.x
   Properties:
    CodeUri: hello-world/
    Handler: bootstrap
    Runtime: provided.al2023
    Architectures:
      - x86_64

   Tree:

   func
   ├── hello-world
   │   ├── go.mod
   │   ├── go.sum
   │   ├── main.go   
   ├── samconfig.toml
   └── template.yaml
   

2. sam build --use-container fails with:

   Build Failed
   Workflow GoModulesBuilder does not support value "False" for building in source. Using default value "True".
    Running GoModulesBuilder:Build
   Error: GoModulesBuilder:Build - Builder Failed: go: github.com/aws/aws-lambda-go@v1.36.1: git init --bare in 
   /root/go/pkg/mod/cache/vcs/3bc91b50dfbdd43d74dbeb370316faf0b6729c7e2769b09aeafd7c167e3f6001: exec: "git": 
   executable file not found in $PATH
   

Fixing hello-world template

I noticed that the following steps help the container build while using the provided.al2023 runtime:

1. Bump the go.mod version to at least 1.18 (due to some outdated dependencies giving the following errors:

   # github.com/aws/aws-lambda-go/lambda
   embedding non-interface type func(context.Context, TIn) (TOut, error) requires go1.18 or later (-lang was set to go1.16; check go.mod)
   

2. Run go mod vendor before sam build. This seems to speed up the build time as well as fixing the not found git errors. Not sure if this is a good idea as the vendor and download of dependencies will be done on another OS than the docker/lambda runtime (if that matters for go, I'm no golang expert).

Own go func

The template and function I try to get to work is more complex with e.g. refs to folders outside of the current function directory (which of course won't work well while Docker is mounting only the function folder). The debugging is a little hard, especially whilie using the provided.al2 runtime

provided.al2

After running the build command it gets stuck for a longer while (~10 minutes) after:

Running workflow 'GoModulesBuilder'
 Running GoModulesBuilder:Build

it does show helpful (debug) error messages like e.g.:

2024-02-04 12:47:18,213 | Mounting <CODE_URI_FULL_PATH> as /tmp/samcli/source:ro,delegated, inside runtime container                                                    
Using the request object from command line argument
Loading workflow module 'aws_lambda_builders.workflows'
Registering workflow 'CustomMakeBuilder' with capability 'Capability(language='provided', dependency_manager=None, application_framework=None)'
Registering workflow 'DotnetCliPackageBuilder' with capability 'Capability(language='dotnet', dependency_manager='cli-package', application_framework=None)'
Registering workflow 'GoModulesBuilder' with capability 'Capability(language='go', dependency_manager='modules', application_framework=None)'
Registering workflow 'JavaGradleWorkflow' with capability 'Capability(language='java', dependency_manager='gradle', application_framework=None)'
Registering workflow 'JavaMavenWorkflow' with capability 'Capability(language='java', dependency_manager='maven', application_framework=None)'
Registering workflow 'NodejsNpmBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm', application_framework=None)'
Registering workflow 'NodejsNpmEsbuildBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm-esbuild', application_framework=None)'
Registering workflow 'PythonPipBuilder' with capability 'Capability(language='python', dependency_manager='pip', application_framework=None)'
Registering workflow 'RubyBundlerBuilder' with capability 'Capability(language='ruby', dependency_manager='bundler', application_framework=None)'
Registering workflow 'RustCargoLambdaBuilder' with capability 'Capability(language='rust', dependency_manager='cargo', application_framework=None)'
Found workflow 'GoModulesBuilder' to support capabilities 'Capability(language='go', dependency_manager='modules', application_framework=None)'
Workflow GoModulesBuilder does not support value "False" for building in source. Using default value "True".
Running workflow 'GoModulesBuilder'

After 10 minutes:

..... replacement directory ../../common does not exist
2024-02-04 12:57:13,459 | Build inside container returned response {"jsonrpc": "2.0", "id": 1, "error": {"code": 400, "message": "GoModulesBuilder:Build - Builder Failed: stat /tmp/samcli/source/bootstrap: directory not    
found"}}

Of course these build bugs can probably be caught faster running a dry run build outside of docker too.

provided.al2023

This runtime still seems to fail faster mainly due to the git errors again and the go mod vendor method seems to fix the problems as well.

[michal-sa]

I finally got some time to try out a more complex project structure containing references outside of the SAM container mount point.

Had to create some pre processing scripts copying the dependent directories into the SAM CodeUri folder, replacing all go.mod replace directives to be relative the mount point and finally running go mod vendor inside all of the folders containing a go.mod file.

After doing this the lambda built inside the container without errors and worked fine during runtime as well. Think this issue could be closed as the build inside container does work (although with some workarounds).

[github-actions]

Patch is released in v1.111.0. Closing

[fynnfluegge]

Not able to get this running, going to use prebuilt .aws-sam directory and mount it with docker-compose. This works fine.