Bug: `sam build --use-container` fails for `go1.x` on `provided.al2`/`al2023` runtimes

[davidjb]

Description:

The standard go1.x runtime is based on AL1, which is both EOL and doesn't support ARM-based CPUs. The suggested replacement is to use the provided.al2 or provided.al20223 runtimes with BuildMethod: go1.x. This works sam build is run locally but only on hosts where issues like aws/aws-lambda-go#340 don't occur. In order to create a normalised build environment, I attempted to build with sam build --use-container instead, which errors with:

aws_lambda_builders.exceptions.WorkflowFailedError: GoModulesBuilder:Resolver - Path resolution for runtime: provided of binary: go was not successful

The reason for this is that the go binary is not present within the sam/build-provided.al2 image and thus preventing builds. A PR had been created at aws/aws-sam-build-images#71 to add Go into the provided and provided.al2 runtimes but is under discussion as of the end of 2022.

Steps to reproduce:

1. Specify a lambda function in template.yml:

AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::Serverless-2016-10-31
Resources:
  MyFunction:
    Type: AWS::Serverless::Function
    Metadata:
      BuildMethod: go1.x
    Properties:
      CodeUri: functions/myfunction/
      Handler: bootstrap
      Runtime: provided.[al2|al2023]
      Architectures: [arm64]

2. Run sam build --use-container.

Observed result:

$ sam build --use-container --debug
2023-06-07 12:39:32,226 | Config file location: /Users/me/project/samconfig.toml
2023-06-07 12:39:32,226 | Config file '/Users/me/project/samconfig.toml' does not exist
2023-06-07 12:39:32,227 | Using SAM Template at /Users/me/project/template.yml
2023-06-07 12:39:32,242 | Using config file: samconfig.toml, config environment: default
2023-06-07 12:39:32,242 | Expand command line arguments to:
2023-06-07 12:39:32,242 | --template_file=/Users/me/project/template.yml --use_container --mount_with=READ --build_dir=.aws-sam/build --cache_dir=.aws-sam/cache
2023-06-07 12:39:32,302 | 'build' command is called
2023-06-07 12:39:32,302 | Starting Build inside a container
2023-06-07 12:39:32,303 | No Parameters detected in the template
2023-06-07 12:39:32,313 | There is no customer defined id or cdk path defined for resource MyFunction, so we will use the resource logical id as the resource id
2023-06-07 12:39:32,313 | 0 stacks found in the template
2023-06-07 12:39:32,314 | No Parameters detected in the template
2023-06-07 12:39:32,321 | There is no customer defined id or cdk path defined for resource MyFunction, so we will use the resource logical id as the resource id
2023-06-07 12:39:32,321 | 1 resources found in the stack
2023-06-07 12:39:32,321 | Found Serverless function with name='MyFunction' and CodeUri='functions/myfunction/'
2023-06-07 12:39:32,321 | --base-dir is not presented, adjusting uri functions/myfunction/ relative to /Users/me/project/template.yml
2023-06-07 12:39:32,322 | 1 resources found in the stack
2023-06-07 12:39:32,322 | Found Serverless function with name='MyFunction' and CodeUri='functions/myfunction/'
2023-06-07 12:39:32,323 | Instantiating build definitions
2023-06-07 12:39:32,324 | Same function build definition found, adding function (Previous: BuildDefinition(provided.al2, /Users/me/project/functions/myfunction, Zip, , 1e222f6c-cb64-4fce-8a77-41ed2bea56bb, {'BuildMethod': 'go1.x'}, {}, arm64, []), Current: BuildDefinition(provided.al2, /Users/me/project/functions/myfunction, Zip, , 634c58e1-f958-46a9-800e-69617725ab30, {'BuildMethod': 'go1.x'}, {}, arm64, []), Function: Function(function_id='MyFunction', name='MyFunction', functionname='MyFunction', runtime='provided.al2', memory=None, timeout=None, handler='bootstrap', imageuri=None, packagetype='Zip', imageconfig=None, codeuri='/Users/me/project/functions/myfunction', environment=None, rolearn=None, layers=[], events=None, metadata={'BuildMethod': 'go1.x', 'SamResourceId': 'MyFunction'}, inlinecode=None, codesign_config_arn=None, architectures=['arm64'], function_url_config=None, stack_path='', runtime_management_config=None))
2023-06-07 12:39:32,325 | Building codeuri: /Users/me/project/functions/myfunction runtime: provided.al2 metadata: {'BuildMethod': 'go1.x'} architecture: arm64 functions: MyFunction
2023-06-07 12:39:32,325 | Building to following folder /Users/me/project/.aws-sam/build/MyFunction

Fetching public.ecr.aws/sam/build-provided.al2:latest-arm64 Docker container image......
2023-06-07 12:39:35,721 | Mounting /Users/me/project/functions/myfunction as /tmp/samcli/source:ro,delegated, inside runtime container
Using the request object from command line argument
Loading workflow module 'aws_lambda_builders.workflows'
Registering workflow 'CustomMakeBuilder' with capability 'Capability(language='provided', dependency_manager=None, application_framework=None)'
Registering workflow 'DotnetCliPackageBuilder' with capability 'Capability(language='dotnet', dependency_manager='cli-package', application_framework=None)'
Registering workflow 'GoModulesBuilder' with capability 'Capability(language='go', dependency_manager='modules', application_framework=None)'
Registering workflow 'JavaGradleWorkflow' with capability 'Capability(language='java', dependency_manager='gradle', application_framework=None)'
Registering workflow 'JavaMavenWorkflow' with capability 'Capability(language='java', dependency_manager='maven', application_framework=None)'
Registering workflow 'NodejsNpmBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm', application_framework=None)'
Registering workflow 'NodejsNpmEsbuildBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm-esbuild', application_framework=None)'
Registering workflow 'PythonPipBuilder' with capability 'Capability(language='python', dependency_manager='pip', application_framework=None)'
Registering workflow 'RubyBundlerBuilder' with capability 'Capability(language='ruby', dependency_manager='bundler', application_framework=None)'
Registering workflow 'RustCargoLambdaBuilder' with capability 'Capability(language='rust', dependency_manager='cargo', application_framework=None)'
Found workflow 'GoModulesBuilder' to support capabilities 'Capability(language='go', dependency_manager='modules', application_framework=None)'
Builder workflow failed
Traceback (most recent call last):
  File "/usr/local/opt/sam-cli/lib64/python3.7/site-packages/aws_lambda_builders/workflow.py", line 78, in wrapper
    if not binary_checker.path_provided
  File "/usr/local/opt/sam-cli/lib64/python3.7/site-packages/aws_lambda_builders/path_resolver.py", line 33, in exec_paths
    return self._which()
  File "/usr/local/opt/sam-cli/lib64/python3.7/site-packages/aws_lambda_builders/path_resolver.py", line 27, in _which
    "Path resolution for runtime: {} of binary: " "{} was not successful".format(self.runtime, self.binary)
ValueError: Path resolution for runtime: provided of binary: go was not successful

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/opt/sam-cli/lib64/python3.7/site-packages/aws_lambda_builders/__main__.py", line 133, in main
    build_in_source=params.get("build_in_source", None),
  File "/usr/local/opt/sam-cli/lib64/python3.7/site-packages/aws_lambda_builders/builder.py", line 170, in build
    return workflow.run()
  File "/usr/local/opt/sam-cli/lib64/python3.7/site-packages/aws_lambda_builders/workflow.py", line 82, in wrapper
    raise WorkflowFailedError(workflow_name=self.NAME, action_name="Resolver", reason=str(ex))
aws_lambda_builders.exceptions.WorkflowFailedError: GoModulesBuilder:Resolver - Path resolution for runtime: provided of binary: go was not successful
2023-06-07 12:39:36,080 | Build inside container returned response {"jsonrpc": "2.0", "id": 1, "error": {"code": 400, "message": "GoModulesBuilder:Resolver - Path resolution for runtime: provided of binary: go was not successful"}}

Build Failed
2023-06-07 12:39:36,158 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2023-06-07 12:39:36,159 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2023-06-07 12:39:36,160 | Unable to find Click Context for getting session_id.
Error: GoModulesBuilder:Resolver - Path resolution for runtime: provided of binary: go was not successful

Expected result:

Build to succeed and not error with --use-container.

Alternatively, if there was a way to set local environment variables such they're passed to go build locally (e.g. when containers aren't used), that would be a workaround to the initial problem.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

{
  "version": "1.85.0",
  "system": {
    "python": "3.8.16",
    "os": "macOS-13.4-arm64-arm-64bit"
  },
  "additional_dependencies": {
    "docker_engine": "23.0.5",
    "aws_cdk": "Not available",
    "terraform": "1.1.0"
  },
  "available_beta_feature_env_vars": [
    "SAM_CLI_BETA_FEATURES",
    "SAM_CLI_BETA_BUILD_PERFORMANCE",
    "SAM_CLI_BETA_TERRAFORM_SUPPORT",
    "SAM_CLI_BETA_RUST_CARGO_LAMBDA"
  ]
}

[jonachehilton]

+1

[jfuss]

@davidjb The halt is mostly due to how we support things like this. As the provided runtime has grown in support, we cannot keep adding runtime specific tools into one image.

We haven't made any decisions on paths forward however you can do this yourself by providing --build-image to the sam build command.

[davidjb]

Thanks @jfuss, glad to know solutions are being considered.

As you've said about --build-image, a workaround is to extend the existing provided.al2 image with a Dockerfile like so:

FROM public.ecr.aws/sam/build-provided.al2:latest-arm64

# Install Go
RUN curl -L https://go.dev/dl/go1.19.linux-arm64.tar.gz | tar -zx -C /usr/local
ENV PATH=$PATH:/usr/local/go/bin:/root/go/bin
# Set GOPROXY envvar to avoid using the default proxy.golang.org proxy
ENV GOPROXY=direct

# Compatible with initial base image
ENTRYPOINT []
CMD ["/bin/bash"]

Then build the image (e.g. docker build -t provided.al2-with_go .) and use that with sam build, with --build-image ... as you say or in samconfig.toml:

[default.build.parameters]
use_container = true
build_image = [
  "TestFunction=provided.al2-with_go",
  "TestFunction2=provided.al2-with_go",
  ...
]

This works, but you end up needing to re-specify the build image for each function explicitly (since other functions use different runtimes and I can't use a global build_image). There's also having to manage build processes for this extra container, and ensuring it's up-to-date etc.

If SAM CLI were create the Docker image dynamically (e.g. if go1.x is requested, create/extend a local provided.al2 image to have go in it), then that'd solve the problem and avoid the need to publish build variations of provided.al2, if that's the concern. Alternatively, the layer produced by my Dockerfile above is quite thin and effectively matches what the existing go1.x build image does. Either way, it would be good to have a solution built in to SAM CLI.

[mildaniel]

This works, but you end up needing to re-specify the build image for each function explicitly (since other functions use different runtimes and I can't use a global build_image)

@davidjb what would your thoughts be for SAM CLI to provide some sort of per-runtime/buildmethod configuration interface? Maybe something along the lines of an environment variable like AWS_SAM_CLI_BUILDMETHOD_CONFIGURATION that will pick up the particular configuration for all functions in your template with the same build method?

We're looking into ways of improving the configurations for the CLI and appreciate any suggestions you may have.

[davidjb]

@mildaniel That could be helpful for specific situations where customisation is need (e.g. a specific Go version for builds); having this as able to be specified in config as well as environment variables so it can be easily checked in to version control.

That said, the problem in this case is just about having go present within the given Docker build container - be it already there in the base image or installed manually. My preference is that SAM CLI is capable of achieving this by default without requiring external config, much in the same way SAM CLI knows how to build for or execute a given runtime given its name (e.g. go1.x).

[vikhyat187]

Hey @davidjb @jfuss can we use the --build-image of Go1.x or would that cause any problems as the Go1.x has been deprecating?

Thanks

[adamdavis40208]

Building on to that last question, will the Go1.x image continue to get updated? It currently has 1.20 on it, but how long will it get updated/when should we plan to stop using it? (and will it get 1.21 for example)

docker run --rm -it public.ecr.aws/sam/build-go1.x:latest bash                                                                  
bash-4.2# go version
go version go1.20 linux/amd64

[davidjb]

@adamdavis40208 As per https://docs.aws.amazon.com/lambda/latest/dg/lambda-golang.html (and various other pages), go1.x (which is based on AL1) is slated for deprecation from 2023-12-31. I couldn't speak to whether its version internally would be updated ahead of then, however.

It's probably the same answer for you @vikhyat187 - using the old, deprecated go1.x may work right now but isn't a solution. Quoting the page above: If you're using the Go 1.x runtime, you must migrate your functions to provided.al2023 or provided.al2.

[shearn89]

Commenting just for notifications really - interested to see what the recommended way forward for Go via SAM is, when SAM is officially supported but the go1.x runtime is not and the recommendation is to use provided.alX.

Seems like a bit of miscommunication between teams - Lambda team dropping support for the Go runtime, but SAM team not adding in the tooling for the recommended solution...