sam local invoke not using ca_bundle

[JaGarRod]

Description:

In #917 I understand that #1143 is intended as a fix. However I am not able to local invoke lambdas that need to pass through my local certificate.

Steps to reproduce:

sam local invoke SSLERRORtestlambda --debug --profile default

My template:

AWSTemplateFormatVersion: "2010-09-09"
Transform: "AWS::Serverless-2016-10-31"

Resources:
  SSLERRORtestlambda:
    Type: 'AWS::Serverless::Function'
    Properties:
      Handler: test.list_clusters
      Runtime: python3.7
      Timeout: 50
      Policies:
        Version: "2012-10-17"
        Statement:
          - Effect: "Allow"
            Action:
              - "elasticmapreduce:ListClusters"
            Resource: "*"

and test.py:

import boto3

def list_clusters(event, context):
    clusters = boto3.client('emr').list_clusters()
    print(clusters)

    return event

.aws/credentials:

[default]
region=eu-west-1
ca_bundle=/path_to_certificate/root_ca_certificate.crt

Observed result:

2021-03-05 11:28:46,237 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2021-03-05 11:28:46,238 | local invoke command is called
2021-03-05 11:28:46,241 | No Parameters detected in the template
2021-03-05 11:28:46,288 | 1 resources found in the stack 
2021-03-05 11:28:46,288 | No Parameters detected in the template
2021-03-05 11:28:46,313 | No Parameters detected in the template
2021-03-05 11:28:46,338 | Found Serverless function with name='SSLERRORtestlambda' and CodeUri='.'
2021-03-05 11:28:46,419 | Found one Lambda function with name 'SSLERRORtestlambda'
2021-03-05 11:28:46,419 | Invoking test.list_clusters (python3.7)
2021-03-05 11:28:46,419 | No environment variables found for function 'SSLERRORtestlambda'
2021-03-05 11:28:46,419 | Environment variables overrides data is standard format
2021-03-05 11:28:46,419 | Loading AWS credentials from session with profile 'default'
2021-03-05 11:28:46,434 | Resolving code path. Cwd=/Users/garciaja/test, CodeUri=.
2021-03-05 11:28:46,434 | Resolved absolute path to code is /Users/garciaja/test
2021-03-05 11:28:46,434 | Code /Users/garciaja/test is not a zip/jar file
2021-03-05 11:28:46,481 | Skip pulling image and use local one: amazon/aws-sam-cli-emulation-image-python3.7:rapid-1.18.1.

2021-03-05 11:28:46,481 | Mounting /Users/garciaja/test as /var/task:ro,delegated inside runtime container
2021-03-05 11:28:46,882 | Starting a timer for 50 seconds for function 'SSLERRORtestlambda'
START RequestId: 7d03e5c3-e091-4c12-9eab-dcb95f81454d Version: $LATEST
[ERROR] SSLError: SSL validation failed for https://eu-west-1.elasticmapreduce.amazonaws.com/ [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:109    raise SSLError(endpoint_url=request.url, error=e)81, in send_responset_exception
END RequestId: 7d03e5c3-e091-4c12-9eab-dcb95f81454d
REPORT RequestId: 7d03e5c3-e091-4c12-9eab-dcb95f81454d	Init Duration: 0.38 ms	Duration: 13348.20 ms	Billed Duration: 13400 ms	Memory Size: 128 MB	Max Memory Used: 128 MB	
2021-03-05 11:29:01,219 | Cleaning all decompressed code dirs
2021-03-05 11:29:01,220 | Sending Telemetry: {'metrics': [{'commandRun': {'requestId': 'deb702e2-0848-44c9-9a1a-d72f83249717', 'installationId': 'e0400e43-63bf-46e8-b442-9a9d208ad761', 'sessionId': '32f4beef-ba1d-40d3-9546-ad8081421279', 'executionEnvironment': 'CLI', 'pyversion': '3.7.10', 'samcliVersion': '1.18.1', 'awsProfileProvided': True, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam local invoke', 'duration': 14987, 'exitReason': 'success', 'exitCode': 0}}]}
2021-03-05 11:29:01,944 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)
{"errorMessage": "SSL validation failed for https://eu-west-1.elasticmapreduce.amazonaws.com/ [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1091)", "errorType": "SSLError", "stackTrace": ["  File \"/var/task/test.py\", line 4, in list_clusters\n    clusters = boto3.client('emr').list_clusters()\n", "  File \"/var/runtime/botocore/client.py\", line 357, in _api_call\n    return self._make_api_call(operation_name, kwargs)\n", "  File \"/var/runtime/botocore/client.py\", line 663, in _make_api_call\n    operation_model, request_dict, request_context)\n", "  File \"/var/runtime/botocore/client.py\", line 682, in _make_request\n    return self._endpoint.make_request(operation_model, request_dict)\n", "  File \"/var/runtime/botocore/endpoint.py\", line 102, in make_request\n    return self._send_request(request_dict, operation_model)\n", "  File \"/var/runtime/botocore/endpoint.py\", line 137, in _send_request\n    success_response, exception):\n", "  File \"/var/runtime/botocore/endpoint.py\", line 256, in _needs_retry\n    caught_exception=caught_exception, request_dict=request_dict)\n", "  File \"/var/runtime/botocore/hooks.py\", line 356, in emit\n    return self._emitter.emit(aliased_event_name, **kwargs)\n", "  File \"/var/runtime/botocore/hooks.py\", line 228, in emit\n    return self._emit(event_name, kwargs)\n", "  File \"/var/runtime/botocore/hooks.py\", line 211, in _emit\n    response = handler(**kwargs)\n", "  File \"/var/runtime/botocore/retryhandler.py\", line 183, in __call__\n    if self._checker(attempts, response, caught_exception):\n", "  File \"/var/runtime/botocore/retryhandler.py\", line 251, in __call__\n    caught_exception)\n", "  File \"/var/runtime/botocore/retryhandler.py\", line 277, in _should_retry\n    return self._checker(attempt_number, response, caught_exception)\n", "  File \"/var/runtime/botocore/retryhandler.py\", line 317, in __call__\n    caught_exception)\n", "  File \"/var/runtime/botocore/retryhandler.py\", line 223, in __call__\n    attempt_number, caught_exception)\n", "  File \"/var/runtime/botocore/retryhandler.py\", line 359, in _check_caught_exception\n    raise caught_exception\n", "  File \"/var/runtime/botocore/endpoint.py\", line 200, in _do_get_response\n    http_response = self._send(request)\n", "  File \"/var/runtime/botocore/endpoint.py\", line 269, in _send\n    return self.http_session.send(request)\n", "  File \"/var/runtime/botocore/httpsession.py\", line 281, in send\n    raise SSLError(endpoint_url=request.url, error=e)\n"]}2021-03-05 11:29:01,944 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2021-03-05 11:29:01,945 | Sending Telemetry: {'metrics': [{'runtimeMetric': {'requestId': 'ced7002a-44cc-485f-b554-cf34024021f6', 'installationId': 'e0400e43-63bf-46e8-b442-9a9d208ad761', 'sessionId': '32f4beef-ba1d-40d3-9546-ad8081421279', 'executionEnvironment': 'CLI', 'pyversion': '3.7.10', 'samcliVersion': '1.18.1', 'runtimes': ['python3.7']}}]}
2021-03-05 11:29:03,670 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)

Expected result:

list of emr clusters

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: macOS 10.15.7 (19H2)
2. sam --version: SAM CLI, version 1.18.1
3. AWS region: eu-west-1

[driverpt]

After some investigation, we found out that this happens because sam local invoke completely ignores any ENV Variables other than the ones declared in the template file.

If you place the Custom CA Bundle e.g.: certificate.pem in the package and set AWS_CA_BUNDLE: /var/task/certificate.pem in the CloudFormation SAM Template it will actually work.

I'd say AWS needs to add a mechanism to read and add the Certificate to the dynamically generated Dockerfile here
e.g.:

ADD <$HOME/.aws/credentials:ca_bundle> /ca-certs/certificate.pem

And set the AWS_CA_BUNDLE=/ca-certs/certificate.pem inside the Container

[driverpt]

Can we have any ETA on this issue ?

[elbayaaa]

@driverpt what is the problem of setting the AWS_CA_BUNDLE as a lambda environment variable? Also, will it work if you set it up in ~/.aws/config file?

[driverpt]

Problem is that we need to have the proper Root CA on every single package and set it (env variable).

This works well if you have a couple of lambdas, if you have dozens it becomes annoying.

SAM ignores the CA bundle in aws config.

So SAM Local Invoke needs some sort of --ca-bundle flag to allow overriding.

[elbayaaa]

@driverpt, Just to clarify; You want to provide the environment variable AWS_CA_BUNDLE during the runtime instead of during the creation of lambda function. Something like

export AWS_CA_BUNDLE=/path/to/certificate.pem
sam local invoke MyFunction

right?

Also, is your problem scoped to container images lambda functions only? hence you are suggesting adding to the generated Dockerfile ?

[driverpt]

This issue occurs when we are running sam local invoke, hence running locally in Container Image.

We need a Custom Root CA for all outgoing web communications as required by our corporate proxy (even inside the container).

[elbayaaa]

@driverpt I think the problem with what you are trying to achieve is that sam local invoke should behave the same way like the lambda functions executed inside the AWS account. So, in your proposal you are defining an environment variable

And set the AWS_CA_BUNDLE=/ca-certs/certificate.pem inside the Container

while AWS lambda accepts this variables only through the configuration (whether from the console or the "Environment" CloudFormationattribute).

Similarly, If you want to use your CA, you need to bundle it with the lambda artifacts, right?

Anyway, I came up with the following workaround, please let me know if it helps:

1. In your template add a placeholder for the AWS_CA_BUNDLE environment variable in the Globals section so that you don't need to duplicate it in each function

Globals:
 Function:
   Environment:
     Variables:
       AWS_CA_BUNDLE:
         Ref: AWS::NoValue # No value so as to be ignored. During local invoke this is overridden via the env-vars option

2. run sam build
3. copy your certificate to each function's build artifacts
   ls -d .aws-sam/build/*/ | xargs -n 1 cp /path/to/my-cacert.pem
4. create the following env.json file

{
  "Parameters": {
    "AWS_CA_BUNDLE": "my-cacert.pem"
  }
}

5. run sam local invoke MyFunction --env-vars env.json

[driverpt]

Again... We already have that work around in place, this works well if you have a couple of lambdas, if you have dozens, it's annoying.

Why not just add the parameter to sam local invoke --ca-bundle?

Locally it's a development environment, I should have control over the development environment.

In this case our corporate proxy requires us a custom CA since it uses SSL Certificate Pinning to secure the connections.

So we would like to please have the option of passing the custom CA Cert as a flag or use the one in local profile .aws/config

[elbayaaa]

We are evaluating your use-case. The main concern with your proposal is that it will make local invoke behaves differently than AWS lambda. For example; if your actual lambda function running on AWS needs to use a custom certificate, it won't neither use this proposed --ca-bundle option nor .aws/config, right?

A side note; The provided workaround scales well with any number of lambdas, it doesn't need any per-function changes, all of the changes are global.