Testing Api Gateway endpoint locally returning `No response from invoke container` and `Invalid lambda response received` in Dockerised sam app

[alliesground]

Description:

I'm dockerising sam app. When accessing api endpoint for HelloWorld locally, it responds with No response from invoke container for HelloWorldFunction and Invalid lambda response received: Lambda response must be valid json.

It works fine whithout dockerisation.

Steps to reproduce:

Dockerfile

FROM ruby:2.7.0-alpine

RUN apk add --update --no-cache \
    build-base \
    postgresql-dev \
    postgresql-client \
    python3 \
    py3-pip \
    util-linux \
    python3-dev

RUN pip3 install aws-sam-cli

ARG USER
ARG HOME
ARG UID

RUN apk add --update \
    sudo

RUN echo "Welcome home: $USER => $UID"

RUN adduser -S -D -G users -u $UID $USER
RUN addgroup -S sudo
RUN echo "%sudo ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/sudo 
RUN adduser $USER sudo

RUN echo "Welcome home: $USER"

WORKDIR ${HOME}

EXPOSE 3001

ENTRYPOINT ["sh", "./entrypoint.sh"]

docker-compose.yml

version: '3.8'
services:
  sam_app:
    build:
      context: ./sam-app
      args:
        - HOME
        - USER
        - UID
    user: "${UID}:100"
    command: ["$PWD"]
    ports:
      - "3001:3001"
    volumes:
      - ./sam-app:$HOME
      - /var/run/docker.sock:/var/run/docker.sock

Entrypoint.sh

BASEDIR="$1"

echo "Basedir => ${BASEDIR}"

sudo sam local start-api \
  --template ./template.yaml \
  --host 0.0.0.0 \
  --port 3001 \
  --docker-volume-basedir "${BASEDIR}/sam-app/" \
  --docker-network drink_default \
  --debug

hello_world/app.rb

def lambda_handler(event:, context:)
  {
    statusCode: 200,
    body: {
      message: "Hello World!",
      # location: response.body
    }.to_json
  }
end

Observed result:

am_app_1  | 2020-12-21 01:31:22,493 | Constructed String representation of Event to invoke Lambda. Event: {"body": null, "headers": {"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9", "Accept-Encoding": "gzip, deflate, br", "Accept-Language": "en-GB,en;q=0.9", "Connection": "keep-alive", "Host": "localhost:3001", "Sec-Fetch-Dest": "document", "Sec-Fetch-Mode": "navigate", "Sec-Fetch-Site": "none", "Sec-Fetch-User": "?1", "Upgrade-Insecure-Requests": "1", "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Safari/537.36", "X-Forwarded-Port": "3001", "X-Forwarded-Proto": "http"}, "httpMethod": "GET", "isBase64Encoded": false, "multiValueHeaders": {"Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"], "Accept-Encoding": ["gzip, deflate, br"], "Accept-Language": ["en-GB,en;q=0.9"], "Connection": ["keep-alive"], "Host": ["localhost:3001"], "Sec-Fetch-Dest": ["document"], "Sec-Fetch-Mode": ["navigate"], "Sec-Fetch-Site": ["none"], "Sec-Fetch-User": ["?1"], "Upgrade-Insecure-Requests": ["1"], "User-Agent": ["Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Safari/537.36"], "X-Forwarded-Port": ["3001"], "X-Forwarded-Proto": ["http"]}, "multiValueQueryStringParameters": null, "path": "/hello", "pathParameters": null, "queryStringParameters": null, "requestContext": {"accountId": "123456789012", "apiId": "1234567890", "domainName": "localhost:3001", "extendedRequestId": null, "httpMethod": "GET", "identity": {"accountId": null, "apiKey": null, "caller": null, "cognitoAuthenticationProvider": null, "cognitoAuthenticationType": null, "cognitoIdentityPoolId": null, "sourceIp": "172.22.0.1", "user": null, "userAgent": "Custom User Agent String", "userArn": null}, "path": "/hello", "protocol": "HTTP/1.1", "requestId": "dd99d1bb-cc40-42f8-a2f2-eeefcc22d111", "requestTime": "21/Dec/2020:01:30:38 +0000", "requestTimeEpoch": 1608514238, "resourceId": "123456", "resourcePath": "/hello", "stage": "Prod"}, "resource": "/hello", "stageVariables": null, "version": "1.0"}
sam_app_1  | 2020-12-21 01:31:22,493 | Found one Lambda function with name 'HelloWorldFunction'
sam_app_1  | 2020-12-21 01:31:22,494 | Invoking app.lambda_handler (ruby2.7)
sam_app_1  | 2020-12-21 01:31:22,494 | No environment variables found for function 'HelloWorldFunction'
sam_app_1  | 2020-12-21 01:31:22,494 | Environment variables overrides data is standard format
sam_app_1  | 2020-12-21 01:31:22,494 | Loading AWS credentials from session with profile 'None'
sam_app_1  | 2020-12-21 01:31:24,549 | Resolving code path. Cwd=/home/sameer/projects/drink/sam-app/, CodeUri=hello_world/
sam_app_1  | 2020-12-21 01:31:24,549 | Resolved absolute path to code is /home/sameer/projects/drink/sam-app/hello_world
sam_app_1  | 2020-12-21 01:31:24,550 | Code /home/sameer/projects/drink/sam-app/hello_world is not a zip/jar file
sam_app_1  | 2020-12-21 01:31:24,574 | Skip pulling image and use local one: amazon/aws-sam-cli-emulation-image-ruby2.7:rapid-1.15.0.
sam_app_1  | 
sam_app_1  | 2020-12-21 01:31:24,574 | Mounting /home/sameer/projects/drink/sam-app/hello_world as /var/task:ro,delegated inside runtime container
sam_app_1  | 2020-12-21 01:31:25,386 | Starting a timer for 3 seconds for function 'HelloWorldFunction'
sam_app_1  | 2020-12-21 01:31:26,707 | Cleaning all decompressed code dirs
sam_app_1  | 2020-12-21 01:31:26,707 | No response from invoke container for HelloWorldFunction
sam_app_1  | 2020-12-21 01:31:26,707 | Invalid lambda response received: Lambda response must be valid json
sam_app_1  | 2020-12-21 01:31:26 172.22.0.1 - - [21/Dec/2020 01:31:26] "GET /hello HTTP/1.1" 502 -
sam_app_1  | 2020-12-21 01:31:26 172.22.0.1 - - [21/Dec/2020 01:31:26] "GET /favicon.ico HTTP/1.1" 403 -

Expected result:

Json output on browser {message: "hello world"}

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: ubuntu VERSION="20.04.1 LTS (Focal Fossa)"
2. sam --version: 1.15.0
3. Homebrew: VERSION="2.6.2"
4. docker: VERSION="19.03.8"

Add --debug flag to command you are running

[jfuss]

@alliesground SAM CLI is meant to be run from your machine and interact with docker, not run SAM CLI within Docker. My suggestion here is to do that instead of Dockerize.

What's the use case on wanting to do this? Why do you need to be running docker within docker?

[rbliss]

I also get this exact error. I will chime in on my use case which may or may not be similar:

In my case, I'm using VSCode's Dev Containers, to create an isolated dev environment SAM is running in. My setup is nearly identical, using a volume to expose the host's docker socket, and SAM's --docker-volume-basedir flag to direct SAM to the appropriate location. Hitting the endpoint gives me the same No response from invoke container for HelloWorldLambda

Unfortunately, SAM's --debug flag doesn't go into much detail about what went on with the lambda container it spins up to serve the request. I started tracing through the CLI source to see what happened, but haven't had time to finish my search.

[alliesground]

@jfuss my use case is similar to that of @rbliss i.e to create an isolated dev environment, which includes react-app and db, all composed with docker-compose.

[alayor]

I'm also having the same problem. My use case is exactly the same as @alliesground.

I followed the instructions on this post. It makes me think this used to work before.
https://medium.com/monsoon-engineering/running-aws-sam-in-a-docker-container-2491596672c2

[gpomykala]

I receive same error when trying to hit a lambda spun with 'sam local start-lambda --docker-network host'. It used to work with older version of SAM CLI. I am not doing anything extraordinary, just a basic integration test of a lambda client hitting lambda hosted in SAM CLI

[jfuss]

@gpomykala Your issue is logged here: #2436 (comment) and workarounds here: #2436 (comment)

@rbliss @alayor @alliesground
We have seen this happen when SAM CLI cannot communicate to the container. Instead of exec-ing within the container, we updated to move to publishing a port and then calling into the container of http. This could be the cause here as well. SAM CLI will communicate to the container over localhost only, so there is probably some docker network trickery going on. I highly suggest to not use SAM CLI with the container like this. It's not really a use case we directly support or recommend.

[alayor]

@jfuss Thank you for your response and links.
I wonder if there could be a way to change the "communicate to the container over localhost" setting.
I just wanted to test if changing "localhost" to "docker.for.mac.localhost" would make it work.
I know this is not recommended, but I figured I could ask anyway.

[jfuss]

@alayor I am open to ways to make this work if we can find a solution that works across OS. I am not sure if moving to the docker.for.mac.localhost (or host.docker.internal which I think might work across OS).

Just mapping this out:

• SAM runs in Container A on a bridge docker network (by default).
• Executing a sam local command will start up a container and publish ports to localhost (not sure if this is localhost of the machine or on the bridge docker network).
• SAM attempts to call localhost: to communicate to the container and fails to get a response.

If in step two, this does publish a port on the machines localhost, then I would assume what you are suggesting could work. Super hacky and seems much easier to just run the CLI on your machine and not have to care about docker at all. Could be a personal preference though.

[alliesground]

@jfuss thanks for your response. It was just an itch of mine to try to dockerize sam-cli especially after reading some articles on the possibilities. Anyways I will move forward with your advice and run CLI on the host machine and maybe try to connect it with other tech stacks running in the container.

[ilyasotkov]

I was able to get SAM local invoke to work inside Dev Container by replacing hardcoded localhost here

https://github.com/aws/aws-sam-cli/blob/develop/samcli/local/docker/container.py#L41

with host.docker.internal. I used this guide for attaching the VS Code debugger: https://github.com/aws/aws-toolkit-vscode/blob/a380685696e39247d6ec0f8f1e9928ba2562f550/docs/debugging-python-lambda-functions.md, also replacing localhost with host.docker.internal in launch.json.

@jfuss I'm running this on macOS, AFAIK this won't work on linux out of the box due to docker/for-linux#264, but a SAM CLI option to specify host would be nice.

[jasonterando]

If in step two, this does publish a port on the machines localhost, then I would assume what you are suggesting could work. Super hacky and seems much easier to just run the CLI on your machine and not have to care about docker at all. Could be a personal preference though.

Given that SAM CLI is launching a container to run SAM template, would it make sense to consider documenting how to launch the image via Docker Compose (or similar tool)?

Probably ought to explain why some of us want to run SAM under Docker (regardless if it's a good idea, possible, etc.) One of the nice things about developing with Docker is that it helps to avoid workstation bloat and conflicts by, for example, having to install yet another package manager (i.e. Homebrew); which, in turn, steps on our environment's default Python version. This is not to pick on Homebrew, it's probably great; although like many tools of its ilk it is much easier to install than to remove when you no longer need it (like having to manually edit ~/bashrc). At the very least, let Python developers know in the docs there's the option of running pip3 install aws-sam-cli.

On a side note, I tried ilyasotkov's workaround/hack in Linux and it didn't work for me. For anybody interested, here's the Dockerfile I set up:

FROM python:alpine
RUN apk update && \
    apk upgrade && \
    apk add bash && \
    apk add --no-cache --virtual build-deps build-base gcc && \
    pip install aws-sam-cli && \
    apk del build-deps

# Replace localhost with host.docker.internal in AWS SAM
RUN sed -i 's/localhost/host.docker.internal/g' /usr/local/lib/python3.9/site-packages/samcli/local/docker/container.py

RUN mkdir /app
WORKDIR /app
EXPOSE 3000
COPY ./.docker/entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]

And the entrypoint.sh...

#!/bin/bash

# Trick to get host.docker.internal working on Linux Docker
# From https://dev.to/bufferings/access-host-from-a-docker-container-4099
HOST_DOMAIN="host.docker.internal"
ping -q -c1 $HOST_DOMAIN > /dev/null 2>&1
if [ $? -ne 0 ]; then
    HOST_IP=$(route | awk 'FNR==3 {print $2}')
    echo -e "$HOST_IP\t$HOST_DOMAIN" >> /etc/hosts
fi

set -o errexit
cd /app
/usr/local/bin/sam local start-api \
  --template ./template.yaml \
  --host 0.0.0.0 \
  --docker-volume-basedir /app \
  --docker-network my-demo-app_local  \
  --skip-pull-image

[xiaoxinghu]

Another common use case for dockerizing sam: CI pipelines. Like @jasonterando said, it's a pain to maintain packages/tools installed on build agents. Different teams need their tools in particular ways. It's more like dependencies for your projects. The tools break over time, so you need to keep your setup predictable. That's what docker provides. If it's "not a good idea" to use sam within docker, then I run out of ideas to make my CI pipeline stable over long period of time. I can't trust my scripts/code to still work 1 year from now, after countless patching/upgrading of the build agents. Not to mention that normally it's not easy to tweak infrastructure setups within big corporates, it's a long and risky process. Sometimes impossible (e.g. they refuse to install arbitrary software on agents, because they gave you docker already).

[ilyasotkov]

I highly suggest to not use SAM CLI with the container like this. It's not really a use case we directly support or recommend.

@jfuss Doesn't the presence of -v, --docker-volume-basedir TEXT option for sam local invoke contradict with that statement? The docs for it say

If Docker is running on a remote machine, you must mount the path where the AWS SAM file exists on the Docker machine, and modify this value to match the remote machine.

So remote Docker hosts are already partially supported.