[ecr] CDK DockerBundling should use the underlying default image from ECR

[srinivasreddych]

As per the latest DockerHub rate limit release, the activity of running cdk bundling is causing us rate limit issues. It would be great if CDK would switch the default dockerimage from “amazon/aws-sam-cli-build-image-” https://hub.docker.com/u/amazon to an image hosted in AWS ECR.

Use Case

Our build systems run cdk deploy which will bundle assets for lambda functions. For bundling the assets, cdk would pull amazon/aws-sam-cli-build-image and perform the activity. It would be great to have CDK rely on an image hosted outside of DockerHub.

Error seen

Unable to find image 'amazon/aws-sam-cli-build-image-python3.6:latest' locally
docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.

[AndyEfaa]

I can confirm the issue. This is causing all of our pipelines to fail.
In our case, the image amazon/aws-sam-cli-build-image-python3.7 is build in the context of PythonFunction(https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-lambda-python.PythonFunction.html).

[TheRealAmazonKendra]

My team is also having this issue with the image amazon/aws-sam-cli-build-image-nodejs12.x. It is currently causing our CI builds to fail.

[koups496]

My pipeline is also failing on amazon/aws-sam-cli-build-image-python3.7 for the same reason

I can confirm the issue. This is causing all of our pipelines to fail.
In our case, the image amazon/aws-sam-cli-build-image-python3.7 is build in the context of PythonFunction(https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-lambda-python.PythonFunction.html).

[erudisch]

We have the same issue with PythonFunction

A way to set the docker login would be great to use the images hosted on ecr.

see: #11544

[AndyEfaa]

Assuming that you build your CDK apps in a pipeline, the issue can be fixed by including a docker login in the buildspec.yaml file as follows:

version: 0.2 
env:
  secrets-manager:
    DOCKERHUB_USERNAME: "dockerhub-access-token:dockerUser"
    DOCKERHUB_ACCESS_TOKEN: "dockerhub-access-token:dockerAccessToken"
phases: 
  install: 
    runtime-versions: 
        python: 3.7 
    commands:
      - echo "Start Build..."
  build: 
    commands: 
      - nohup /usr/bin/dockerd --host=unix:///var/run/docker.sock --host=tcp://127.0.0.1:2375 --storage-driver=overlay2 &
      - timeout 15 sh -c "until docker info; do echo .; sleep 1; done"
      - docker login -u ${DOCKERHUB_USERNAME} -p ${DOCKERHUB_ACCESS_TOKEN}
      - echo "Do rest of cdk build..."
  post_build:
    commands:
      - echo "Do post build..."
artifacts: 
  files: '**/*'

Works great for us.

Thx to @skinny85 for the hint.

[nbaillie]

With the release of Amazon ECR Public Gallery
could it be moved to use : public.ecr.aws/amazonlinux/amazonlinux:latest

Im guessing however aws-sam-cli-build* might also move there at some point.

[srinivasreddych]

Hello @MrArnoldPalmer Any estimate timeline around when CDK would start leveraging base images from AWS Public ECR? For now, we are still stuck in doing docker login -u username -p pwd. Thanks!

[MrArnoldPalmer]

@srinivasreddych no estimate on this right now, though I understand the pain this is causing as pretty much every developer using docker as started running into these new rate limits. docker login is the workaround I recommend for now.

I need to look around a bit for other usage of dockerhub images in the cdk codebase that affect user's synth/deploy. I'll provide an update when I'm able to do that.

[outdoorpet]

We are running into this issue using CDK Python lambda asset bundling as described here. It isn't clear to me how I can fix this with supplying docker credentials as a work around...

[zxkane]

Same here. It’s better moving the image to https://gallery.ecr.aws/ .

[mrpackethead]

Adding more weight to this one, its clobbering my build process repeatably. Using Python Lambdas. I can'te see how to fix this either.

[mrpackethead]

So, here is what we did to work around this annoying issue..

We took a copy of build image and placed it in our own ECR. Then created a bundling image.. Here is an example of how you might be able to work around it.

from aws_cdk import(
    aws_lambda,
    aws_iam as iam,
    core
)

import os

ECR_REPO_HOSTNAME = "<youraccount>.dkr.ecr.<yourregion>.amazonaws.com"     # change as appropriate. 
ECR_REPO_NAME = "amazon/aws-sam-cli-build-image-python3.8"
IMAGE_TAG = "latest"
REGION = 'ap-southeast-2'

def PyLambdaBundlingImage():

    image_url = f"{ECR_REPO_HOSTNAME}/{ECR_REPO_NAME}:{IMAGE_TAG}"

    # if this is part of a code build process leave this commented, other wise if its from a remote machine, you'll need a profile
    #profile = 'contact-cloud-and-infra'                                                                  
    #get_login_password = (f"aws ecr get-login-password --region {REGION} --profile {profile}")

    get_login_password = (f"aws ecr get-login-password --region {REGION}")
    docker_login_from_stdin = (f"docker login --username AWS --password-stdin {ECR_REPO_HOSTNAME}")
    docker_login = " | ".join([get_login_password, docker_login_from_stdin])
    docker_pull = f"docker pull {image_url}"

    # fetch image
    os.system(" && ".join([docker_login, docker_pull]))
    print("Docker tasks done, returning to CDK")


    # https://docs.aws.amazon.com/cdk/api/latest/python/aws_cdk.core/BundlingOptions.html#aws_cdk.core.BundlingOptions
    bundling_image = core.BundlingOptions(
        image=core.BundlingDockerImage.from_registry(image_url),
        command=[
            "/bin/bash",
            "-c",
            "&&".join(
                [
                    " ".join(
                            [
                                "if [ -e /asset-input/requirements.txt ]; then",
                                "  pip install --target /asset-output -r /asset-input/requirements.txt;",
                                "fi",
                            ]
                        ),
                    "cp /asset-input/* /asset-output",
                ]
            ),
        ],
    )
    
    return(bundling_image)


class LambdaFunctions(core.Construct):
  
    def __init__(self, scope: core.Construct, id: str, project_name: str, region, account, repo_account, *, prefix=None):
        super().__init__(scope, id)

        
        self.ssmlookuplambda = aws_lambda.Function(self, "LocalSSM",
            function_name=f'{project_name}-Prereq-SSMLookup',
            handler="getssmstack.on_event",                     # file, and handler.
            runtime=aws_lambda.Runtime(
                name="python3.8"  
            ),
            code=aws_lambda.Code.from_asset(
                path="application/lambda_fn/getssmstack",         #path 
                asset_hash_type=core.AssetHashType.SOURCE,
                bundling = PyLambdaBundlingImage()
            ),
        )
        
        self.ssmlookuplambda.add_to_role_policy(
            iam.PolicyStatement(
                effect=iam.Effect.ALLOW,
                actions=[
                    'ssm:DescribeParameters',
                    'ssm:GetParameters'
                ],
                resources=['*']   
            )
        )

[guydelta]

I had a similiar issue on the nodejs side of things, and hopefully this might help to point someone in a direction that might help.

Both my NodejsFunction constructs and LayerVersion constructs for our cdk definitions utilized the docker library's amazon/aws-sam-cli-build-image-nodejs12.x:latest build images. And very quickly I ran into the rate limiting issue from docker again...

Being "cheap"... I mean wanting to save our company money I didn't want to create a new docker account and go through all that hassle, rather I utilized the aws ecr library as follow:

within cdk for nodejs by default I call the lambda construct as follow:

this.handler = new aws_lambda_nodejs.NodejsFunction(this, id + 'Fn', {
	bundling: {
		externalModules: [...],
	},
	layers: [resources.layer],
	...
});

and the layer is constructed in a different stack in a similiar fashion.
This utilizes the default docker library...

But changing it as follow resolved it for me:

this.handler = new aws_lambda_nodejs.NodejsFunction(this, id + 'Fn', {
	bundling: {
		externalModules: [...],
		dockerImage: BundlingDockerImage.fromRegistry('public.ecr.aws/r2l9o2s7/aws-sam-cli-build-image-nodejs12.x:latest')
	},
	layers: [resources.layer],
	...
});

[srinivasreddych]

I see the sam image used by CDK is made available within Public ECR.
Reference: https://gallery.ecr.aws/v5z1s6z1/aws-sam-cli-build-image-python3.7

Any idea if we can start leveraging it? @MrArnoldPalmer

[pgarbe]

This is my workaround: I define a new runtime and set the bundlingDockerImage to the one in ECR gallery.

    const pythonRuntime = new lambda.Runtime('python3.7', lambda.RuntimeFamily.PYTHON, {
      supportsInlineCode: true,
      supportsCodeGuruProfiling: true,
      bundlingDockerImage: 'public.ecr.aws/sam/build-python3.7',
    });

    const func = new pythonlambda.PythonFunction(this, 'lambda', {
      runtime: pythonRuntime, // Still uses 
      ...
    }

Just the output is a bit weird as it also shows the default value for the IMAGE argument

Step 1/9 : ARG IMAGE=amazon/aws-sam-cli-build-image-python3.7
Step 2/9 : FROM $IMAGE
latest: Pulling from sam/build-python3.7

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[stowns]

This is my workaround: I define a new runtime and set the bundlingDockerImage to the one in ECR gallery.

    const pythonRuntime = new lambda.Runtime('python3.7', lambda.RuntimeFamily.PYTHON, {
      supportsInlineCode: true,
      supportsCodeGuruProfiling: true,
      bundlingDockerImage: 'public.ecr.aws/sam/build-python3.7',
    });

    const func = new pythonlambda.PythonFunction(this, 'lambda', {
      runtime: pythonRuntime, // Still uses 
      ...
    }

Just the output is a bit weird as it also shows the default value for the IMAGE argument

Step 1/9 : ARG IMAGE=amazon/aws-sam-cli-build-image-python3.7
Step 2/9 : FROM $IMAGE
latest: Pulling from sam/build-python3.7

For whatever reason this doesn't seem to work, at least consistently in 1.74.0. My CdkPipeline randomly fails due to rate-limiting of the python amazon/aws-sam-cli-build-image-python3.7 image. If i retry 1 or 2 times it tends to work. My output when it fails loooks like

stderr: Sending build context to Docker daemon  6.703kB
Step 1/4 : ARG IMAGE=amazon/aws-sam-cli-build-image-python3.7
Step 2/4 : FROM $IMAGE
toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
Subprocess exited with error 1
[Container] 2021/05/05 02:24:17 Command did not exit successfully npx cdk synth exit status 1
[Container] 2021/05/05 02:24:17 Phase complete: BUILD State: FAILED
[Container] 2021/05/05 02:24:17 Phase context status code: COMMAND_EXECUTION_ERROR Message: Error while executing command: npx cdk synth. Reason: exit status 1

my Fn implementation...

new PythonFunction(this, 'MyFn', {
  entry: path.join(__dirname, 'lambda', 'fns', 'my_fn'),
  runtime: new Runtime('python3.7', RuntimeFamily.PYTHON, {
    bundlingDockerImage: 'public.ecr.aws/sam/build-python3.7',
    supportsInlineCode: true
  }),
  role: myRole
})

[pgarbe]

Can you update to a newer version? This has already been fixed.