(cdk-pipelines): Docker rate limit, guidance needed or auth mechanism

[arpowers]

Currently can't deploy because of a rate limiting issue with Docker. You may have seen this in other issues, but the problem here is I have not hit this limit myself (only this single deploy) so for some reason, I believe Docker is flagging all of Codebuild as one "anonymous" source.

Anyway, we need a way to authenticate when using DockerImageAsset from @aws-cdk/aws-ecr-assets or other guidance on how to circumvent this

Error message:

Step 1/9 : FROM node:14-alpine AS build
--
40 | toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
41 | error  : [100%] fail: docker build --tag cdkasset-d0160342c2de47434404b7f88eaefd92e63e83f7178db2ff9a88cc4e0b62f5b2 --file ./docker/ci.Dockerfile . exited with error code 1: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
42 | Failure: Error: docker build --tag cdkasset-d0160342c2de47434404b7f88eaefd92e63e83f7178db2ff9a88cc4e0b62f5b2 --file ./docker/ci.Dockerfile . exited with error code 1: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

Proposed Solution

We need a way to authenticate to associate our accounts with the pull instead of anon.

[erudisch]

maybe this is helpful:

#10999

[arpowers]

@rix0rrr I know you guys are probably busy with the conference this week, but I can't deploy code due to this problem...

From the output, looks like you are logging in with an AWS user and that is getting rate limited. Not just for me but for everyone.

here's further info:

CDK Code:

/**
     * Build an image with the entire monorepo to work with
     */
    const dockerImage = new DockerImageAsset(this, baseId("DockerImage"), {
      directory: ".",
      file: `./docker/ci.Dockerfile`,
    })

    const image = ecs.ContainerImage.fromDockerImageAsset(dockerImage)

Terminal / Codebuild Output:

[Container] 2020/11/30 17:43:08 Running command cdk-assets --path "assembly-PipelineStack-Pre/PipelineStackPreService574433AC.assets.json" --verbose publish "c1dd1c8474c338cb046b5097f5c51b8d8b00b1c1c54618aa198354cebe8d7a36:433775104113-us-east-1"
--
28 | verbose: Loaded manifest from assembly-PipelineStack-Pre/PipelineStackPreService574433AC.assets.json: 2 assets found
29 | verbose: Applied selection: 1 assets selected.
30 | info   : [0%] start: Publishing c1dd1c8474c338cb046b5097f5c51b8d8b00b1c1c54618aa198354cebe8d7a36:433775104113-us-east-1
31 | verbose: Assume arn:aws:iam::433775104113:role/cdk-hnb659fds-image-publishing-role-433775104113-us-east-1
32 | verbose: [0%] check: Check 433775104113.dkr.ecr.us-east-1.amazonaws.com/cdk-hnb659fds-container-assets-433775104113-us-east-1:c1dd1c8474c338cb046b5097f5c51b8d8b00b1c1c54618aa198354cebe8d7a36
33 | verbose: [0%] debug: docker login --username AWS --password-stdin https://433775104113.dkr.ecr.us-east-1.amazonaws.com
34 | verbose: [0%] debug: docker inspect cdkasset-c1dd1c8474c338cb046b5097f5c51b8d8b00b1c1c54618aa198354cebe8d7a36
35 | verbose: [0%] build: Building Docker image at /codebuild/output/src177682768/src/asset.c1dd1c8474c338cb046b5097f5c51b8d8b00b1c1c54618aa198354cebe8d7a36
36 | verbose: [0%] debug: docker build --tag cdkasset-c1dd1c8474c338cb046b5097f5c51b8d8b00b1c1c54618aa198354cebe8d7a36 --file ./docker/ci.Dockerfile .
37 | Sending build context to Docker daemon  1.198MB
38 |  
39 | Step 1/9 : FROM node:14-alpine AS build
40 | toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
41 | error  : [100%] fail: docker build --tag cdkasset-c1dd1c8474c338cb046b5097f5c51b8d8b00b1c1c54618aa198354cebe8d7a36 --file ./docker/ci.Dockerfile . exited with error code 1: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
42 | Failure: Error: docker build --tag cdkasset-c1dd1c8474c338cb046b5097f5c51b8d8b00b1c1c54618aa198354cebe8d7a36 --file ./docker/ci.Dockerfile . exited with error code 1: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
43 | at ChildProcess.<anonymous> (/usr/local/lib/node_modules/cdk-assets/lib/private/shell.js:46:24)
44 | at Object.onceWrapper (events.js:422:26)
45 | at ChildProcess.emit (events.js:315:20)
46 | at ChildProcess.EventEmitter.emit (domain.js:482:12)
47 | at maybeClose (internal/child_process.js:1021:16)
48 | at Process.ChildProcess._handle.onexit (internal/child_process.js:286:5)
49 |  
50 | [Container] 2020/11/30 17:43:09 Command did not exit successfully cdk-assets --path "assembly-PipelineStack-Pre/PipelineStackPreService574433AC.assets.json" --verbose publish "c1dd1c8474c338cb046b5097f5c51b8d8b00b1c1c54618aa198354cebe8d7a36:433775104113-us-east-1" exit status 1
51 | [Container] 2020/11/30 17:43:09 Phase complete: BUILD State: FAILED
52 | [Container] 2020/11/30 17:43:09 Phase context status code: COMMAND_EXECUTION_ERROR Message: Error while executing command: cdk-assets --path "assembly-PipelineStack-Pre/PipelineStackPreService574433AC.assets.json" --verbose publish "c1dd1c8474c338cb046b5097f5c51b8d8b00b1c1c54618aa198354cebe8d7a36:433775104113-us-east-1". Reason: exit status 1
53 | [Container] 2020/11/30 17:43:09 Entering phase POST_BUILD

[arpowers]

again today

[JorisLimousinKaizen]

Same problem for us, this is a big issue.

[straygar]

@JorisLimousinKaizen @arkon Not a fix, but have you tried the workaround in the thread @erudisch mentioned?

#10999 (comment)

[JorisLimousinKaizen]

Haven't tried it no, it has been working again since a few days, not sure if it has been fixed though.

EDIT: Got rate-limited again. :(

[JorisLimousinKaizen]

Just got the following answer from the AWS support in case it helps, will try it myself soon:

From reading your correspondence I understand that you are currently facing an issue in your CDK pipelines, where builds are failing with the following error:

---

toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

---

This is a known issue that occurs now that DockerHub have introduced pull rate limiting. To resolve the error that you receive when throttling happens, you must configure CodeBuild to authenticate the layer pulls using your DockerHub account credentials. Full steps to resolve this issue can be found at the following knowledge center article [1]. However, to summarize;

• Store your DockerHub credentials with AWS Secrets Manager
• Resolve throttling that happens in the PROVISIONING phase
• Resolve throttling that happens in other phases
• Configure CodeBuild to retrieve the secret

If you have any follow-up questions or concerns, please feel free to contact me, I'm happy to help. Please note, my working hours are 9AM -5PM Monday - Friday, Irish Standard Time as I am based in Dublin, Ireland.

[1] How do I resolve the “error pulling image configuration: toomanyrequests” error when I use Docker images in CodeBuild? - https://aws.amazon.com/premiumsupport/knowledge-center/codebuild-docker-pull-image-error

[rix0rrr]

Until we have a proper solution for this, you can try the ECR Mirror construct we have developed here: https://github.com/awslabs/aws-delivlib#ecr-mirror

[cvrajeesh]

one workaround is to use base images from Amazon ECR Public Gallery (https://gallery.ecr.aws/) or create your own public ECR repository and push your images to that.

In my case, I switched from docker hub base image golang:1.14.13 to public ECR base image public.ecr.aws/bitnami/golang:1.14.13

-- FROM golang:1.14.13
++ FROM public.ecr.aws/bitnami/golang:1.14.13

[rix0rrr]

Duplicate of #10999

[guydelta]

I also had this issue and resolved it for me as follow in order to get nodejs constructs to work without going to docker each time, but to use aws's own library: #11296 (comment)

So if you just need to build your lambda/lambdalayer constructs for nodej, then hopefully that reply will help.

[chand1012]

In my case, I switched from docker hub base image golang:1.14.13 to public ECR base image public.ecr.aws/bitnami/golang:1.14.13

I would try this, but I actually got rate limited on an AWS image that's being hosted on Docker Hub that's vital to the CDK build process.

Why are the AWS images for CDK hosted on Docker Hub and not ECR? It seems a little odd if people are going to have to worry about rate limits when building.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.