[pipelines] Docker logins for assets

[tobli]

When using CDK Pipelines the autogenerated Assets action will build Docker images, and publish to the cdk-provided ECR. However if included Dockerfiles build on images in a non-public repository (e.g. an ECR in a different account), those builds will fail since the Assets action has no way of specifying sources to docker login in to.

Use Case

Prior to using Pipelines we've used a shared ECR in a dedicated account to both store our internal base images, as well as images built on top of those. A single docker login would cover both pull and push from that repo.

However when switching to Pipelines, the destination repo changes. Pipelines transparently handles login to to that, but provides no configuration option for docker registries that need to be logged in to prior to asset building.

Proposed Solution

Other

The error message from the Assets/DockerAsset1 CodeBuild project was:

Step 1/13 : FROM <account>.dkr.ecr.eu-west-1.amazonaws.com/...
| Get https://<account>.dkr.ecr.eu-west-1.amazonaws.com/...: no basic auth credentials

• 👋 I may be able to implement this feature request
• ⚠️ This feature might incur a breaking change

---

This is a 🚀 Feature Request

[rix0rrr]

How would you expect this to work on the desktop? I.e. if you do cdk deploy?

[rix0rrr]

(What I'm saying is this might have implications for the asset manifest and CLI as well)

[tobli]

TBH I had no expectations of a particular behaviour when deploying locally. I only started using docker image assets after having set up a Pipeline. That said my understanding of cdk synth from having used cdk previously was that it represented the "compile CF templates" step.

[tobli]

The requirement of having done docker login was also never explicit when developing locally, since I was already logged in. Only when moving the docker build to CodeBuild did this become apparent.

[tobli]

Here's the hack I added to my pipeline stack (after the pipeline was created) to get past this:

  private static addECRLogin (pipeline: CdkPipeline, sourceECRs: string[]) {
    for (const action of pipeline.stage('Assets')?.actions) {
      const actionProperties = action.actionProperties;
      if (actionProperties.actionName.startsWith('Docker')) {
        // workaround for https://github.com/aws/aws-cdk/issues/10999
        const publishAction = action as PublishAssetsAction;
        const commands: string[] = (publishAction as any).commands;
        for (const sourceECR of sourceECRs) {
          // NOTE: this makes the simplifying assumption that the sourceECR is in the same region as the pipeline
          const command = `aws ecr get-login-password --region ${Stack.of(pipeline).region} | docker login --username AWS --password-stdin ${sourceECR}`;
          if (!commands.includes(command)) {
            // login needs to happen before the asset publication (that's where docker images are built)
            commands.unshift(command);
          }
        }

        new Policy(pipeline, 'AllowECRLoginAndPull', {
          statements: [
            new PolicyStatement({
              actions: [
                'ecr:GetAuthorizationToken',
                'ecr:GetDownloadUrlForLayer',
                'ecr:BatchGetImage',
              ],
              resources: ['*'],
              sid: 'AllowECRLoginAndPull',
            }),
          ],
        }).attachToRole(actionProperties.role!);
      }
    }
  }

[nsquires413]

@tobli What do you pass in as sourceECRs for this?

[pgarbe]

@nsquires413 It's the full ECR repo name, e.g. 123456789012.dkr.ecr.eu-west-1.amazonaws.com

[nsquires413]

@tobli @pgarbe One more question if you don't mind. I'm trying to translate this method into C#.

Looking at the 'PublishAssetsAction' source code it looks like the it has a private member 'commands'.

It seems like you were still able to access this field by casting the action as 'any'? Is that right? Does anyone know if this is possible in C#?

[tobli]

It seems like you were still able to access this field by casting the action as 'any'? Is that right? Does anyone know if this is possible in C#?

Correct, it's accessing private field via the cast. That's of course risky, but Typescript lets you do that. If C# does I have no clue I'm afraid.

[rix0rrr]

When we get to implementing this, see Quip doc with reference PameA2ADtrao.

See also #11774

[quincycs]

I had same problem, but I just put docker login as a pre-build command on the codebuild project.

https://github.com/quincycs/aws-cdk-q-starter/blob/master/cdk-app/src/PipelineStack.ts#L102

This was an easy fix for me, so maybe I don't understand the issue.

[tobli]

@quincycs this issue arises when using aws-ecr-assets to build a Docker image as part of the pipeline. Cdk Assets will create a new Assets build pipeline step as an implementation detail. It's for that the login step is needed.

[karthikns16]

In python, we don't have access to the publishAssetAction.commands. I ended up writing a script to access the pipeline.template.json using aws_cdk.cx_api and update the buildspec phases for all docker asset. This script was executed as part of synth_command in the pipeline after we run cdk synth.

Example:

pipelines.SimpleSynthAction(
synth_command = "cdk synth; python <script created to update the template>"
)

Output Asset Buildspec.yml

{
    "version": "0.2",
    "phases": {
        "install": {
            "commands": "npm install -g cdk-assets@1.85.0"
        },
        "build": {
            "commands": [
                "cdk-assets --path \"assemblyXXXXX/XXXX.assets.json\" --verbose publish \"XXXXX\""
            ]
        },
        "pre_build": {
            "commands": "aws ecr get-login-password --region XXXX | docker login --username AWS --password-stdin XXXX.dkr.ecr.XXXX.amazonaws.com"
        }
    }
}

[blimmer]

I needed to login to a dockerhub paid account, so I worked around like this. The auth token is stored in
Secrets Manager for security.

function maybeAddDockerLogin(scope: cdk.Construct, pipeline: pipelines.CdkPipeline) {
  let assetStage: codepipeline.IStage;
  try {
    assetStage = pipeline.stage('Assets');
  } catch {
    // If a stage with a given name cannot be found, CDK throws an exception
    return;
  }

  const dockerAssetActions = assetStage.actions.filter((action) => {
    const actionProperties = action.actionProperties;
    return actionProperties.actionName.startsWith('Docker');
  });

  if (dockerAssetActions.length === 0) {
    return;
  }

  // Replace the last parameter with the name of your secret.
  // This secret is the access token for a paid dockerhub user
  // https://docs.docker.com/docker-hub/access-tokens/
  const dockerHubSecret = secrets.Secret.fromSecretNameV2(scope, 'SharedSecretImport', 'shared/dockerhub-access-token');
  const secretAccessPolicy = new iam.Policy(scope, 'SecretAccessPolicy', {
    statements: [
      new iam.PolicyStatement({
        sid: 'AllowFetchingDockerHubSecret',
        actions: ['secretsmanager:GetSecretValue'],
        resources: [`${dockerHubSecret.secretArn}-??????`],
      }),
    ],
  });

  dockerAssetActions.forEach((action) => {
    const publishAction = action as any;
    const codeBuildProject = publishAction.node.defaultChild.node.defaultChild;
    codeBuildProject.environment.environmentVariables = codeBuildProject.environment.environmentVariables ?? [];
    codeBuildProject.environment.environmentVariables.push({
      name: 'DOCKER_AUTH_TOKEN',
      type: 'SECRETS_MANAGER',
      value: dockerHubSecret.secretName,
    });
    secretAccessPolicy.attachToRole(action.actionProperties.role!);

    const commands: string[] = publishAction.commands;
    const command = 'echo ${DOCKER_AUTH_TOKEN} | docker login --username gingereng --password-stdin';
    if (!commands.includes(command)) {
      // login needs to happen before the asset publication (that's where docker images are built)
      commands.unshift(command);
    }
  });
}

(credit to @tobli's comment above for the majority of this)

[jabrennem]

hi @tobli - do you know when the 'Assets' stage is created? I tried using your hack in python, but the command pipeline.stage('Assets') throws an error for me saying that there is no Assets stage available. Only self Update, Build, Source, and Staging (my core.Stage). I was hoping to add some commands and policy statements to the code build project.

[karthikns16]

@rix0rrr , could we implement a boolean flag in CDKPipeline construct which is defaults FALSE. If enabled, we could add a blanket ECR login command in pre-build step of PublishAssetAction if the assetType === "Docker". The ECR should be in same account/region as the pipeline?

[david-fractiunate]

+1 Push, we need this.... CDK Docker Assets are not usable without it for our team.
Without a docker-login we face:

exited with error code 1: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

to fast...

CDK V.1.109.0 Python

It seems i cannot access the Assets Stage with
peline.stage(stage_name='Assets').actions

jsii.errors.JavaScriptError:
  Error: Pipeline does not contain a stage named 'Assets'. Available stages: Source, Build, UpdatePipeline

Neither could i access the commands:

  action_properties = action.action_properties
      if action_properties.actionName.startswith("Docker"):
        publish_action: PublishAssetsAction = action
        commands: [] = getattr(publish_action, "commands")
        command = f"aws ecr get-login-password --region ${Stack.of(pipeline).region} | docker login --username AWS --password-stdin ${sourceECR}"
        # login needs to happen before the asset publication (that's where docker images are built)
        if command not in commands:
          commands[:0] = [command]

Thanks a lot!

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[aax]

hi @tobli - do you know when the 'Assets' stage is created? I tried using your hack in python, but the command pipeline.stage('Assets') throws an error for me saying that there is no Assets stage available. Only self Update, Build, Source, and Staging (my core.Stage). I was hoping to add some commands and policy statements to the code build project.

@jabrennem The stage is created when you add an application stage which actually contains assets to be published.

[FelixRelli]

For anybody looking at this, #15364 added dockerCredentials in CodePipeline to use:

    const pipeline = new CodePipeline(this, 'MyPipeline', {
      ...,
      publishAssetsInParallel: true,
      dockerCredentials: [
        DockerCredential.ecr(
          [
            Repository.fromRepositoryArn(
              this,
              'PublicEcr',
              'arn:aws:ecr:us-east-1:763104351884:repository/*'
            ),
          ]
        ),
      ],
  }

For some reason publishAssetsInParallel must be set true though (default). Otherwiese got "This BuildSpec contains CloudFormation references and is supported by publishInParallel=false" and the build failed.

[cheruvian]

For some reason publishAssetsInParallel must be set true though (default). Otherwiese got "This BuildSpec contains CloudFormation references and is supported by publishInParallel=false" and the build failed.

Just banged my head on this for 4 hours today... -_- This has to be a bug right?