(pipelines): Allow configuring log retention period for CdkPipeline

[twitu]

Cloudwatch logs are costly resources and CdkPipeline by default creates separate log groups for each build action. This means in practice building 30 lambda functions creates 30 log groups each of which is updated on each build. All other stages and actions also create their own logs groups.

Further compounding the problem is the default retention period for the logs which is Infinite (or 2 years). This can very very quickly increase the cost of the infrastructure.

Use Case

Frequent deployment creates a lot of logs that retained infinitely adding up to the total cost of infrastructure.

Proposed Solution

Maybe another parameter in the CdkPipeline construct for passing in a custom log group like in api-gateway might work. However that may not be a good level of abstraction as people might want to keep build logs separate from deploy logs.

Another approach might be to create a parameter for retention period that is applied to all log groups created under the CdkPipeline construct.

Workaround is probably creating your own CodePipeline which would defeat the purpose of having CdkPipeline.

This might also be considered for GA release discussion for CdkPipeline #10872

Other

• 👋 I may be able to implement this feature request
• ⚠️ This feature might incur a breaking change

---

This is a 🚀 Feature Request

[rix0rrr]

Yes, I see the point of this.

I'm going to take a stab and say configuring log group retention should be a CodeBuild construct library feature. If we want to configure it en-masse, there could be something like an Aspect or a context key which can configure it for an entire construct tree perhaps.

[rafalwrzeszcz]

A little dirty workaround I created in my stack is:

        this.node.findAll().forEach {
            if (it is IProject) {
                println("Adding log group for ${it.node.id}")
                LogGroup(
                    this,
                    "${it.node.id}LogGroup",
                    LogGroupProps.builder()
                        .logGroupName("/aws/codebuild/${it.projectName}")
                        .retention(RetentionDays.TWO_WEEKS)
                        .build()
                )
            }
        }

I think similar setup could be made from within the construct soemwhere at https://github.com/aws/aws-cdk/blob/v1.113.0/packages/@aws-cdk/pipelines/lib/synths/simple-synth-action.ts#L371 - repeated for build-synth and self-update step actions

@twitu passing LogGroup construct to CodeBuild project won't work as CodeBuild logs are not customizable (can only be disabled) - they are always logged into /aws/codebuild/${PROJECT_NAME}.

[abend-arg]

A little dirty workaround I created in my stack is:

        this.node.findAll().forEach {
            if (it is IProject) {
                println("Adding log group for ${it.node.id}")
                LogGroup(
                    this,
                    "${it.node.id}LogGroup",
                    LogGroupProps.builder()
                        .logGroupName("/aws/codebuild/${it.projectName}")
                        .retention(RetentionDays.TWO_WEEKS)
                        .build()
                )
            }
        }

I think similar setup could be made from within the construct soemwhere at https://github.com/aws/aws-cdk/blob/v1.113.0/packages/@aws-cdk/pipelines/lib/synths/simple-synth-action.ts#L371 - repeated for build-synth and self-update step actions

@twitu passing LogGroup construct to CodeBuild project won't work as CodeBuild logs are not customizable (can only be disabled) - they are always logged into /aws/codebuild/${PROJECT_NAME}.

I ran into this issue and this approach, though a bit hacky, worked like a charm. For those who are working with Typescript I am leaving you the piece of code accomplishing this thing for CDK version 1.128:

        import {Project} from '@aws-cdk/aws-codebuild'
        import {LogRetention, RetentionDays} from '@aws-cdk/aws-logs'
        //
        this.node.findAll().forEach((construct, index) => {
            if (construct instanceof Project) {
                new LogRetention(this, `${PREFIX}${PROD}LogRetention${index}`, {
                    logGroupName: `/aws/codebuild/${construct.projectName}`,
                    retention: RetentionDays.THREE_DAYS,
                })
            }
        });

this.node lives in the pipeline stack.

[HsiehShuJeng]

This is what I do in Typescript from the enlightenment by @rafalwrzeszcz .

import * as cdk from '@aws-cdk/core';
import * as cb from '@aws-cdk/aws-codebuild';
import * as logs from '@aws-cdk/aws-logs';

/**
 * Finds out all constructs that are `@aws-cdk/aws-codebuild.Project`.
 * @see https://github.com/aws/aws-cdk/issues/13250
 */
 class CodeBuildLooker implements cdk.IAspect {
  counter: number = 1;
  public visit(node: cdk.IConstruct): void {
    if (node instanceof cb.Project) {
      const projectLogGroup = new logs.LogGroup(pipelineStack, `CodeBuilLogGroup${this.counter}`, {
        logGroupName: `/aws/codebuild/${node.projectName}`,
        retention: logs.RetentionDays.THREE_MONTHS
      });
      const cfnCbProject = node.node.defaultChild as cb.CfnProject;
      cfnCbProject.logsConfig = {
        cloudWatchLogs: {
          groupName: projectLogGroup.logGroupName,
          status: 'ENABLED'
        }
      };
      console.log(`Added a custom log group for ${cfnCbProject.logicalId}`);
      this.counter += 1;
    }
  }
}

// Add this statement after the initialization of a stack where a pipelines.pipeline exists.
cdk.Aspects.of(pipelineStack).add(new CodeBuildLooker());

[HsiehShuJeng]

This is what I do in Typescript from the enlightenment by @rafalwrzeszcz .

import * as cdk from '@aws-cdk/core';
import * as cb from '@aws-cdk/aws-codebuild';
import * as logs from '@aws-cdk/aws-logs';

/**
 * Finds out all constructs that are `@aws-cdk/aws-codebuild.Project`.
 * @see https://github.com/aws/aws-cdk/issues/13250
 */
 class CodeBuildLooker implements cdk.IAspect {
  counter: number = 1;
  public visit(node: cdk.IConstruct): void {
    if (node instanceof cb.Project) {
      const projectLogGroup = new logs.LogGroup(pipelineStack, `CodeBuilLogGroup${this.counter}`, {
        logGroupName: `/aws/codebuild/${node.projectName}`,
        retention: logs.RetentionDays.THREE_MONTHS
      });
      const cfnCbProject = node.node.defaultChild as cb.CfnProject;
      cfnCbProject.logsConfig = {
        cloudWatchLogs: {
          groupName: projectLogGroup.logGroupName,
          status: 'ENABLED'
        }
      };
      console.log(`Added a custom log group for ${cfnCbProject.logicalId}`);
      this.counter += 1;
    }
  }
}

// Add this statement after the initialization of a stack where a pipelines.pipeline exists.
cdk.Aspects.of(pipelineStack).add(new CodeBuildLooker());

This implementation won't work since it would fail with the circular dependencies error message when being about to deploy with the template generated by the CDK application in the CDK pipeline.
The following implementation works for me as well, the cdk version 1.129.0 (build fb43f89) and the Typescript version is Version 4.3.5.

A little dirty workaround I created in my stack is:

        this.node.findAll().forEach {
            if (it is IProject) {
                println("Adding log group for ${it.node.id}")
                LogGroup(
                    this,
                    "${it.node.id}LogGroup",
                    LogGroupProps.builder()
                        .logGroupName("/aws/codebuild/${it.projectName}")
                        .retention(RetentionDays.TWO_WEEKS)
                        .build()
                )
            }
        }

I think similar setup could be made from within the construct soemwhere at https://github.com/aws/aws-cdk/blob/v1.113.0/packages/@aws-cdk/pipelines/lib/synths/simple-synth-action.ts#L371 - repeated for build-synth and self-update step actions
@twitu passing LogGroup construct to CodeBuild project won't work as CodeBuild logs are not customizable (can only be disabled) - they are always logged into /aws/codebuild/${PROJECT_NAME}.

I ran into this issue and this approach, though a bit hacky, worked like a charm. For those who are working with Typescript I am leaving you the piece of code accomplishing this thing for CDK version 1.128:

        import {Project} from '@aws-cdk/aws-codebuild'
        import {LogRetention, RetentionDays} from '@aws-cdk/aws-logs'
        //
        this.node.findAll().forEach((construct, index) => {
            if (construct instanceof Project) {
                new LogRetention(this, `${PREFIX}${PROD}LogRetention${index}`, {
                    logGroupName: `/aws/codebuild/${construct.projectName}`,
                    retention: RetentionDays.THREE_DAYS,
                })
            }
        });

this.node lives in the pipeline stack.

[kornicameister]

class ProjectLogGroup:

    def visit(self, stack: cdk.Stack) -> None:
        for node in stack.node.find_all():
            self._visit(stack, node)

    def _visit(self, scope: cdk.Stack, node: cdk.IConstruct) -> None:
        if isinstance(node, cb.Project):
            cfn_project = t.cast(cb.CfnProject, node.node.default_child)

            log_group = logs.LogGroup(
                scope,
                f'{cdk.Names.unique_id(node)}LogGroup',
                log_group_name=f'/aws/codebuild/{node.project_name}',
                removal_policy=cdk.RemovalPolicy.DESTROY,
                retention=logs.RetentionDays.TWO_WEEKS,
            )

            cfn_project.logs_config = cb.CfnProject.LogsConfigProperty(
                cloud_watch_logs=cb.CfnProject.CloudWatchLogsConfigProperty(
                    status='ENABLED',
                    group_name=log_group.log_group_name,
                ),
            )

that code does not even create log groups for me :/

CDK: 1.139.0

[lschierer]

I'd like this for CDK2 as well.

[knovichikhin]

1. LogRetention is custom resource (a lambda) that brings its own problems. And its own logs, code storage, etc.
2. Creating a log in /aws/codebuild/ does not work. It does not create any logs.

The simplest solution I found is to create your own log and assign it to your projects. It even works just fine if you have dozens of different stacks with pipelines in them. One stack can own the log and the rest can reference it by name.

If you want a separate log per pipeline or per project, adjust shared log name to include those. E.g. /buildLogs/pipelineName.

This works with new pipeline API and with CdkPipeline.

import * as cdk from 'monocdk';
import * as iam from "monocdk/aws-iam";
import * as codebuild from "monocdk/aws-codebuild";
import * as logs from 'monocdk/aws-logs';

const buildLogName = 'BuildLog'

/**
 * Inspect provided stack for code build projects and assign them a fixed build log.
 *
 * @param scope stack to inspect for code build projects
 */
export function attachBuildLog(scope: cdk.Stack) {
    new logs.LogGroup(scope, buildLogName, {
        logGroupName: buildLogName,
        retention: logs.RetentionDays.ONE_WEEK
    });

    scope.node.findAll().forEach((codeBuildProject) => {
        if (codeBuildProject instanceof codebuild.Project) {
            const cfnCodeBuildProject = codeBuildProject.node.defaultChild as codebuild.CfnProject;
            cfnCodeBuildProject.logsConfig = {
                cloudWatchLogs: {
                    groupName: buildLogName,
                    status: 'ENABLED'
                }
            };
            codeBuildProject.role?.addManagedPolicy(iam.ManagedPolicy.fromAwsManagedPolicyName('CloudWatchLogsFullAccess'));
            console.log(`Assigned ${buildLogName} log to project ${codeBuildProject.projectName} in stack ${scope.stackName}.`)
        }
    })
}

[github-actions]

This issue has received a significant amount of attention so we are automatically upgrading its priority. A member of the community will see the re-prioritization and provide an update on the issue.