[aws-eks] Enable Control Plane logs in EKS cluster

[stefanolczak]

Use Case

Enabling Control Plane logging in EKS cluster is only possible by calling EKS API after cluster is created. Doing it in CDK requires to create Custom Resource with code that calls the API. It would be nice to have it as an argument for creating EKS cluster from CDK.

Proposed Solution

Since EKS is created from python lambda when kubectlEnabled flag is enabled there is a simple way to create the EKS cluster with logging enabled. Currently the lambda code uses boto3 method eks.create_cluster() where we can pass arguments to enable logging on created cluster. (https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/eks.html#EKS.Client.create_cluster).
The lambda uses config as an argument for this method :

aws-cdk/packages/@aws-cdk/aws-eks/lib/cluster-resource/index.py

Line 69 in c3b3c93

resp = eks.create_cluster(**config)

The config is passed as a properties of custom resource and is created here:

aws-cdk/packages/@aws-cdk/aws-eks/lib/cluster.ts

Lines 364 to 379 in c3b3c93

	const clusterProps: CfnClusterProps = {
	name: this.physicalName,
	roleArn: this.role.roleArn,
	version: props.version,
	resourcesVpcConfig: {
	securityGroupIds: [securityGroup.securityGroupId],
	subnetIds
	}
	};
	let resource;
	this.kubectlEnabled = props.kubectlEnabled === undefined ? true : props.kubectlEnabled;
	if (this.kubectlEnabled) {
	resource = new ClusterResource(this, 'Resource', clusterProps);
	this._defaultMastersRole = resource.creationRole;
	} else {

So I suggest to expose a way to include logging properties in the config so it should be passed to eks.create_cluster() method without any more changes. That should result in enabling logging on newly created EKS cluster.

---

This is a 🚀 Feature Request

[SomayaB]

Hi @stefanolczak, thanks for submitting a feature request! We will update this issue when there is any progress.

[yjw113080]

Any updates on this? Waiting for this feature as well.

[eladb]

This is not highly prioritized at the moment, but more than happy to take contributions.

[iliapolo]

Note that there is an abandoned PR for this: #8497

Consider resurrecting it once we pick this up again.

[rameshmimit]

Any update on this feature?

[iliapolo]

@rameshmimit We are discussing this issue internally, we'll update here soon.

[micheal-hill]

In the meantime, what workarounds are available? It seems to me that click-ops or AWS cli are the alternatives, but neither is amicable to automation - is that correct? CLI will error if there's no change required.

[iliapolo]

Since cluster logging can be updated post cluster creation, you can create a custom resource that updates that config.
The AWS Custom Resource should be a good fit for this.

[daisuke-yoshimoto]

Any update on this?

[jtomaszewski]

If anybody wants the workaround, here's code we wrote in our CDK to enable the logging for the cluster using AwsCustomResource.

import { FargateCluster } from "@aws-cdk/aws-eks";
import { Stack } from "@aws-cdk/core";
import {
  AwsCustomResource,
  AwsCustomResourcePolicy,
} from "@aws-cdk/custom-resources";

// Enables logs for the cluster.
//
// Taken from
// https://github.com/aws/aws-cdk/issues/4159#issuecomment-855625700
export function setupClusterLogging(
  stack: Stack,
  cluster: FargateCluster
): void {
  new AwsCustomResource(stack, "ClusterLogsEnabler", {
    policy: AwsCustomResourcePolicy.fromSdkCalls({
      resources: [`${cluster.clusterArn}/update-config`],
    }),
    onCreate: {
      physicalResourceId: { id: `${cluster.clusterArn}/LogsEnabler` },
      service: "EKS",
      action: "updateClusterConfig",
      region: stack.region,
      parameters: {
        name: cluster.clusterName,
        logging: {
          clusterLogging: [
            {
              enabled: true,
              types: [
                "api",
                "audit",
                "authenticator",
                "controllerManager",
                "scheduler",
              ],
            },
          ],
        },
      },
    },
    onDelete: {
      physicalResourceId: { id: `${cluster.clusterArn}/LogsEnabler` },
      service: "EKS",
      action: "updateClusterConfig",
      region: stack.region,
      parameters: {
        name: cluster.clusterName,
        logging: {
          clusterLogging: [
            {
              enabled: false,
              types: [
                "api",
                "audit",
                "authenticator",
                "controllerManager",
                "scheduler",
              ],
            },
          ],
        },
      },
    },
  });
}

[jasonumiker]

So I did the above with a AwsCustomResource and it worked great until we wanted to create a FargateProfile. Even adding an explicit dependency from the FargateProfile on the logging custom resource didn't help - it fails with a "you can't update the logs while we are creating a Fargate Profile" error. Having fought with it all day I am now having to look at going lower level to a Lambda-backed custom resource to get access to a is_complete_handler or something.

[micheal-hill]

@jasonumiker in our case we're using Fargate, but the logging config is set after Fargate has been setup. We're also using eks.FargateCluster in case that's relevant.

[jasonumiker]

Thanks - yeah I eventually tried flipping the dependency and that seems to work 🤞

[SpringMT]

CloudFormation (NOT CDK) supports EKS control plane logging settings.
https://aws.amazon.com/about-aws/whats-new/2021/11/amazon-eks-cluster-configuration-aws-cloudformation/
How About CDK?

[bweigel]

CloudFormation (NOT CDK) supports EKS control plane logging settings.

I thought this would be easy...just use an escape hatch, like this:

    const cfnCluster = cluster.node.defaultChild as eks.CfnCluster;
    cfnCluster.logging = {
      clusterLogging: {
        enabledTypes: [{ type: 'api' }, { type: 'audit' }, { type: 'authenticator' }, { type: 'controllerManager' }, { type: 'scheduler' }]
      }
    }

But alas, it seems like the CDK does not use the low-level CfnCluster (aka AWS::EKS::Cluster) to provision a cluster, but depends on a custom resource:

aws-cdk/packages/@aws-cdk/aws-eks/lib/cluster-resource.ts

Lines 73 to 102 in b8a4a9a

	const resource = new CustomResource(this, 'Resource', {
	resourceType: CLUSTER_RESOURCE_TYPE,
	serviceToken: provider.serviceToken,
	properties: {
	// the structure of config needs to be that of 'aws.EKS.CreateClusterRequest' since its passed as is
	// to the eks.createCluster sdk invocation.
	Config: {
	name: props.name,
	version: props.version,
	roleArn: props.roleArn,
	encryptionConfig: props.encryptionConfig,
	kubernetesNetworkConfig: props.kubernetesNetworkConfig,
	resourcesVpcConfig: {
	subnetIds: (props.resourcesVpcConfig as CfnCluster.ResourcesVpcConfigProperty).subnetIds,
	securityGroupIds: (props.resourcesVpcConfig as CfnCluster.ResourcesVpcConfigProperty).securityGroupIds,
	endpointPublicAccess: props.endpointPublicAccess,
	endpointPrivateAccess: props.endpointPrivateAccess,
	publicAccessCidrs: props.publicAccessCidrs,
	},
	},
	AssumeRoleArn: this.adminRole.roleArn,
	// IMPORTANT: increment this number when you add new attributes to the
	// resource. Otherwise, CloudFormation will error with "Vendor response
	// doesn't contain XXX key in object" (see #8276) by incrementing this
	// number, you will effectively cause a "no-op update" to the cluster
	// which will return the new set of attribute.
	AttributesRevision: 2,
	},
	});

Long story short the quick "escape hatch" hack does not work 😭

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.