Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(eks): control plane logs #8497

Closed
Closed
Show file tree
Hide file tree
Changes from 28 commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
e38402e
support encoding for boolean values
eduardomourar Jun 10, 2020
71f59a1
Merge branch 'master' into feature/enable-control-plane-logs
eduardomourar Jun 11, 2020
494330f
improve description custom resource properties
eduardomourar Jun 17, 2020
e82e8a6
remove decode boolean from eks
eduardomourar Jun 17, 2020
8b582d3
fix linting
eduardomourar Jun 17, 2020
e8bf3aa
add decode booleans to runtime code
eduardomourar Jun 18, 2020
83f0cb4
add unit tests in core lib
eduardomourar Jun 19, 2020
72226b0
encode values for helm charts
eduardomourar Jun 19, 2020
f777ef4
add support for control plane logging
eduardomourar Jun 19, 2020
0ec6571
update hash on core test
eduardomourar Jun 20, 2020
e37d673
update integ for iam
eduardomourar Jun 20, 2020
a250704
Merge 'master' into feature branch
eduardomourar Jun 20, 2020
b15bdea
increase unit test coverage
eduardomourar Jun 20, 2020
8c79e70
update readme
eduardomourar Jun 20, 2020
8628281
add env variables to handle encoded values
eduardomourar Jun 22, 2020
d84610e
update eks logging api
eduardomourar Jun 22, 2020
bad1672
update iam integ test
eduardomourar Jun 23, 2020
b4cdc58
update integ tests
eduardomourar Jun 23, 2020
3e53574
update as per suggestion
eduardomourar Jun 23, 2020
e799afb
suggested changes
eduardomourar Jun 23, 2020
438e3d5
Merge 'master' into feature branch
eduardomourar Jun 23, 2020
a609ab9
add helm wait comment
eduardomourar Jun 23, 2020
5c70951
revert back all changes to runtime code
eduardomourar Jun 23, 2020
86af207
update test files
eduardomourar Jun 23, 2020
d538205
Update integ.core-custom-resources.expected.json
Jun 23, 2020
876a012
Merge branch 'master' into feature/enable-control-plane-logs
eduardomourar Jun 23, 2020
da23fc4
Merge branch 'master' into feature/enable-control-plane-logs
eduardomourar Jun 23, 2020
4d65eb3
Update integ.core-custom-resources.expected.json
Jun 24, 2020
abb5780
Update packages/@aws-cdk/aws-eks/README.md
Jun 24, 2020
53d647f
Reorg file
Jun 24, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 30 additions & 0 deletions packages/@aws-cdk/aws-eks/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -386,6 +386,36 @@ A convenience method for mapping a role to the `system:masters` group is also av
cluster.awsAuth.addMastersRole(role)
```

### Kubernetes Control Plane Logging

Amazon [EKS control plane logging](https://docs.aws.amazon.com/eks/latest/userguide/control-plane-logs.html)
provides audit and diagnostic logs directly from the Kubernetes control plane
to CloudWatch Logs in your account.

By default, cluster control plane logs aren't sent to CloudWatch Logs. You must
enable each log type individually to send logs for your cluster. You can activate
all control plane logs by having the property `controlPlaneLogging: true`,
then the following setting will be applied:

```ts
new eks.Cluster(this, 'Cluster', {
controlPlaneLoggingOptions: {
api: true,
audit: true,
authenticator: true,
controllerManager: true,
scheduler: true,
}
});
```

For more details about the log types available check the
[API reference documentation](https://docs.aws.amazon.com/eks/latest/APIReference/API_LogSetup.html).

**NOTE**: the control plane logging is only available if the cluster
has the `kubectlEnabled: true`, because there is still no
eladb marked this conversation as resolved.
Show resolved Hide resolved
CloudFormation support to change that configuration.

### Cluster Security Group

When you create an Amazon EKS cluster, a
Expand Down
20 changes: 20 additions & 0 deletions packages/@aws-cdk/aws-eks/lib/cluster-resource-handler/common.ts
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,22 @@ export interface EksUpdateId {

export type ResourceEvent = AWSLambda.CloudFormationCustomResourceEvent & EksUpdateId;

/**
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Revert

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is actually needed because the provide framework is not be handling it anymore

* Decodes encoded true/false values
*/
function decodeBooleans(object: object) {
eduardomourar marked this conversation as resolved.
Show resolved Hide resolved
return JSON.parse(JSON.stringify(object), (_k, v) => {
switch (v) {
case 'TRUE:BOOLEAN':
return true;
case 'FALSE:BOOLEAN':
return false;
default:
return v;
}
});
}

export abstract class ResourceHandler {
protected readonly requestId: string;
protected readonly logicalResourceId: string;
Expand All @@ -33,6 +49,10 @@ export abstract class ResourceHandler {
throw new Error('AssumeRoleArn must be provided');
}

if (event.ResourceProperties.Config) {
this.event.ResourceProperties.Config = decodeBooleans(event.ResourceProperties.Config);
}

eks.configureAssumeRole({
RoleArn: roleToAssume,
RoleSessionName: `AWSCDK.EKSCluster.${this.requestType}.${this.requestId}`,
Expand Down
99 changes: 97 additions & 2 deletions packages/@aws-cdk/aws-eks/lib/cluster-resource.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,73 @@ import { ArnComponents, Construct, CustomResource, Lazy, Stack, Token } from '@a
import { CLUSTER_RESOURCE_TYPE } from './cluster-resource-handler/consts';
import { ClusterResourceProvider } from './cluster-resource-provider';
import { CfnClusterProps } from './eks.generated';
import { ControlPlaneLogging } from './logging';

/**
* Encodes values (specially boolean) as special strings
*
* Because CloudFormation converts every input as string for custom resources,
* we will use these encoded values to cast them into proper types.
*/
function encodeValues(object: object) {
return JSON.parse(JSON.stringify(object), (_k, v) => {
switch (v) {
case true:
return 'TRUE:BOOLEAN';
case false:
return 'FALSE:BOOLEAN';
default:
return v;
}
});
}

/**
* The available cluster control plane log types.
*/
export type LogKind = keyof ControlPlaneLogging;

/**
* An object representing the enabled or disabled Kubernetes control plane logs for your cluster.
*/
export interface LogSetup {
/**
* If a log type is enabled, that log type exports its control plane logs to CloudWatch Logs.
* If a log type is not enabled, that log type does not export its control plane logs.
* Each individual log type can be enabled or disabled independently.
*/
readonly enabled: boolean;

/**
* The available cluster control plane log types.
*/
readonly types: LogKind[];
}

/**
* An object representing the logging configuration for resources in your cluster.
*/
export interface Logging {

/**
* The cluster control plane logging configuration for your cluster.
*/
clusterLogging: LogSetup[];
}

/**
* Configuration props for EKS Cluster custom resource.
*/
export interface ClusterResourceProps extends CfnClusterProps {
/**
* Enable or disable exporting the Kubernetes control plane logs for your cluster to CloudWatch Logs.
*
* @see https://docs.aws.amazon.com/eks/latest/userguide/control-plane-logs.html
*
* @default - Fully disabled, meaning that cluster control plane logs are not exported to CloudWatch Logs.
*/
readonly logging?: ControlPlaneLogging;
}

/**
* A low-level CFN resource Amazon EKS cluster implemented through a custom
Expand Down Expand Up @@ -32,7 +99,7 @@ export class ClusterResource extends Construct {

private readonly trustedPrincipals: string[] = [];

constructor(scope: Construct, id: string, props: CfnClusterProps) {
constructor(scope: Construct, id: string, props: ClusterResourceProps) {
super(scope, id);

const stack = Stack.of(this);
Expand Down Expand Up @@ -113,11 +180,39 @@ export class ClusterResource extends Construct {
resources: [ '*' ],
}));

const enabledLogTypes: LogKind[] = [];
const disabledLogTypes: LogKind[] = [];
for (const [key, value] of Object.entries(props.logging || {})) {
if (value === true) {
enabledLogTypes.push(key as LogKind);
} else {
disabledLogTypes.push(key as LogKind);
}
}
let logging: undefined | Logging;
if (enabledLogTypes.length > 0) {
logging = {
clusterLogging: [
{
enabled: true,
types: enabledLogTypes,
},
{
enabled: false,
types: disabledLogTypes,
},
],
};
}

const resource = new CustomResource(this, 'Resource', {
resourceType: CLUSTER_RESOURCE_TYPE,
serviceToken: provider.serviceToken,
properties: {
Config: props,
Config: encodeValues({
...props,
logging,
}),
AssumeRoleArn: this.creationRole.roleArn,

// IMPORTANT: increment this number when you add new attributes to the
Expand Down
47 changes: 46 additions & 1 deletion packages/@aws-cdk/aws-eks/lib/cluster.ts
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ import { HelmChart, HelmChartOptions } from './helm-chart';
import { KubernetesPatch } from './k8s-patch';
import { KubernetesResource } from './k8s-resource';
import { KubectlProvider } from './kubectl-provider';
import { ControlPlaneLogging } from './logging';
import { Nodegroup, NodegroupOptions } from './managed-nodegroup';
import { ServiceAccount, ServiceAccountOptions } from './service-account';
import { LifecycleLabel, renderAmazonLinuxUserData, renderBottlerocketUserData } from './user-data';
Expand Down Expand Up @@ -244,6 +245,21 @@ export interface ClusterProps extends ClusterOptions {
*/
readonly kubectlEnabled?: boolean;

/**
* Enable or disable exporting the Kubernetes control plane logs for your cluster to CloudWatch Logs.
*
* @default false Cluster control plane logs are not exported to CloudWatch Logs.
*/
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happens if I enabled logs and then set to undefined (remove it). Would it set all the logging to false?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it will send an empty value for logging in the call to updateClusterConfig(), and it would result in setting the default value from AWS

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have you tested this? My intuition is that if I send an updateClusterConfig request with an empty value for logging it will not perform any changes to my logging configuration, which is not what users expect if they enable logs and then remove this property in their CDK app.

I suspect that we may either need to always send the two enabled/disable lists or add heuristics to the UPDATE handler to calculate the "diff" from the previous value and produce an appropriate update.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i will try this out and let you know, but this diff is being done in the runtime code here:

updateLogging: JSON.stringify(newProps.logging) !== JSON.stringify(oldProps.logging),

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This only determines that there was a change, not which log types should now be enabled/disabled based on that change.

For example, say, "oldProps" included two enabled types (say ["typeA", "typeB"]) and now "newProps" include only a single type (say ["typeB"]). Then, the request we need to send to updateClusterConfig is only to disable (we can, if we want also include an "enable" for typeB, but technically it's not required because it is already enabled).

LMK if this makes sense.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i believe we will need to send the whole configuration explicitly, including the disabled one. i am getting the following error when sending the exact same configuration to updateClusterConfig:

Failed to update resource. Error: No changes needed for the logging config provided

readonly controlPlaneLogging?: boolean;

/**
* EKS control plane logging options.
*
* @default - none
* @see https://docs.aws.amazon.com/eks/latest/userguide/control-plane-logs.html
*/
readonly controlPlaneLoggingOptions?: ControlPlaneLogging;

/**
* Number of instances to allocate as an initial capacity for this cluster.
* Instance type can be configured through `defaultCapacityInstanceType`,
Expand Down Expand Up @@ -349,6 +365,11 @@ export class Cluster extends Resource implements ICluster {
*/
public readonly kubectlEnabled: boolean;

/**
* The Control Plane logging configuration for your cluster.
*/
public readonly controlPlaneLogging?: ControlPlaneLogging;

/**
* The auto scaling group that hosts the default capacity for this cluster.
* This will be `undefined` if the `defaultCapacityType` is not `EC2` or
Expand Down Expand Up @@ -441,9 +462,33 @@ export class Cluster extends Resource implements ICluster {
let resource;
this.kubectlEnabled = props.kubectlEnabled === undefined ? true : props.kubectlEnabled;
if (this.kubectlEnabled) {
resource = new ClusterResource(this, 'Resource', clusterProps);
let logging: undefined | ControlPlaneLogging;
if (props.controlPlaneLogging === true && !props.controlPlaneLoggingOptions) {
logging = {
api: true,
audit: true,
authenticator: true,
controllerManager: true,
scheduler: true,
};
} else if (Object.keys(props.controlPlaneLoggingOptions || {}).length > 0) {
if (props.controlPlaneLogging === false) {
throw new Error('Cannot configure control plane logging if "controlPlaneLogging" is false');
}
logging = props.controlPlaneLoggingOptions;
}
this.controlPlaneLogging = logging;
resource = new ClusterResource(this, 'Resource', {
...clusterProps,
logging,
});
this._clusterResource = resource;
} else {
if (props.controlPlaneLogging || props.controlPlaneLoggingOptions) {
throw new Error('Cannot configure control plane logging if kubectl is disabled');
} else {
this.controlPlaneLogging = undefined;
}
resource = new CfnCluster(this, 'Resource', clusterProps);
}

Expand Down
5 changes: 4 additions & 1 deletion packages/@aws-cdk/aws-eks/lib/helm-chart.ts
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,10 @@ export class HelmChart extends Construct {
Release: props.release ?? this.node.uniqueId.slice(-53).toLowerCase(), // Helm has a 53 character limit for the name
Chart: props.chart,
Version: props.version,
Wait: props.wait ?? false,
// We will only pass down to the custom resource an explicit value
// if wait is set to true. Falsy values will not because
// that is the default behavior.
Wait: props.wait || undefined,
eduardomourar marked this conversation as resolved.
Show resolved Hide resolved
Timeout: timeout,
Values: (props.values ? stack.toJsonString(props.values) : undefined),
Namespace: props.namespace ?? 'default',
Expand Down
1 change: 1 addition & 0 deletions packages/@aws-cdk/aws-eks/lib/index.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
export * from './aws-auth';
export * from './aws-auth-mapping';
export * from './cluster';
export * from './logging';
export * from './eks.generated';
export * from './fargate-profile';
export * from './helm-chart';
Expand Down
50 changes: 50 additions & 0 deletions packages/@aws-cdk/aws-eks/lib/logging.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
/**
* EKS cluster control plane logging configuration
*/
export interface ControlPlaneLogging {
/**
* Kubernetes API server component logs – Your cluster's API server is the
* control plane component that exposes the Kubernetes API.
*
* @default false
* @see https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/
*/
readonly api?: boolean;

/**
* Audit (audit) – Kubernetes audit logs provide a record of the individual
* users, administrators, or system components that have affected your cluster.
*
* @default false
* @see https://kubernetes.io/docs/tasks/debug-application-cluster/audit/
*/
readonly audit?: boolean;

/**
* Authenticator – Authenticator logs are unique to Amazon EKS. These logs
* represent the control plane component that Amazon EKS uses for Kubernetes
* Role Based Access Control (RBAC) authentication using IAM credentials.
*
* @default false
* @see https://kubernetes.io/docs/admin/authorization/rbac/
*/
readonly authenticator?: boolean;

/**
* Controller manager – The controller manager manages the core control
* loops that are shipped with Kubernetes.
*
* @default false
* @see https://kubernetes.io/docs/reference/command-line-tools-reference/kube-controller-manager/
*/
readonly controllerManager?: boolean;

/**
* Scheduler – The scheduler component manages when and where to run
* pods in your cluster.
*
* @default false
* @see https://kubernetes.io/docs/reference/command-line-tools-reference/kube-scheduler/
*/
readonly scheduler?: boolean;
}
Loading