Skip to content

Commit

Permalink
fix(aws-eks): proxy support and allow assigning a security group to a…
Browse files Browse the repository at this point in the history 
…ll cluster handler functions (#17200)

## Summary

This PR is intended for CDK EKS users who require all traffic to be routed through a proxy. Currently if a user does not allow internet connections to the VPC without going through a proxy, then deploying an EKS cluster will result in a timeout error:

```sh
Received response status [FAILED] from custom resource. Message returned: Error: 2021-10-20T14:20:47.028Z d86e3ef4-45ce-4130-988f-c4663f7f8c80 Task timed out after 60.06 seconds
```
Fixes: #12469, SIM D29159517
Related to but does not resolve: `https://github.com/aws/aws-cdk/issues/12171`

## ⚙️ Changes

_Expand each list item for additional details._

<details>
<summary><strong>Corrected "Cluster Handler" docs to clarify that 2 lambdas are created (<code>onEventHandler</code>, <code>isCompleteHandler</code>)</strong></summary>
<br />

Our docs [currently describe the "Cluster Handler" as one Lambda function that interacts with the EKS API](https://docs.aws.amazon.com/cdk/api/latest/docs/aws-eks-readme.html#cluster-handler). However this is not accurate. The "Cluster Handler" actually creates [two Lambdas](https://github.com/aws/aws-cdk/blob/0cabb9f2d2f50c03337cd6f35bf47fc54ada3a21/packages/%40aws-cdk/aws-eks/lib/cluster-resource-provider.ts#L69-L96) for the Custom Resource, `onEventHandler` and `isCompleteHandler`, both interact with the AWS API.

</details>

<details>
<summary><strong>Passes the <code>clusterHandlerEnvironment</code> to both Cluster Handler Lambdas</strong></summary>
<br />

The `clusterHandlerEnvironment` is the [recommended method](https://docs.aws.amazon.com/cdk/api/latest/docs/aws-eks-readme.html#cluster-handler) of passing a proxy url (i.g. `http_proxy: 'http://my-proxy.com:3128'`) to the Cluster Handler. 

Currently the `clusterHandlerEnvironment` is only passed to the Cluster Handler's `onEventHandler` Lambda. [The `onEventHandler` was believed to be the only Cluster Handler Lambda that interacts with the AWS EKS API](#12469 (comment)), however this is not entirely true. Both the `onEventHandler` and `isCompleteHandler` call the AWS EKS API.

Following the execution process of `isCompleteHandler` when creating an EKS cluster:

1. [`index.isComplete()` (this is the Lambda handler)](https://github.com/aws/aws-cdk/blob/0cabb9f2d2f50c03337cd6f35bf47fc54ada3a21/packages/%40aws-cdk/aws-eks/lib/cluster-resource-handler/index.ts#L48)
2. [`common.isComplete()`](https://github.com/aws/aws-cdk/blob/0cabb9f2d2f50c03337cd6f35bf47fc54ada3a21/packages/%40aws-cdk/aws-eks/lib/cluster-resource-handler/common.ts#L59)
3. [`cluster.isCreateComplete()`](https://github.com/aws/aws-cdk/blob/0cabb9f2d2f50c03337cd6f35bf47fc54ada3a21/packages/%40aws-cdk/aws-eks/lib/cluster-resource-handler/cluster.ts#L56)
4. [`cluster.isActive()`](https://github.com/aws/aws-cdk/blob/0cabb9f2d2f50c03337cd6f35bf47fc54ada3a21/packages/%40aws-cdk/aws-eks/lib/cluster-resource-handler/cluster.ts#L196)
5. [Request to EKS API](https://github.com/aws/aws-cdk/blob/0cabb9f2d2f50c03337cd6f35bf47fc54ada3a21/packages/%40aws-cdk/aws-eks/lib/cluster-resource-handler/cluster.ts#L198) (results in timeout because proxy is not used)

This change allows the user to pass proxy urls as environment variables to **both** Lambdas using `clusterHandlerEnvironment`.

</details>

<details>
<summary><strong>Renames the prop <code>onEventLayer</code> -> <code>proxyAgentLayer</code>, and provides the layer to both Cluster Handler Lambdas</strong></summary>
<br />

The proxy-agent layer is now used in both `onEventHandler` and `isCompleteHandler` lambdas in order to support proxy configurations. Because of this change, i've deprecated the original `onEventLayer` and created a new prop `proxyAgentLayer` since we will now be passing this prop into more than just the `onEventHandler` Lambda.

The `onEventLayer` prop was introduced [a few weeks ago (sept 24)](#16657) so it should not impact many users (if any). The prop would only be used if the user wishes to bundle the layer themselves with a custom proxy agent. 

This prop follows the [same user customization we allow with the kubectl handler](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-eks.Cluster.html#kubectllayer). 

Another suitable name for this prop could have been `clusterHandlerLayer` but I chose `proxyAgentLayer` because it represents **what** the layer is used for, instead of describing **where** it's used. This also follows the convention of the pre-existing [`kubectlLayer` prop](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-eks.Cluster.html#kubectllayer).

</details>

<details>
<summary><strong>Adds the EKS cluster prop <code>clusterHandlerSecurityGroup</code></strong></summary>
<br />

If a proxy address is provided to the Cluster Handler Lambdas, but the proxy instance is not open to the world, then the dynamic IPs of the Cluster Handler Lambdas will be denied access. To solve this, i've implemented a new Cluster prop `clusterHandlerSecurityGroup`. This `clusterHandlerSecurityGroup` prop will allow the user to pass a Security Group to both Lambda functions and the Custom Resource provider. 

This is very similar to how we [already allow users to pass Security Groups to the Kubectl Handler](https://github.com/aws/aws-cdk/blob/7f194000697b85deb410ae0d7f7d4ac3c2654bcc/packages/%40aws-cdk/aws-eks/lib/kubectl-provider.ts#L83)

</details>

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
  • Loading branch information
@ryparker
ryparker authored Nov 5, 2021
1 parent c66ac89 commit 7bbd10d
Show file tree
Hide file tree
Showing 8 changed files with 145 additions and 51 deletions.
17 changes: 11 additions & 6 deletions packages/@aws-cdk/aws-eks/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -102,8 +102,8 @@ The following is a qualitative diagram of the various possible components involv

```text
+-----------------------------------------------+ +-----------------+
| EKS Cluster | kubectl | |
|-----------------------------------------------|<-------------+| Kubectl Handler |
| EKS Cluster | kubectl | |
| ----------- |<-------------+| Kubectl Handler |
| | | |
| | +-----------------+
| +--------------------+ +-----------------+ |
Expand Down Expand Up @@ -539,16 +539,21 @@ If the endpoint does not expose private access (via `EndpointAccess.PUBLIC`) **o

#### Cluster Handler

The `ClusterHandler` is a Lambda function responsible to interact with the EKS API in order to control the cluster lifecycle. To provision this function inside the VPC, set the `placeClusterHandlerInVpc` property to `true`. This will place the function inside the private subnets of the VPC based on the selection strategy specified in the [`vpcSubnets`](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-eks.Cluster.html#vpcsubnetsspan-classapi-icon-api-icon-experimental-titlethis-api-element-is-experimental-it-may-change-without-noticespan) property.
The `ClusterHandler` is a set of Lambda functions (`onEventHandler`, `isCompleteHandler`) responsible for interacting with the EKS API in order to control the cluster lifecycle. To provision these functions inside the VPC, set the `placeClusterHandlerInVpc` property to `true`. This will place the functions inside the private subnets of the VPC based on the selection strategy specified in the [`vpcSubnets`](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-eks.Cluster.html#vpcsubnetsspan-classapi-icon-api-icon-experimental-titlethis-api-element-is-experimental-it-may-change-without-noticespan) property.

You can configure the environment of this function by specifying it at cluster instantiation. For example, this can be useful in order to configure an http proxy:
You can configure the environment of the Cluster Handler functions by specifying it at cluster instantiation. For example, this can be useful in order to configure an http proxy:

```ts
const cluster = new eks.Cluster(this, 'hello-eks', {
version: eks.KubernetesVersion.V1_21,
clusterHandlerEnvironment: {
'http_proxy': 'http://proxy.myproxy.com'
}
https_proxy: 'http://proxy.myproxy.com'
},
/**
* If the proxy is not open publicly, you can pass a security group to the
* Cluster Handler Lambdas so that it can reach the proxy.
*/
clusterHandlerSecurityGroup: proxyInstanceSecurityGroup
});
```

Expand Down
6 changes: 0 additions & 6 deletions packages/@aws-cdk/aws-eks/lib/cluster-resource-handler/common.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -41,12 +41,6 @@ export abstract class ResourceHandler {
}

public onEvent() {
// eslint-disable-next-line @typescript-eslint/no-require-imports, import/no-extraneous-dependencies
const ProxyAgent: any = require('proxy-agent');
aws.config.update({
httpOptions: { agent: new ProxyAgent() },
});

switch (this.requestType) {
case 'Create': return this.onCreate();
case 'Update': return this.onUpdate();
Expand Down
6 changes: 6 additions & 0 deletions packages/@aws-cdk/aws-eks/lib/cluster-resource-handler/index.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,13 @@ import { EksClient } from './common';
import * as consts from './consts';
import { FargateProfileResourceHandler } from './fargate';

// eslint-disable-next-line @typescript-eslint/no-require-imports, import/no-extraneous-dependencies
const ProxyAgent = require('proxy-agent');

aws.config.logger = console;
aws.config.update({
httpOptions: { agent: new ProxyAgent() },
});

let eks: aws.EKS | undefined;

Expand Down
33 changes: 21 additions & 12 deletions packages/@aws-cdk/aws-eks/lib/cluster-resource-provider.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -36,11 +36,18 @@ export interface ClusterResourceProviderProps {
readonly environment?: { [key: string]: string };

/**
* An AWS Lambda layer that includes the NPM dependency `proxy-agent`.
*
* If not defined, a default layer will be used.
*/
* An AWS Lambda layer that includes the NPM dependency `proxy-agent`.
*
* If not defined, a default layer will be used.
*/
readonly onEventLayer?: lambda.ILayerVersion;

/**
* The security group to associate with the functions.
*
* @default - No security group.
*/
readonly securityGroup?: ec2.ISecurityGroup;
}

/**
Expand All @@ -66,6 +73,9 @@ export class ClusterResourceProvider extends NestedStack {
private constructor(scope: Construct, id: string, props: ClusterResourceProviderProps) {
super(scope as CoreConstruct, id);

// The NPM dependency proxy-agent is required in order to support proxy routing with the AWS JS SDK.
const nodeProxyAgentLayer = new NodeProxyAgentLayer(this, 'NodeProxyAgentLayer');

const onEvent = new lambda.Function(this, 'OnEventHandler', {
code: lambda.Code.fromAsset(HANDLER_DIR),
description: 'onEvent handler for EKS cluster resource provider',
Expand All @@ -75,24 +85,22 @@ export class ClusterResourceProvider extends NestedStack {
timeout: Duration.minutes(1),
vpc: props.subnets ? props.vpc : undefined,
vpcSubnets: props.subnets ? { subnets: props.subnets } : undefined,
securityGroups: props.securityGroup ? [props.securityGroup] : undefined,
// Allow user to override the layer.
layers: props.onEventLayer ? [props.onEventLayer] : [nodeProxyAgentLayer],
});

// Allow user to customize the layer
if (!props.onEventLayer) {
// `NodeProxyAgentLayer` provides `proxy-agent` which is needed to configure `aws-sdk-js` with a user provided proxy.
onEvent.addLayers(new NodeProxyAgentLayer(this, 'NodeProxyAgentLayer'));
} else {
onEvent.addLayers(props.onEventLayer);
}

const isComplete = new lambda.Function(this, 'IsCompleteHandler', {
code: lambda.Code.fromAsset(HANDLER_DIR),
description: 'isComplete handler for EKS cluster resource provider',
runtime: HANDLER_RUNTIME,
environment: props.environment,
handler: 'index.isComplete',
timeout: Duration.minutes(1),
vpc: props.subnets ? props.vpc : undefined,
vpcSubnets: props.subnets ? { subnets: props.subnets } : undefined,
securityGroups: props.securityGroup ? [props.securityGroup] : undefined,
layers: [nodeProxyAgentLayer],
});

this.provider = new cr.Provider(this, 'Provider', {
Expand All @@ -102,6 +110,7 @@ export class ClusterResourceProvider extends NestedStack {
queryInterval: Duration.minutes(1),
vpc: props.subnets ? props.vpc : undefined,
vpcSubnets: props.subnets ? { subnets: props.subnets } : undefined,
securityGroups: props.securityGroup ? [props.securityGroup] : undefined,
});

props.adminRole.grant(onEvent.role!, 'sts:AssumeRole');
Expand Down
2 changes: 2 additions & 0 deletions packages/@aws-cdk/aws-eks/lib/cluster-resource.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ export interface ClusterResourceProps {
readonly subnets?: ec2.ISubnet[];
readonly secretsEncryptionKey?: kms.IKey;
readonly onEventLayer?: lambda.ILayerVersion;
readonly clusterHandlerSecurityGroup?: ec2.ISecurityGroup;
}

/**
Expand Down Expand Up @@ -66,6 +67,7 @@ export class ClusterResource extends CoreConstruct {
vpc: props.vpc,
environment: props.environment,
onEventLayer: props.onEventLayer,
securityGroup: props.clusterHandlerSecurityGroup,
});

const resource = new CustomResource(this, 'Resource', {
Expand Down
96 changes: 71 additions & 25 deletions packages/@aws-cdk/aws-eks/lib/cluster.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -130,10 +130,21 @@ export interface ICluster extends IResource, ec2.IConnectable {
readonly kubectlMemory?: Size;

/**
* An AWS Lambda layer that includes the NPM dependency `proxy-agent`.
*
* If not defined, a default layer will be used.
*/
* A security group to associate with the Cluster Handler's Lambdas.
* The Cluster Handler's Lambdas are responsible for calling AWS's EKS API.
*
* Requires `placeClusterHandlerInVpc` to be set to true.
*
* @default - No security group.
* @attribute
*/
readonly clusterHandlerSecurityGroup?: ec2.ISecurityGroup;

/**
* An AWS Lambda layer that includes the NPM dependency `proxy-agent`.
*
* If not defined, a default layer will be used.
*/
readonly onEventLayer?: lambda.ILayerVersion;

/**
Expand Down Expand Up @@ -309,6 +320,14 @@ export interface ClusterAttributes {
*/
readonly kubectlMemory?: Size;

/**
* A security group id to associate with the Cluster Handler's Lambdas.
* The Cluster Handler's Lambdas are responsible for calling AWS's EKS API.
*
* @default - No security group.
*/
readonly clusterHandlerSecurityGroupId?: string;

/**
* An AWS Lambda Layer which includes the NPM dependency `proxy-agent`. This layer
* is used by the onEvent handler to route AWS SDK requests through a proxy.
Expand Down Expand Up @@ -452,13 +471,6 @@ export interface ClusterOptions extends CommonClusterOptions {
*/
readonly kubectlEnvironment?: { [key: string]: string };

/**
* Custom environment variables when interacting with the EKS endpoint to manage the cluster lifecycle.
*
* @default - No environment variables.
*/
readonly clusterHandlerEnvironment?: { [key: string]: string };

/**
* An AWS Lambda Layer which includes `kubectl`, Helm and the AWS CLI.
*
Expand Down Expand Up @@ -491,26 +503,40 @@ export interface ClusterOptions extends CommonClusterOptions {
readonly kubectlMemory?: Size;

/**
* An AWS Lambda Layer which includes the NPM dependency `proxy-agent`.
* Custom environment variables when interacting with the EKS endpoint to manage the cluster lifecycle.
*
* @default - No environment variables.
*/
readonly clusterHandlerEnvironment?: { [key: string]: string };

/**
* A security group to associate with the Cluster Handler's Lambdas.
* The Cluster Handler's Lambdas are responsible for calling AWS's EKS API.
*
* Requires `placeClusterHandlerInVpc` to be set to true.
*
* @default - No security group.
*/
readonly clusterHandlerSecurityGroup?: ec2.ISecurityGroup;

/**
* An AWS Lambda Layer which includes the NPM dependency `proxy-agent`. This layer
* is used by the onEvent handler to route AWS SDK requests through a proxy.
*
* By default, the provider will use the layer included in the
* "aws-lambda-layer-node-proxy-agent" SAR application which is available in all
* commercial regions.
*
* To deploy the layer locally, visit
* https://github.com/aws-samples/aws-lambda-layer-node-proxy-agent/blob/master/cdk/README.md
* for instructions on how to prepare the .zip file and then define it in your
* app as follows:
* To deploy the layer locally define it in your app as follows:
*
* ```ts
* const layer = new lambda.LayerVersion(this, 'node-proxy-agent-layer', {
* const layer = new lambda.LayerVersion(this, 'proxy-agent-layer', {
* code: lambda.Code.fromAsset(`${__dirname}/layer.zip`)),
* compatibleRuntimes: [lambda.Runtime.NODEJS_14_X]
* compatibleRuntimes: [lambda.Runtime.NODEJS_12_X]
* })
* ```
*
* @default - the layer provided by the `aws-lambda-layer-node-proxy-agent` SAR app.
* @see https://github.com/aws-samples/aws-lambda-layer-node-proxy-agent
* @default - a layer bundled with this module.
*/
readonly onEventLayer?: lambda.ILayerVersion;

Expand Down Expand Up @@ -749,6 +775,7 @@ abstract class ClusterBase extends Resource implements ICluster {
public abstract readonly kubectlSecurityGroup?: ec2.ISecurityGroup;
public abstract readonly kubectlPrivateSubnets?: ec2.ISubnet[];
public abstract readonly kubectlMemory?: Size;
public abstract readonly clusterHandlerSecurityGroup?: ec2.ISecurityGroup;
public abstract readonly prune: boolean;
public abstract readonly openIdConnectProvider: iam.IOpenIdConnectProvider;
public abstract readonly awsAuth: AwsAuth;
Expand Down Expand Up @@ -902,7 +929,7 @@ abstract class ClusterBase extends Resource implements ICluster {
// cluster or if `mapRole` is set to false. By default this should happen.
let mapRole = options.mapRole ?? true;
if (mapRole && !(this instanceof Cluster)) {
// do the mapping...
// do the mapping...
Annotations.of(autoScalingGroup).addWarning('Auto-mapping aws-auth role for imported cluster is not supported, please map role manually');
mapRole = false;
}
Expand Down Expand Up @@ -1099,11 +1126,21 @@ export class Cluster extends ClusterBase {
*/
public readonly kubectlMemory?: Size;

/**
* A security group to associate with the Cluster Handler's Lambdas.
* The Cluster Handler's Lambdas are responsible for calling AWS's EKS API.
*
* Requires `placeClusterHandlerInVpc` to be set to true.
*
* @default - No security group.
*/
public readonly clusterHandlerSecurityGroup?: ec2.ISecurityGroup;

/**
* The AWS Lambda layer that contains the NPM dependency `proxy-agent`. If
* undefined, a SAR app that contains this layer will be used.
*/
public readonly onEventLayer?: lambda.ILayerVersion;
readonly onEventLayer?: lambda.ILayerVersion;

/**
* Determines if Kubernetes resources can be pruned automatically.
Expand Down Expand Up @@ -1188,9 +1225,11 @@ export class Cluster extends ClusterBase {
this.endpointAccess = props.endpointAccess ?? EndpointAccess.PUBLIC_AND_PRIVATE;
this.kubectlEnvironment = props.kubectlEnvironment;
this.kubectlLayer = props.kubectlLayer;
this.onEventLayer = props.onEventLayer;
this.kubectlMemory = props.kubectlMemory;

this.onEventLayer = props.onEventLayer;
this.clusterHandlerSecurityGroup = props.clusterHandlerSecurityGroup;

const privateSubnets = this.selectPrivateSubnets().slice(0, 16);
const publicAccessDisabled = !this.endpointAccess._config.publicAccess;
const publicAccessRestricted = !publicAccessDisabled
Expand All @@ -1215,6 +1254,10 @@ export class Cluster extends ClusterBase {
throw new Error('Cannot place cluster handler in the VPC since no private subnets could be selected');
}

if (props.clusterHandlerSecurityGroup && !placeClusterHandlerInVpc) {
throw new Error('Cannot specify clusterHandlerSecurityGroup without placeClusterHandlerInVpc set to true');
}

const resource = this._clusterResource = new ClusterResource(this, 'Resource', {
name: this.physicalName,
environment: props.clusterHandlerEnvironment,
Expand All @@ -1241,6 +1284,7 @@ export class Cluster extends ClusterBase {
secretsEncryptionKey: props.secretsEncryptionKey,
vpc: this.vpc,
subnets: placeClusterHandlerInVpc ? privateSubnets : undefined,
clusterHandlerSecurityGroup: this.clusterHandlerSecurityGroup,
onEventLayer: this.onEventLayer,
});

Expand Down Expand Up @@ -1827,8 +1871,9 @@ class ImportedCluster extends ClusterBase {
public readonly kubectlSecurityGroup?: ec2.ISecurityGroup | undefined;
public readonly kubectlPrivateSubnets?: ec2.ISubnet[] | undefined;
public readonly kubectlLayer?: lambda.ILayerVersion;
public readonly onEventLayer?: lambda.ILayerVersion;
public readonly kubectlMemory?: Size;
public readonly clusterHandlerSecurityGroup?: ec2.ISecurityGroup | undefined;
public readonly onEventLayer?: lambda.ILayerVersion;
public readonly prune: boolean;

// so that `clusterSecurityGroup` on `ICluster` can be configured without optionality, avoiding users from having
Expand All @@ -1845,8 +1890,9 @@ class ImportedCluster extends ClusterBase {
this.kubectlEnvironment = props.kubectlEnvironment;
this.kubectlPrivateSubnets = props.kubectlPrivateSubnetIds ? props.kubectlPrivateSubnetIds.map((subnetid, index) => ec2.Subnet.fromSubnetId(this, `KubectlSubnet${index}`, subnetid)) : undefined;
this.kubectlLayer = props.kubectlLayer;
this.onEventLayer = props.onEventLayer;
this.kubectlMemory = props.kubectlMemory;
this.clusterHandlerSecurityGroup = props.clusterHandlerSecurityGroupId ? ec2.SecurityGroup.fromSecurityGroupId(this, 'ClusterHandlerSecurityGroup', props.clusterHandlerSecurityGroupId) : undefined;
this.onEventLayer = props.onEventLayer;
this.prune = props.prune ?? true;

let i = 1;
Expand Down
2 changes: 0 additions & 2 deletions packages/@aws-cdk/aws-eks/package.json
View file
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,6 @@
"@aws-cdk/aws-iam": "0.0.0",
"@aws-cdk/aws-kms": "0.0.0",
"@aws-cdk/aws-lambda": "0.0.0",
"@aws-cdk/aws-lambda-nodejs": "0.0.0",
"@aws-cdk/aws-ssm": "0.0.0",
"@aws-cdk/core": "0.0.0",
"@aws-cdk/custom-resources": "0.0.0",
Expand All @@ -113,7 +112,6 @@
"@aws-cdk/aws-iam": "0.0.0",
"@aws-cdk/aws-kms": "0.0.0",
"@aws-cdk/aws-lambda": "0.0.0",
"@aws-cdk/aws-lambda-nodejs": "0.0.0",
"@aws-cdk/aws-ssm": "0.0.0",
"@aws-cdk/core": "0.0.0",
"@aws-cdk/custom-resources": "0.0.0",
Expand Down
Loading

0 comments on commit 7bbd10d

Please sign in to comment.