Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(eks): pass helm chart values to aws-load-balancer-controller #29723

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
/// !cdk-integ pragma:disable-update-workflow
import * as ec2 from 'aws-cdk-lib/aws-ec2';
import { App, CfnOutput, Duration, Stack } from 'aws-cdk-lib';
import { App, CfnOutput, Duration, Stack, StackProps } from 'aws-cdk-lib';
import * as integ from '@aws-cdk/integ-tests-alpha';
import * as cdk8s from 'cdk8s';
import * as kplus from 'cdk8s-plus-27';
Expand All @@ -9,10 +9,15 @@ import { Pinger } from './pinger/pinger';
import * as eks from 'aws-cdk-lib/aws-eks';

const LATEST_VERSION: eks.AlbControllerVersion = eks.AlbControllerVersion.V2_8_2;

interface EksClusterAlbControllerStackProps extends StackProps {
albControllerHelmChartValues?: {[key:string]: any};
}

class EksClusterAlbControllerStack extends Stack {

constructor(scope: App, id: string) {
super(scope, id);
constructor(scope: App, id: string, props: EksClusterAlbControllerStackProps = {}) {
super(scope, id, props);

// just need one nat gateway to simplify the test
const vpc = new ec2.Vpc(this, 'Vpc', { maxAzs: 2, natGateways: 1, restrictDefaultSecurityGroup: false });
Expand All @@ -22,6 +27,7 @@ class EksClusterAlbControllerStack extends Stack {
...getClusterVersionConfig(this, eks.KubernetesVersion.V1_30),
albController: {
version: LATEST_VERSION,
helmChartValues: props.albControllerHelmChartValues,
},
});

Expand Down Expand Up @@ -71,8 +77,9 @@ class EksClusterAlbControllerStack extends Stack {

const app = new App();
const stack = new EksClusterAlbControllerStack(app, 'aws-cdk-eks-cluster-alb-controller');
const stackWithAlbControllerValues = new EksClusterAlbControllerStack(app, 'aws-cdk-eks-cluster-alb-controller-values', { albControllerHelmChartValues: { enableWafv2: false } });
new integ.IntegTest(app, 'aws-cdk-cluster-alb-controller-integ', {
testCases: [stack],
testCases: [stack, stackWithAlbControllerValues],
// Test includes assets that are updated weekly. If not disabled, the upgrade PR will fail.
diffAssets: false,
});
Expand Down
19 changes: 19 additions & 0 deletions packages/aws-cdk-lib/aws-eks/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -691,6 +691,25 @@ if (cluster.albController) {
}
```

If you need to configure the underlying ALB Controller, you can pass optional values that will be forwarded to the underling HelmChart construct.
For example, if your organization uses AWS Firewall Manager you will need to disable aws-load-balancer-controller's waf modification behavior or it will periodically disassociate WAFs that are not specified in the Ingress' annotations.

Note: supported value options may vary depending on the version of the aws-load-balancer-controller helm chart you are using. You should consult the [ArtifactHub documentation](https://artifacthub.io/packages/helm/aws/aws-load-balancer-controller) for your version to ensure you are configuring the chart correctly.

```ts
new eks.Cluster(this, 'HelloEKS', {
version: eks.KubernetesVersion.V1_29,
albController: {
version: eks.AlbControllerVersion.V2_6_2,
helmChartValues: {
enableWaf: false,
enableWafv2: false,
},
},
});
```


### VPC Support

You can specify the VPC of the cluster using the `vpc` and `vpcSubnets` properties:
Expand Down
43 changes: 43 additions & 0 deletions packages/aws-cdk-lib/aws-eks/lib/alb-controller.ts
Original file line number Diff line number Diff line change
Expand Up @@ -271,6 +271,27 @@ export interface AlbControllerOptions {
* @default - Corresponds to the predefined version.
*/
readonly policy?: any;

/**
* Override values to be used by the chart.
* For nested values use a nested dictionary. For example:
* helmChartValues: {
* autoscaling: false,
* ingressClassParams: { create: true }
* }
*
* Note that the following values are set by the controller and cannot be overridden:
* - clusterName
* - serviceAccount.create
* - serviceAccount.name
* - region
* - vpcId
* - image.repository
* - image.tag
*
* @default - No values are provided to the chart.
*/
readonly helmChartValues?: {[key: string]: any};
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think that this is quite the right direction here. From what I can tell, only one or two values need to be overridden for your use case and this is basically giving the user carte blanche to override far more than that. I chatted with @paulhcsun about this PR and I'm thinking that it would likely be better to have a function that can be called after instantiation to disable the load balancer wafv2 behavior instead of adding a prop.

In general, we don't want props to contradict one another or allow values that we know are not allowed (i.e. the list of values you added in the docstring).

What I'm really having trouble with here is understanding how those values are being set in the first place, in your use case. Can you explain what's going on in your specific case a bit more so that our suggestions for the fix don't send you in circles?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would also probably suggest that the function to update this behavior should be scoped to the ALB controller but, again, I'd like to understand better how these fields are being set before saying you should make this code change.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hi @TheRealAmazonKendra thanks for your review.

you are correct that for my specific use case I only need to update one or two values, but my intention with this PR is to address the issue of aws-cdk-lib/aws-eks package not exposing the ability to set whatever values the user wants (apart from the ones it is already setting).

the AlbController is just a thin wrapper around the HelmChart construct, yet its constructor's props do not expose any ability to pass values to the underlying helm chart. IMO this was a poor design choice, as I don't think aws-cdk should be the arbiter of which values can be passed to the aws-load-balancer-controller helm chart.

the initial implementation of the Cluster construct did not include any props related to the albController. They were later added as a convenience due to the fact that it is commonly utilized to integrate with aws elbv2. my assumption is that the AlbController construct was defined at the same time, due to the fact that AlbControllerOptions = Omit<AlbControllerProps, 'cluster'>. However, the lack of an interface exposing configuration of the helm chart's values means that any user leveraging aws firewall manager either cannot use the "official" construct for the helm chart, or they have to accept that aws-load-balancer-controller will periodically disassociate an fms-applied web acl and their albs willbe unprotected until fms can detect & remediate (usually this is a couple minutes, but we have had occurrences of the remediation failing and albs being unprotected for hours).

Due to the fact that a helm chart's values.yml is the way in which every helm chart's default behavior is overridden, i feel that adding a prop for passing values to the AlbController is a valid design choice. There may be other default behaviors of the helm chart beyond waf management that users may want to disable. Due to the fact that the supported values for aws-load-balancer-controller may vary from one version to the next, I didn't feel it was wise to define a type/interface for a subset of configurable values that will then need to be kept up-to-date. I think if users decide to pass values to the AlbController, they are at the point where they know they need to and we should trust that they are able to decide which values to set.

as far as your suggestion for a function to update values after instantiation, this would need to be exposed on both the AlbController construct and the HelmChart construct. i feel like this is not a great approach and it expands the scope of the solution beyond the AlbController construct.

in short, i feel that this is a fix for an overly-restrictive construct props definition.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hello, I would also want to be able to edit values through this way, in my case for values enableCertManager, cluster.dnsDomain and some more.
I wonder why this functionality wasn't introduced from the beginning...

}

/**
Expand Down Expand Up @@ -331,6 +352,8 @@ export class AlbController extends Construct {
}

// https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.2/deploy/installation/#add-controller-to-cluster
const { helmChartValues = {} } = props;
this.validateHelmChartValues(helmChartValues);
const chart = new HelmChart(this, 'Resource', {
cluster: props.cluster,
chart: 'aws-load-balancer-controller',
Expand All @@ -342,6 +365,8 @@ export class AlbController extends Construct {
wait: true,
timeout: Duration.minutes(15),
values: {
...helmChartValues,
// if you modify these values, you must also update this.restrictedHelmChartValueKeys
clusterName: props.cluster.clusterName,
serviceAccount: {
create: false,
Expand Down Expand Up @@ -380,4 +405,22 @@ export class AlbController extends Construct {
}
return resources.map(rewriteResource);
}

private validateHelmChartValues(values: {[key: string]: any}) {
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@paulhcsun i have already implemented the validation as a function that restricts the keys the user is allowed to modify. do you mean that overriding the values should be an instance method? this does not make sense to me and does not align with how other properties of the AlbController HelmChart wrapper are set. for example, there is no albController.updateVersion() method, the helm chart version is passed in the constructor.

an instance method would also require updating the HelmChart construct to support modifying values after initialization (the AlbController constructor initializes the underlying HelmChart construct), which again makes the solution more complicated and expands beyond the scope of the AlbController - the HelmChart construct does not need to support value modification as it allows for passing arbitrary values already.

let me know if any modifications to this implementation are required and i'll be happy to make them.

const valuesKeySet = new Set(Object.keys(values));
const invalidKeys = new Set([...valuesKeySet].filter((key) => this.restrictedHelmChartValueKeys.has(key)));
if (invalidKeys.size > 0) {
throw new Error(`The following aws-load-balancer-controller HelmChart value keys are restricted and cannot be overridden: ${Array.from(this.restrictedHelmChartValueKeys).join(', ')}. (Invalid keys: ${Array.from(invalidKeys).join(', ')})`);
}
}

private get restrictedHelmChartValueKeys(): Set<string> {
return new Set([
'clusterName',
'serviceAccount',
'region',
'vpcId',
'image',
]);
}
}
64 changes: 64 additions & 0 deletions packages/aws-cdk-lib/aws-eks/test/alb-controller.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -153,6 +153,70 @@ test('correct helm chart version is set for selected alb controller version', ()
});
});

test('can pass values to the aws-load-balancer-controller helm chart', () => {
const { stack } = testFixture();

const cluster = new Cluster(stack, 'Cluster', {
version: KubernetesVersion.V1_27,
});

AlbController.create(stack, {
cluster,
version: AlbControllerVersion.V2_6_2,
repository: 'custom',
helmChartValues: {
enableWafv2: false,
},
});

Template.fromStack(stack).hasResourceProperties(HelmChart.RESOURCE_TYPE, {
Version: '1.6.2', // The helm chart version associated with AlbControllerVersion.V2_6_2
Values: {
'Fn::Join': [
'',
[
'{"enableWafv2":false,"clusterName":"',
{
Ref: 'Cluster9EE0221C',
},
'","serviceAccount":{"create":false,"name":"aws-load-balancer-controller"},"region":"us-east-1","vpcId":"',
{
Ref: 'ClusterDefaultVpcFA9F2722',
},
'","image":{"repository":"custom","tag":"v2.6.2"}}',
],
],
},
});
});

test('values passed to the aws-load-balancer-controller result in an error if they conflict with restricted values set by the construct', () => {
const { stack } = testFixture();

const cluster = new Cluster(stack, 'Cluster', {
version: KubernetesVersion.V1_27,
});

expect(() => AlbController.create(stack, {
cluster,
version: AlbControllerVersion.V2_6_2,
repository: 'custom',
helmChartValues: {
clusterName: 'test-cluster',
serviceAccount: {
create: true,
name: 'test-service-account',
},
region: 'test-region',
vpcId: 'test-vpc-id',
image: {
repository: 'test-repository',
tag: 'test-tag',
},
},
})).toThrow(/The following aws-load-balancer-controller HelmChart value keys are restricted and cannot be overridden: .*/);
});

describe('AlbController AwsAuth creation', () => {
const setupTest = (authenticationMode?: AuthenticationMode) => {
const { stack } = testFixture();
Expand Down
Loading