aws-eks: addNodegroupCapacity with standard AMI, custom launch template, and kubelet-extra-args

[JohnYoungers]

Describe the feature

For our NodeGroups, we're providing a custom launch template to activate the EC2's instance store volume:

const launchTemplate = new ec2.CfnLaunchTemplate(this, "lt", {
  launchTemplateData: {
    ...,
    userData: cdk.Fn.base64(`MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="==MYBOUNDARY=="

--==MYBOUNDARY==
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash
echo "Attempting to mount instance store"
if ( lsblk | fgrep -q nvme1n1 ); then
mkdir -p /mnt/data /var/lib/kubelet /var/lib/containerd
mkfs.xfs /dev/nvme1n1
mount /dev/nvme1n1 /mnt/data
chmod 0755 /mnt/data
mv /var/lib/kubelet /mnt/data/
mv /var/lib/containerd/mnt/data/
ln -sf /mnt/data/kubelet /var/lib/kubelet
ln -sf /mnt/data/containerd /var/lib/containerd
fi

--==MYBOUNDARY==--\\
`),
  },
});

Which is then referenced like so:

this.cluster.addNodegroupCapacity("ng", {
  ...,
  launchTemplateSpec: {
   id: launchTemplate.ref,
   version: launchTemplate.attrLatestVersionNumber,
  },
});

The actual launch template user data will also contain this:

#!/bin/bash
set -ex
B64_CLUSTER_CA=...
API_SERVER_URL=...
K8S_CLUSTER_DNS_IP=10.100.0.10
/etc/eks/bootstrap.sh ... --kubelet-extra-args '--node-labels=...,eks.amazonaws.com/capacityType=...,eks.amazonaws.com/nodegroup=...,kube/nodetype=...,eks.amazonaws.com/sourceLaunchTemplateId=... --b64-cluster-ca $B64_CLUSTER_CA --apiserver-endpoint $API_SERVER_URL --dns-cluster-ip $K8S_CLUSTER_DNS_IP --use-max-pods false

For this scenario, what's the best way to provide additional flags for kubelet-extra-args? In the documentation it looked like the example pertained to custom AMIs opposed to the standard

Use Case

I'm looking to provide --logging-format=json, but I'm not sure the best route to go with the standard AMI with a custom launch template

Proposed Solution

No response

Other Information

No response

Acknowledgements

• I may be able to implement this feature request
• This feature might incur a breaking change

CDK version used

2.84.0

Environment details (OS name and version, etc.)

linux

[indrora]

The "official" way of doing this is through the use of BootstrapOptions: https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_eks.BootstrapOptions.html#kubeletextraargs

This is passed to addAutoScalingGroupCapacity: https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_eks.Cluster.html#addwbrautowbrscalingwbrgroupwbrcapacityid-options

Is there a reason you're using addNodegroupCapacity rather than addAutoScalingGroupCapacity?

[JohnYoungers]

Is there a reason you're using addNodegroupCapacity rather than addAutoScalingGroupCapacity?

The primary reason is what's noted in the message regarding setting up the EC2's instance store, but looking at the bootstrap script I'm guessing the flag --local-disks is doing something similar: https://github.com/awslabs/amazon-eks-ami/blob/main/templates/al2/runtime/bootstrap.sh#L35

The secondary reason we're using addNodegroupCapacity is to be able to specify more than one instance type for the pool:

const allowedInstanceTypes= [...];
this.cluster.addNodegroupCapacity("ng", {
    instanceTypes: allowedInstanceTypes.map((i) => new ec2.InstanceType(`${i}.2xlarge`)),
    capacityType: eks.CapacityType.SPOT,
  });

Assuming that flag sets up the instance store, I think being able to specify multiple instance types would be the only thing preventing us from using addAutoScalingGroupCapacity

[JohnYoungers]

@pahud - in issue #29983 (comment) you note addNodeGroupCapacity should likely be the default option: if that's the case, is it possible to create a group that has the instance store enabled as well as providing custom bootstrap options (setting logging format, verbosity, etc)?

[pahud]

@JohnYoungers

What addNodeGroupCapacity does is essentially new NodeGroup() for you. With that being said, the most flexible way I'd recommend is still new NodeGroup() with its launchTemplate support. This essentially could allow you to customize everything with a custom AMI. The tradeoff is that you need to bake your own LaunchTemplate as well as the UserData and that's a bit beyond the scope of CDK. The challenge would be configuring a custom LaunchTemplate and UserData for that.

[github-actions]

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.