[eks/karpenter] Add support for kubelet config, fix IAM support for…

… `v1alpha` cleanup (cloudposse/terraform-aws-components#1076)

src/CHANGELOG.md

@@ -0,0 +1,15 @@
+## Components [PR #1076](https://github.com/cloudposse/terraform-aws-components/pull/1076)
+
+- Allow specifying elements of `spec.template.spec.kubelet`
+- Make taint values optional
+
+The `var.node_pools` map now includes a `kubelet` field that allows specifying elements of `spec.template.spec.kubelet`.
+This is useful for configuring the kubelet to use custom settings, such as reserving resources for system daemons.
+
+For more information, see:
+
+- [Karpenter documentation](https://karpenter.sh/docs/concepts/nodepools/#spectemplatespeckubelet)
+- [Kubernetes documentation](https://kubernetes.io/docs/reference/config-api/kubelet-config.v1beta1/)
+
+The `value` fields of the `taints` and `startup_taints` lists in the `var.node_pools` map are now optional. This is in
+alignment with the Kubernetes API, where `key` and `effect` are required, but the `value` field is optional.

src/README.md

@@ -203,7 +203,7 @@ components:
 | <a name="input_labels_as_tags"></a> [labels\_as\_tags](#input\_labels\_as\_tags) | Set of labels (ID elements) to include as tags in the `tags` output.<br>Default is to include all labels.<br>Tags with empty values will not be included in the `tags` output.<br>Set to `[]` to suppress all generated tags.<br>**Notes:**<br>  The value of the `name` tag, if included, will be the `id`, not the `name`.<br>  Unlike other `null-label` inputs, the initial setting of `labels_as_tags` cannot be<br>  changed in later chained modules. Attempts to change it will be silently ignored. | `set(string)` | <pre>[<br>  "default"<br>]</pre> | no |
 | <a name="input_name"></a> [name](#input\_name) | ID element. Usually the component or solution name, e.g. 'app' or 'jenkins'.<br>This is the only ID element not also included as a `tag`.<br>The "name" tag is set to the full `id` string. There is no tag with the value of the `name` input. | `string` | `null` | no |
 | <a name="input_namespace"></a> [namespace](#input\_namespace) | ID element. Usually an abbreviation of your organization name, e.g. 'eg' or 'cp', to help ensure generated IDs are globally unique | `string` | `null` | no |
-| <a name="input_node_pools"></a> [node\_pools](#input\_node\_pools) | Configuration for node pools. See code for details. | <pre>map(object({<br>    # The name of the Karpenter provisioner. The map key is used if this is not set.<br>    name = optional(string)<br>    # Whether to place EC2 instances launched by Karpenter into VPC private subnets. Set it to `false` to use public subnets.<br>    private_subnets_enabled = bool<br>    # The Disruption spec controls how Karpenter scales down the node group.<br>    # See the example (sadly not the specific `spec.disruption` documentation) at https://karpenter.sh/docs/concepts/nodepools/ for details<br>    disruption = optional(object({<br>      # Describes which types of Nodes Karpenter should consider for consolidation.<br>      # If using 'WhenUnderutilized', Karpenter will consider all nodes for consolidation and attempt to remove or<br>      # replace Nodes when it discovers that the Node is underutilized and could be changed to reduce cost.<br>      # If using `WhenEmpty`, Karpenter will only consider nodes for consolidation that contain no workload pods.<br>      consolidation_policy = optional(string, "WhenUnderutilized")<br><br>      # The amount of time Karpenter should wait after discovering a consolidation decision (`go` duration string, s, m, or h).<br>      # This value can currently (v0.36.0) only be set when the consolidationPolicy is 'WhenEmpty'.<br>      # You can choose to disable consolidation entirely by setting the string value 'Never' here.<br>      # Earlier versions of Karpenter called this field `ttl_seconds_after_empty`.<br>      consolidate_after = optional(string)<br><br>      # The amount of time a Node can live on the cluster before being removed (`go` duration string, s, m, or h).<br>      # You can choose to disable expiration entirely by setting the string value 'Never' here.<br>      # This module sets a default of 336 hours (14 days), while the Karpenter default is 720 hours (30 days).<br>      # Note that Karpenter calls this field "expiresAfter", and earlier versions called it `ttl_seconds_until_expired`,<br>      # but we call it "max_instance_lifetime" to match the corresponding field in EC2 Auto Scaling Groups.<br>      max_instance_lifetime = optional(string, "336h")<br><br>      # Budgets control the the maximum number of NodeClaims owned by this NodePool that can be terminating at once.<br>      # See https://karpenter.sh/docs/concepts/disruption/#disruption-budgets for details.<br>      # A percentage is the percentage of the total number of active, ready nodes not being deleted, rounded up.<br>      # If there are multiple active budgets, Karpenter uses the most restrictive value.<br>      # If left undefined, this will default to one budget with a value of nodes: 10%.<br>      # Note that budgets do not prevent or limit involuntary terminations.<br>      # Example:<br>      #   On Weekdays during business hours, don't do any deprovisioning.<br>      #     budgets = {<br>      #       schedule = "0 9 * * mon-fri"<br>      #       duration = 8h<br>      #       nodes    = "0"<br>      #     }<br>      budgets = optional(list(object({<br>        # The schedule specifies when a budget begins being active, using extended cronjob syntax.<br>        # See https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/#schedule-syntax for syntax details.<br>        # Timezones are not supported. This field is required if Duration is set.<br>        schedule = optional(string)<br>        # Duration determines how long a Budget is active after each Scheduled start.<br>        # If omitted, the budget is always active. This is required if Schedule is set.<br>        # Must be a whole number of minutes and hours, as cron does not work in seconds,<br>        # but since Go's `duration.String()` always adds a "0s" at the end, that is allowed.<br>        duration = optional(string)<br>        # The percentage or number of nodes that Karpenter can scale down during the budget.<br>        nodes = string<br>      })), [])<br>    }), {})<br>    # Karpenter provisioner total CPU limit for all pods running on the EC2 instances launched by Karpenter<br>    total_cpu_limit = string<br>    # Karpenter provisioner total memory limit for all pods running on the EC2 instances launched by Karpenter<br>    total_memory_limit = string<br>    # Set a weight for this node pool.<br>    # See https://karpenter.sh/docs/concepts/scheduling/#weighted-nodepools<br>    weight      = optional(number, 50)<br>    labels      = optional(map(string))<br>    annotations = optional(map(string))<br>    # Karpenter provisioner taints configuration. See https://aws.github.io/aws-eks-best-practices/karpenter/#create-provisioners-that-are-mutually-exclusive for more details<br>    taints = optional(list(object({<br>      key    = string<br>      effect = string<br>      value  = string<br>    })))<br>    startup_taints = optional(list(object({<br>      key    = string<br>      effect = string<br>      value  = string<br>    })))<br>    # Karpenter node metadata options. See https://karpenter.sh/docs/concepts/nodeclasses/#specmetadataoptions for more details<br>    metadata_options = optional(object({<br>      httpEndpoint            = optional(string, "enabled")<br>      httpProtocolIPv6        = optional(string, "disabled")<br>      httpPutResponseHopLimit = optional(number, 2)<br>      # httpTokens can be either "required" or "optional"<br>      httpTokens = optional(string, "required")<br>    }), {})<br>    # The AMI used by Karpenter provisioner when provisioning nodes. Based on the value set for amiFamily, Karpenter will automatically query for the appropriate EKS optimized AMI via AWS Systems Manager (SSM)<br>    ami_family = string<br>    # Karpenter nodes block device mappings. Controls the Elastic Block Storage volumes that Karpenter attaches to provisioned nodes.<br>    # Karpenter uses default block device mappings for the AMI Family specified.<br>    # For example, the Bottlerocket AMI Family defaults with two block device mappings,<br>    # and normally you only want to scale `/dev/xvdb` where Containers and there storage are stored.<br>    # Most other AMIs only have one device mapping at `/dev/xvda`.<br>    # See https://karpenter.sh/docs/concepts/nodeclasses/#specblockdevicemappings for more details<br>    block_device_mappings = list(object({<br>      deviceName = string<br>      ebs = optional(object({<br>        volumeSize          = string<br>        volumeType          = string<br>        deleteOnTermination = optional(bool, true)<br>        encrypted           = optional(bool, true)<br>        iops                = optional(number)<br>        kmsKeyID            = optional(string, "alias/aws/ebs")<br>        snapshotID          = optional(string)<br>        throughput          = optional(number)<br>      }))<br>    }))<br>    # Set acceptable (In) and unacceptable (Out) Kubernetes and Karpenter values for node provisioning based on Well-Known Labels and cloud-specific settings. These can include instance types, zones, computer architecture, and capacity type (such as AWS spot or on-demand). See https://karpenter.sh/v0.18.0/provisioner/#specrequirements for more details<br>    requirements = list(object({<br>      key      = string<br>      operator = string<br>      # Operators like "Exists" and "DoesNotExist" do not require a value<br>      values = optional(list(string))<br>    }))<br>  }))</pre> | n/a | yes |
+| <a name="input_node_pools"></a> [node\_pools](#input\_node\_pools) | Configuration for node pools. See code for details. | <pre>map(object({<br>    # The name of the Karpenter provisioner. The map key is used if this is not set.<br>    name = optional(string)<br>    # Whether to place EC2 instances launched by Karpenter into VPC private subnets. Set it to `false` to use public subnets.<br>    private_subnets_enabled = bool<br>    # The Disruption spec controls how Karpenter scales down the node group.<br>    # See the example (sadly not the specific `spec.disruption` documentation) at https://karpenter.sh/docs/concepts/nodepools/ for details<br>    disruption = optional(object({<br>      # Describes which types of Nodes Karpenter should consider for consolidation.<br>      # If using 'WhenUnderutilized', Karpenter will consider all nodes for consolidation and attempt to remove or<br>      # replace Nodes when it discovers that the Node is underutilized and could be changed to reduce cost.<br>      # If using `WhenEmpty`, Karpenter will only consider nodes for consolidation that contain no workload pods.<br>      consolidation_policy = optional(string, "WhenUnderutilized")<br><br>      # The amount of time Karpenter should wait after discovering a consolidation decision (`go` duration string, s, m, or h).<br>      # This value can currently (v0.36.0) only be set when the consolidationPolicy is 'WhenEmpty'.<br>      # You can choose to disable consolidation entirely by setting the string value 'Never' here.<br>      # Earlier versions of Karpenter called this field `ttl_seconds_after_empty`.<br>      consolidate_after = optional(string)<br><br>      # The amount of time a Node can live on the cluster before being removed (`go` duration string, s, m, or h).<br>      # You can choose to disable expiration entirely by setting the string value 'Never' here.<br>      # This module sets a default of 336 hours (14 days), while the Karpenter default is 720 hours (30 days).<br>      # Note that Karpenter calls this field "expiresAfter", and earlier versions called it `ttl_seconds_until_expired`,<br>      # but we call it "max_instance_lifetime" to match the corresponding field in EC2 Auto Scaling Groups.<br>      max_instance_lifetime = optional(string, "336h")<br><br>      # Budgets control the the maximum number of NodeClaims owned by this NodePool that can be terminating at once.<br>      # See https://karpenter.sh/docs/concepts/disruption/#disruption-budgets for details.<br>      # A percentage is the percentage of the total number of active, ready nodes not being deleted, rounded up.<br>      # If there are multiple active budgets, Karpenter uses the most restrictive value.<br>      # If left undefined, this will default to one budget with a value of nodes: 10%.<br>      # Note that budgets do not prevent or limit involuntary terminations.<br>      # Example:<br>      #   On Weekdays during business hours, don't do any deprovisioning.<br>      #     budgets = {<br>      #       schedule = "0 9 * * mon-fri"<br>      #       duration = 8h<br>      #       nodes    = "0"<br>      #     }<br>      budgets = optional(list(object({<br>        # The schedule specifies when a budget begins being active, using extended cronjob syntax.<br>        # See https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/#schedule-syntax for syntax details.<br>        # Timezones are not supported. This field is required if Duration is set.<br>        schedule = optional(string)<br>        # Duration determines how long a Budget is active after each Scheduled start.<br>        # If omitted, the budget is always active. This is required if Schedule is set.<br>        # Must be a whole number of minutes and hours, as cron does not work in seconds,<br>        # but since Go's `duration.String()` always adds a "0s" at the end, that is allowed.<br>        duration = optional(string)<br>        # The percentage or number of nodes that Karpenter can scale down during the budget.<br>        nodes = string<br>      })), [])<br>    }), {})<br>    # Karpenter provisioner total CPU limit for all pods running on the EC2 instances launched by Karpenter<br>    total_cpu_limit = string<br>    # Karpenter provisioner total memory limit for all pods running on the EC2 instances launched by Karpenter<br>    total_memory_limit = string<br>    # Set a weight for this node pool.<br>    # See https://karpenter.sh/docs/concepts/scheduling/#weighted-nodepools<br>    weight      = optional(number, 50)<br>    labels      = optional(map(string))<br>    annotations = optional(map(string))<br>    # Karpenter provisioner taints configuration. See https://aws.github.io/aws-eks-best-practices/karpenter/#create-provisioners-that-are-mutually-exclusive for more details<br>    taints = optional(list(object({<br>      key    = string<br>      effect = string<br>      value  = optional(string)<br>    })))<br>    startup_taints = optional(list(object({<br>      key    = string<br>      effect = string<br>      value  = optional(string)<br>    })))<br>    # Karpenter node metadata options. See https://karpenter.sh/docs/concepts/nodeclasses/#specmetadataoptions for more details<br>    metadata_options = optional(object({<br>      httpEndpoint            = optional(string, "enabled")<br>      httpProtocolIPv6        = optional(string, "disabled")<br>      httpPutResponseHopLimit = optional(number, 2)<br>      # httpTokens can be either "required" or "optional"<br>      httpTokens = optional(string, "required")<br>    }), {})<br>    # The AMI used by Karpenter provisioner when provisioning nodes. Based on the value set for amiFamily, Karpenter will automatically query for the appropriate EKS optimized AMI via AWS Systems Manager (SSM)<br>    ami_family = string<br>    # Karpenter nodes block device mappings. Controls the Elastic Block Storage volumes that Karpenter attaches to provisioned nodes.<br>    # Karpenter uses default block device mappings for the AMI Family specified.<br>    # For example, the Bottlerocket AMI Family defaults with two block device mappings,<br>    # and normally you only want to scale `/dev/xvdb` where Containers and there storage are stored.<br>    # Most other AMIs only have one device mapping at `/dev/xvda`.<br>    # See https://karpenter.sh/docs/concepts/nodeclasses/#specblockdevicemappings for more details<br>    block_device_mappings = list(object({<br>      deviceName = string<br>      ebs = optional(object({<br>        volumeSize          = string<br>        volumeType          = string<br>        deleteOnTermination = optional(bool, true)<br>        encrypted           = optional(bool, true)<br>        iops                = optional(number)<br>        kmsKeyID            = optional(string, "alias/aws/ebs")<br>        snapshotID          = optional(string)<br>        throughput          = optional(number)<br>      }))<br>    }))<br>    # Set acceptable (In) and unacceptable (Out) Kubernetes and Karpenter values for node provisioning based on Well-Known Labels and cloud-specific settings. These can include instance types, zones, computer architecture, and capacity type (such as AWS spot or on-demand). See https://karpenter.sh/v0.18.0/provisioner/#specrequirements for more details<br>    requirements = list(object({<br>      key      = string<br>      operator = string<br>      # Operators like "Exists" and "DoesNotExist" do not require a value<br>      values = optional(list(string))<br>    }))<br>    # Any values for spec.template.spec.kubelet allowed by Karpenter.<br>    # Not fully specified, because they are subject to change.<br>    # See:<br>    #   https://karpenter.sh/docs/concepts/nodepools/#spectemplatespeckubelet<br>    #   https://kubernetes.io/docs/reference/config-api/kubelet-config.v1beta1/<br>    kubelet = optional(any, {})<br>  }))</pre> | n/a | yes |
 | <a name="input_regex_replace_chars"></a> [regex\_replace\_chars](#input\_regex\_replace\_chars) | Terraform regular expression (regex) string.<br>Characters matching the regex will be removed from the ID elements.<br>If not set, `"/[^a-zA-Z0-9-]/"` is used to remove all characters other than hyphens, letters and digits. | `string` | `null` | no |
 | <a name="input_region"></a> [region](#input\_region) | AWS Region | `string` | n/a | yes |
 | <a name="input_stage"></a> [stage](#input\_stage) | ID element. Usually used to indicate role, e.g. 'prod', 'staging', 'source', 'build', 'test', 'deploy', 'release' | `string` | `null` | no |

src/main.tf

@@ -8,6 +8,11 @@ locals {
   public_subnet_ids  = module.vpc.outputs.public_subnet_ids
 
   node_pools = { for k, v in var.node_pools : k => v if local.enabled }
+  kubelets_specs_filtered = { for k, v in local.node_pools : k => {
+    for kk, vv in v.kubelet : kk => vv if vv != null
+    }
+  }
+  kubelet_specs = { for k, v in local.kubelets_specs_filtered : k => v if length(v) > 0 }
 }
 
 # https://karpenter.sh/docs/concepts/nodepools/
@@ -40,8 +45,8 @@ resource "kubernetes_manifest" "node_pool" {
       )
       template = {
         metadata = {
-          labels      = each.value.labels
-          annotations = each.value.annotations
+          labels      = coalesce(each.value.labels, {})
+          annotations = coalesce(each.value.annotations, {})
         }
         spec = merge({
           nodeClassRef = {
@@ -64,6 +69,9 @@ resource "kubernetes_manifest" "node_pool" {
           },
           try(length(each.value.startup_taints), 0) == 0 ? {} : {
             startupTaints = each.value.startup_taints
+          },
+          try(local.kubelet_specs[each.key], null) == null ? {} : {
+            kubelet = local.kubelet_specs[each.key]
           }
         )
       }