EKS

Prerequisites

• VPC with enough IP address space. See the requirements

Resources Provisioned

• EKS Cluster
• KMS Key is used to encrypt K8S Secrets
• IAM Role for Service Account is enabled
• KMS Key for EBS volumes is created and ASG is given permission to use the key
• A "default" self managed node group on Bottlerocket is provisioned.

In addition, it is expected that all worker nodes will share one IAM Role and a common security group.

• IAM Role contains the necessary policies for nodes to join the EKS cluster and optionally manage ENI for CNI purposes. IAM Role for Service Account usage is strongly recommended.
• The Security Group has default rules to allow the cluster to function. Additional groups can be added for additional node groups.

Instance refresh on ASG is handled by AWS Node Termination Handler. For the purposes of instance refresh, the following resources are created:

• SQS Queue where events published to be consumed by Node Termination Handler is published.

The Queue ARN will be subsequently used by the eks_self_managed_nodes module to provision additional node groups.

Usage

Defining Providers

Definining providers in reusable modules is deprecated and causes features like modules for_each and count to not work. In addition to the aws providers, the main module and submodules can require additional Kubernetes providers to be configured.

Main Module

The main module uses the kubernetes provider.

provider "aws" {
  # ...
}

module "eks" {
  # ...
  create_aws_auth_configmap = false
  manage_aws_auth_configmap = false # set this to true after cluster creation is completed and reapply to set IRSA
}

data "aws_eks_cluster_auth" "this" {
  name = module.eks.cluster_name
}

provider "kubernetes" {
  host                   = module.eks.cluster_endpoint
  cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)
  token                  = data.aws_eks_cluster_auth.this.token
}

Sub-modules

Other modules might make use of the kubernetes or helm providers

provider "aws" {
  # ...
}

data "aws_eks_cluster" "this" {
  name = var.cluster_name
}

data "aws_eks_cluster_auth" "this" {
  name = var.cluster_name
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.this.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.this.certificate_authority[0].data)
  token                  = data.aws_eks_cluster_auth.this.token
}

provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.this.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.this.certificate_authority[0].data)
    token                  = data.aws_eks_cluster_auth.this.token
  }

  experiments {
    manifest = true
  }
}

Karpenter

We need to allow Karpenter controller to start EC2 instances hence, we need to add role mapping for it:

role_mapping = local.autoscaling_mode == "karpenter" ? concat([
    for role in local.eks_master_roles :
    {
      rolearn  = role.arn
      groups   = ["system:masters"]
      username = role.user
    }],
    [
      {
        rolearn = module.eks.worker_iam_role_arn
        groups = [
          "system:bootstrappers",
          "system:nodes",
        ]
        username = "system:node:{{EC2PrivateDNSName}}"
      }
    ]
    ) : [
    for role in local.eks_master_roles :
    {
      rolearn  = role.arn
      groups   = ["system:masters"]
      username = role.user
    }
  ]

Then we create Karpenter module with configuration as follows:

locals {
  # Karpenter Provisioners Config
  karpenter_provisioners = [
    {
      name                           = "provisioner_name"
      provider_ref_nodetemplate_name = "default"
      karpenter_provisioner_node_labels = {
        "label_key" = "label_value"
      }

      karpenter_provisioner_node_taints = [
        {
          key       = "taintkey",
          value     = "taintvalue",
          effect    = "NoSchedule"
          timeAdded = timestamp() # required if not terraform plan complains
        }
      ]
    },
  ]
  # Karpenter Nodetemplate Config
  karpenter_nodetemplates = [
    {
      name = "default"
      karpenter_subnet_selector_map = {
        "Name" = "subnet-name-here-*"
      }
      karpenter_security_group_selector_map = {
        "aws-ids" = module.eks.worker_security_group_id
      }
      karpenter_nodetemplate_tag_map = {
        "karpenter.sh/discovery" = module.eks.cluster_name
      }
    },
  ]
}

module "karpenter" {
  # source  = "SPHTech-Platform/eks/aws//modules/essentials"
  # version = "~> 0.8.0"

  source = "git::https://github.com/SPHTech-Platform/terraform-aws-eks.git//modules/karpenter?ref=karpenter"

  cluster_name        = local.cluster_name
  cluster_endpoint    = data.aws_eks_cluster.this.endpoint
  oidc_provider_arn   = module.eks.oidc_provider_arn
  worker_iam_role_arn = module.eks.worker_iam_role_arn

  autoscaling_mode        = local.autoscaling_mode
  karpenter_provisioners  = local.karpenter_provisioners
  karpenter_nodetemplates = local.karpenter_nodetemplates

}

Requirements

Name	Version
terraform	>= 1.4
aws	>= 4.47
helm	~> 2.6
kubectl	1.14.0
kubernetes	>= 2.10

Providers

Name	Version
aws	>= 4.47
kubernetes	>= 2.10

Modules

Name	Source	Version
ebs_csi_irsa_role	terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks	~> 5.11.2
eks	terraform-aws-modules/eks/aws	~> 19.17.0
fargate_profiles	./modules/fargate_profile	n/a
karpenter	./modules/karpenter	n/a
kms_ebs	SPHTech-Platform/kms/aws	~> 0.1.0
kms_secret	SPHTech-Platform/kms/aws	~> 0.1.0
node_groups	./modules/eks_managed_nodes	n/a
vpc_cni_irsa_role	terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks	~> 5.11.2

Resources

Name	Type
aws_iam_role.cluster	resource
aws_iam_role.workers	resource
aws_iam_role_policy.ebs_csi_kms	resource
aws_iam_role_policy_attachment.cluster	resource
aws_iam_role_policy_attachment.workers	resource
aws_iam_service_linked_role.autoscaling	resource
kubernetes_config_map_v1.amazon_vpc_cni	resource
kubernetes_manifest.fargate_node_security_group_policy	resource
kubernetes_manifest.fargate_node_security_group_policy_for_karpenter	resource
aws_ami.eks_default_bottlerocket	data source
aws_arn.cluster	data source
aws_caller_identity.current	data source
aws_iam_policy_document.ec2_assume_role_policy	data source
aws_iam_policy_document.eks_assume_role_policy	data source
aws_iam_policy_document.kms_csi_ebs	data source
aws_iam_policy_document.kms_ebs	data source
aws_partition.current	data source
aws_subnet.subnets	data source

Inputs

Name	Description	Type	Default	Required
additional_karpenter_security_group_ids	Additional security group IDs to add to the Karpenter node groups	list(string)	[]	no
autoscaling_mode	Autoscaling mode: cluster_autoscaler or karpenter	string	"karpenter"	no
aws_auth_fargate_profile_pod_execution_role_arns	List of Fargate profile pod execution role ARNs to add to the aws-auth configmap	list(string)	[]	no
cluster_additional_security_group_ids	List of additional, externally created security group IDs to attach to the cluster control plane	list(string)	[]	no
cluster_addons	Map of cluster addon configurations to enable for the cluster. Addon name can be the map keys or set with name	any	{}	no
cluster_addons_timeouts	Create, update, and delete timeout configurations for the cluster addons	map(string)	{}	no
cluster_enabled_log_types	A list of the desired control plane logs to enable. For more information, see Amazon EKS Control Plane Logging documentation (https://docs.aws.amazon.com/eks/latest/userguide/control-plane-logs.html)	list(string)	[ "audit", "api", "authenticator" ]	no
cluster_endpoint_private_access	Indicates whether or not the Amazon EKS private API server endpoint is enabled	bool	true	no
cluster_endpoint_public_access	Indicates whether or not the Amazon EKS public API server endpoint is enabled	bool	true	no
cluster_endpoint_public_access_cidrs	List of CIDR blocks which can access the Amazon EKS public API server endpoint	list(string)	[ "0.0.0.0/0" ]	no
cluster_iam_boundary	IAM boundary for the cluster IAM role, if any	string	null	no
cluster_iam_role	Cluster IAM Role name. If undefined, is the same as the cluster name	string	""	no
cluster_ip_family	The IP family used to assign Kubernetes pod and service addresses. Valid values are ipv4 (default) and ipv6. You can only specify an IP family when you create a cluster, changing this value will force a new cluster to be created	string	"ipv4"	no
cluster_name	EKS Cluster Name	string	n/a	yes
cluster_security_group_additional_rules	List of additional security group rules to add to the cluster security group created. Set source_node_security_group = true inside rules to set the node_security_group as source	any	{}	no
cluster_security_group_name	Cluster security group name	string	null	no
cluster_service_ipv4_cidr	The CIDR block to assign Kubernetes service IP addresses from. If you don't specify a block, Kubernetes assigns addresses from either the 10.100.0.0/16 or 172.20.0.0/16 CIDR blocks	string	null	no
cluster_service_ipv6_cidr	The CIDR block to assign Kubernetes pod and service IP addresses from if ipv6 was specified when the cluster was created. Kubernetes assigns service addresses from the unique local address range (fc00::/7) because you can't specify a custom IPv6 CIDR block when you create the cluster	string	null	no
cluster_version	EKS Cluster Version	string	"1.27"	no
create_aws_auth_configmap	Determines whether to create the aws-auth configmap. NOTE - this is only intended for scenarios where the configmap does not exist (i.e. - when using only self-managed node groups). Most users should use manage_aws_auth_configmap	bool	false	no
create_aws_observability_ns	Whether to create AWS Observability Namespace.	bool	true	no
create_aws_observability_ns_for_karpenter	Create aws-observability namespace flag	bool	false	no
create_cluster_security_group	Determines if a security group is created for the cluster. Note: the EKS service creates a primary security group for the cluster by default	bool	true	no
create_cni_ipv6_iam_policy	Whether to create CNI IPv6 IAM policy.	bool	false	no
create_fargate_log_group_for_karpenter	value for create_fargate_log_group	bool	false	no
create_fargate_logger_configmap	Whether to create AWS Fargate logger configmap.	bool	true	no
create_fargate_logger_configmap_for_karpenter	create_fargate_logger_configmap flag	bool	false	no
create_fargate_logging_policy_for_karpenter	value for create_fargate_logging_policy	bool	false	no
create_fargate_profile_for_karpenter	Create fargate profile flag	bool	false	no
create_node_security_group	Determines whether to create a security group for the node groups or use the existing node_security_group_id	bool	true	no
default_group_ami_id	The AMI from which to launch the defualt group instance. If not supplied, EKS will use its own default image	string	""	no
default_group_instance_types	Instance type for the default node group	list(string)	[ "m5a.xlarge", "m5.xlarge", "m5n.xlarge", "m5zn.xlarge" ]	no
default_group_launch_template_name	Name of the default node group launch template	string	"default"	no
default_group_max_size	Configuration for max default node group size	number	5	no
default_group_min_size	Configuration for min default node group size	number	1	no
default_group_name	Name of the default node group	string	"default"	no
default_group_node_labels	Additional node label for default group	map(string)	{}	no
default_group_subnet_ids	Subnet IDs to create the default group ASGs in	list(string)	[]	no
default_group_volume_size	Size of the persistentence volume for the default group	number	50	no
eks_managed_node_group_defaults	Map of EKS managed node group default configurations	any	{ "create_iam_role": false, "disk_size": 50, "ebs_optimized": true, "enable_monitoring": true, "metadata_options": { "http_endpoint": "enabled", "http_protocol_ipv6": "disabled", "http_put_response_hop_limit": 1, "http_tokens": "required", "instance_metadata_tags": "disabled" }, "protect_from_scale_in": false, "update_launch_template_default_version": true }	no
eks_managed_node_groups	Map of EKS managed node group definitions to create	any	{}	no
enable_cluster_windows_support	Determines whether to create the amazon-vpc-cni configmap and windows worker roles into aws-auth.	bool	false	no
fargate_cluster	Whether to create eks cluster with fargate mode. If true, default node group also will be fargate, otherwise managed	bool	false	no
fargate_profile_defaults	Map of Fargate Profile default configurations	any	{}	no
fargate_profiles	Map of maps of fargate_profiles to create	any	{}	no
force_imdsv2	Force IMDSv2 metadata server.	bool	true	no
force_irsa	Force usage of IAM Roles for Service Account	bool	true	no
iam_role_additional_policies	Additional policies to be added to the IAM role	set(string)	[]	no
karpenter_chart_version	Chart version for Karpenter	string	"v0.34.6"	no
karpenter_default_subnet_selector_tags	Subnet selector tags for Karpenter default node class	map(string)	{ "kubernetes.io/role/internal-elb": "1" }	no
karpenter_nodeclasses	List of nodetemplate maps	list(object({ nodeclass_name = string karpenter_subnet_selector_maps = list(map(any)) karpenter_security_group_selector_maps = list(map(any)) karpenter_ami_selector_maps = list(map(any)) karpenter_node_role = string karpenter_node_tags_map = map(string) karpenter_ami_family = string karpenter_node_user_data = string karpenter_node_metadata_options = map(any) karpenter_block_device_mapping = list(object({ deviceName = string ebs = object({ encrypted = bool volumeSize = string volumeType = string kmsKeyID = optional(string) deleteOnTermination = bool }) })) }))	[]	no
karpenter_nodepools	List of Provisioner maps	list(object({ nodepool_name = string nodeclass_name = string karpenter_nodepool_node_labels = map(string) karpenter_nodepool_annotations = map(string) karpenter_nodepool_node_taints = list(map(string)) karpenter_nodepool_startup_taints = list(map(string)) karpenter_requirements = list(object({ key = string operator = string values = list(string) }) ) karpenter_nodepool_disruption = object({ consolidation_policy = string consolidate_after = optional(string) expire_after = string }) karpenter_nodepool_disruption_budgets = list(map(any)) karpenter_nodepool_weight = number }))	[ { "karpenter_nodepool_annotations": {}, "karpenter_nodepool_disruption": { "consolidation_policy": "WhenUnderutilized", "expire_after": "168h" }, "karpenter_nodepool_disruption_budgets": [ { "nodes": "10%" } ], "karpenter_nodepool_node_labels": {}, "karpenter_nodepool_node_taints": [], "karpenter_nodepool_startup_taints": [], "karpenter_nodepool_weight": 10, "karpenter_requirements": [ { "key": "karpenter.k8s.aws/instance-category", "operator": "In", "values": [ "t", "m" ] }, { "key": "karpenter.k8s.aws/instance-cpu", "operator": "In", "values": [ "2", "4" ] }, { "key": "karpenter.k8s.aws/instance-memory", "operator": "Gt", "values": [ "2048" ] }, { "key": "karpenter.k8s.aws/instance-generation", "operator": "Gt", "values": [ "2" ] }, { "key": "karpenter.sh/capacity-type", "operator": "In", "values": [ "on-demand" ] }, { "key": "kubernetes.io/arch", "operator": "In", "values": [ "amd64" ] }, { "key": "kubernetes.io/os", "operator": "In", "values": [ "linux" ] } ], "nodeclass_name": "default", "nodepool_name": "default" } ]	no
manage_aws_auth_configmap	Determines whether to manage the contents of the aws-auth configmap	bool	true	no
node_security_group_additional_rules	List of additional security group rules to add to the node security group created. Set source_cluster_security_group = true inside rules to set the cluster_security_group as source	any	{}	no
node_security_group_enable_recommended_rules	Determines whether to enable recommended security group rules for the node security group created. This includes node-to-node TCP ingress on ephemeral ports and allows all egress traffic	bool	true	no
only_critical_addons_enabled	Enabling this option will taint default node group with CriticalAddonsOnly=true:NoSchedule taint. Changing this forces a new resource to be created.	bool	false	no
role_mapping	List of IAM roles to give access to the EKS cluster	list(object({ rolearn = string username = string groups = list(string) }))	[]	no
skip_asg_role	Skip creating ASG Service Linked Role if it's already created	bool	false	no
subnet_ids	A list of subnet IDs where the EKS cluster (ENIs) will be provisioned along with the nodes/node groups. Node groups can be deployed within a different set of subnet IDs from within the node group configuration	list(string)	n/a	yes
tags	A map of tags to add to all resources	map(string)	{}	no
user_mapping	List of IAM Users to give access to the EKS Cluster	list(object({ userarn = string username = string groups = list(string) }))	[]	no
vpc_id	VPC ID to deploy the cluster into	string	n/a	yes
worker_security_group_name	Worker security group name	string	null	no
workers_iam_boundary	IAM boundary for the workers IAM role, if any	string	null	no
workers_iam_role	Workers IAM Role name. If undefined, is the same as the cluster name suffixed with 'workers'	string	""	no

Outputs

Name	Description
cluster_arn	The ARN of the EKS cluster
cluster_certificate_authority_data	Base64 Encoded Cluster CA Data
cluster_endpoint	Endpoint of the EKS Cluster
cluster_iam_role_arn	IAM Role ARN used by cluster
cluster_iam_role_name	IAM Role Name used by Cluster
cluster_name	EKS Cluster name created
cluster_oidc_issuer_url	The URL on the EKS cluster for the OpenID Connect identity provider
cluster_platform_version	Platform version of the EKS Cluster
cluster_primary_security_group_id	Primary Security Group ID of the EKS cluster
cluster_security_group_id	Security Group ID of the master nodes
cluster_version	Version of the EKS Cluster
ebs_kms_key_arn	KMS Key ARN used for EBS encryption
ebs_kms_key_id	KMS Key ID used for EBS encryption
fargate_namespaces_for_security_group	value for fargate_namespaces_for_security_group
oidc_provider_arn	OIDC Provider ARN for IRSA
worker_iam_role_arn	IAM Role ARN used by worker nodes
worker_iam_role_name	IAM Role Name used by worker nodes
worker_security_group_id	Security Group ID of the worker nodes