Flatcar nodes

CHANGELOG.md

@@ -10,6 +10,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ### Changed
 
 - Use CAPBK to provision bastion node with Flatcar AMI.
+- Use CAPBK to provision control plane nodes with Flatcar AMI.
+- Use CAPBK to provision worker nodes with Flatcar AMI.
+- Migrating from Ubuntu AMi to Flatcar AMI is a **breaking change** that requires manual steps.
+- Apply default OS setting for flatcar and os hardening.
 - Update CAPA CRs API version from `v1beta1` to `v1beta2`.
 - Values schema: disallow additional properties on the `.nodePools` object. This is a **breaking change** where node pool names are in use that do not match the pattern `^[a-z0-9]{5,10}$`.
 

helm/cluster-aws/README.md

@@ -19,7 +19,7 @@ Properties within the `.providerSpecific` top-level object
 | :----------- | :-------------- | :--------------- |
 | `providerSpecific.ami` | **Amazon machine image (AMI)** - If specified, this image will be used to provision EC2 instances.|**Type:** `string`<br/>|
 | `providerSpecific.awsClusterRoleIdentityName` | **Cluster role identity name** - Name of an AWSClusterRoleIdentity object. This in turn refers to the IAM role used to create all AWS cloud resources when creating the cluster. The role can be in another AWS account in order to create all resources in that account. Note: This name does not refer directly to an IAM role name/ARN.|**Type:** `string`<br/>**Value pattern:** `^[-a-zA-Z0-9_\.]{1,63}$`<br/>**Default:** `"default"`|
-| `providerSpecific.flatcarAwsAccount` | **AWS account owning Flatcar image** - AWS account ID owning the Flatcar Container Linux AMI.|**Type:** `string`<br/>**Default:** `"075585003325"`|
+| `providerSpecific.flatcarAwsAccount` | **AWS account owning Flatcar image** - AWS account ID owning the Flatcar Container Linux AMI.|**Type:** `string`<br/>**Default:** `"706635527432"`|
 | `providerSpecific.region` | **Region**|**Type:** `string`<br/>|
 
 ### Connectivity
@@ -50,8 +50,12 @@ Properties within the `.connectivity` top-level object
 | `connectivity.dns.mode` | **Mode** - Whether the Route53 hosted zone of this cluster should be public or private.|**Type:** `string`<br/>**Default:** `"public"`|
 | `connectivity.dns.resolverRulesOwnerAccount` | **Resolver rules owner** - ID of the AWS account that created the resolver rules to be associated with the workload cluster VPC.|**Type:** `string`<br/>|
 | `connectivity.network` | **Network**|**Type:** `object`<br/>|
-| `connectivity.network.podCidr` | **Pod subnet** - IPv4 address range for pods, in CIDR notation.|**Type:** `string`<br/>**Default:** `"100.64.0.0/12"`|
-| `connectivity.network.serviceCidr` | **Service subnet** - IPv4 address range for services, in CIDR notation.|**Type:** `string`<br/>**Default:** `"172.31.0.0/16"`|
+| `connectivity.network.pods` | **Pods**|**Type:** `object`<br/>|
+| `connectivity.network.pods.cidrBlocks` | **Pod subnets**|**Type:** `array`<br/>**Default:** `["100.64.0.0/12"]`|
+| `connectivity.network.pods.cidrBlocks[*]` | **Pod subnet** - IPv4 address range for pods, in CIDR notation.|**Type:** `string`<br/>**Example:** `"10.244.0.0/16"`<br/>|
+| `connectivity.network.services` | **Services**|**Type:** `object`<br/>|
+| `connectivity.network.services.cidrBlocks` | **K8s Service subnets**|**Type:** `array`<br/>**Default:** `["172.31.0.0/16"]`|
+| `connectivity.network.services.cidrBlocks[*]` | **Service subnet** - IPv4 address range for kubernetes services, in CIDR notation.|**Type:** `string`<br/>**Example:** `"172.31.0.0/16"`<br/>|
 | `connectivity.network.vpcCidr` | **VPC subnet** - IPv4 address range to assign to this cluster's VPC, in CIDR notation.|**Type:** `string`<br/>**Default:** `"10.0.0.0/16"`|
 | `connectivity.proxy` | **Proxy** - Whether/how outgoing traffic is routed through proxy servers.|**Type:** `object`<br/>|
 | `connectivity.proxy.enabled` | **Enable**|**Type:** `boolean`<br/>|
@@ -85,7 +89,7 @@ Properties within the `.controlPlane` top-level object
 | `controlPlane.apiMode` | **API mode** - Whether the Kubernetes API server load balancer should be reachable from the internet (public) or internal only (private).|**Type:** `string`<br/>**Default:** `"public"`|
 | `controlPlane.containerdVolumeSizeGB` | **Containerd volume size (GB)**|**Type:** `integer`<br/>**Default:** `100`|
 | `controlPlane.etcdVolumeSizeGB` | **Etcd volume size (GB)**|**Type:** `integer`<br/>**Default:** `100`|
-| `controlPlane.instanceType` | **EC2 instance type**|**Type:** `string`<br/>**Default:** `"m5.xlarge"`|
+| `controlPlane.instanceType` | **EC2 instance type**|**Type:** `string`<br/>**Default:** `"r6i.xlarge"`|
 | `controlPlane.kubeletVolumeSizeGB` | **Kubelet volume size (GB)**|**Type:** `integer`<br/>**Default:** `100`|
 | `controlPlane.machineHealthCheck` | **Machine health check**|**Type:** `object`<br/>|
 | `controlPlane.machineHealthCheck.enabled` | **Enable**|**Type:** `boolean`<br/>**Default:** `true`|
@@ -112,7 +116,7 @@ For Giant Swarm internal use only, not stable, or not supported by UIs.
 | :----------- | :-------------- | :--------------- |
 | `internal.hashSalt` | **Hash salt** - If specified, this token is used as a salt to the hash suffix of some resource names. Can be used to force-recreate some resources.|**Type:** `string`<br/>|
 | `internal.kubernetesVersion` | **Kubernetes version**|**Type:** `string`<br/>**Example:** `"1.24.7"`<br/>**Default:** `"1.24.10"`|
-| `internal.nodePools` | **Default node pool**|**Type:** `object`<br/>**Default:** `{"def00":{"customNodeLabels":["label=default"],"instanceType":"m5.xlarge","minSize":3}}`|
+| `internal.nodePools` | **Default node pool**|**Type:** `object`<br/>**Default:** `{"def00":{"customNodeLabels":["label=default"],"instanceType":"r6i.xlarge","minSize":3}}`|
 | `internal.nodePools.PATTERN` | **Node pool**|**Type:** `object`<br/>**Key pattern:**<br/>`PATTERN`=`^[a-z0-9]{5,10}$`<br/>|
 | `internal.nodePools.PATTERN.availabilityZones` | **Availability zones**|**Type:** `array`<br/>**Key pattern:**<br/>`PATTERN`=`^[a-z0-9]{5,10}$`<br/>|
 | `internal.nodePools.PATTERN.availabilityZones[*]` | **Availability zone**|**Type:** `string`<br/>**Key pattern:**<br/>`PATTERN`=`^[a-z0-9]{5,10}$`<br/>|
@@ -130,6 +134,10 @@ For Giant Swarm internal use only, not stable, or not supported by UIs.
 | `internal.nodePools.PATTERN.subnetTags` | **Subnet tags** - Tags to filter which AWS subnets will be used for this node pool.|**Type:** `array`<br/>**Key pattern:**<br/>`PATTERN`=`^[a-z0-9]{5,10}$`<br/>|
 | `internal.nodePools.PATTERN.subnetTags[*]` | **Subnet tag**|**Type:** `object`<br/>**Key pattern:**<br/>`PATTERN`=`^[a-z0-9]{5,10}$`<br/>|
 | `internal.nodePools.PATTERN.subnetTags[*].*` | **Tag value**|**Type:** `string`<br/>**Key pattern:**<br/>`PATTERN`=`^[a-z0-9]{5,10}$`<br/>**Value pattern:** `^[ a-zA-Z0-9\._:/=+-@]+$`<br/>|
+| `internal.sandboxContainerImage` | **Kubectl image**|**Type:** `object`<br/>|
+| `internal.sandboxContainerImage.name` | **Repository**|**Type:** `string`<br/>**Default:** `"giantswarm/pause"`|
+| `internal.sandboxContainerImage.registry` | **Registry**|**Type:** `string`<br/>**Default:** `"quay.io"`|
+| `internal.sandboxContainerImage.tag` | **Tag**|**Type:** `string`<br/>**Default:** `"3.9"`|
 
 ### Kubectl image
 Properties within the `.kubectlImage` top-level object
@@ -148,6 +156,7 @@ Properties within the `.metadata` top-level object
 | `metadata.description` | **Cluster description** - User-friendly description of the cluster's purpose.|**Type:** `string`<br/>|
 | `metadata.name` | **Cluster name** - Unique identifier, cannot be changed after creation.|**Type:** `string`<br/>|
 | `metadata.organization` | **Organization**|**Type:** `string`<br/>|
+| `metadata.servicePriority` | **Service priority** - The relative importance of this cluster.|**Type:** `string`<br/>**Default:** `"highest"`|
 
 ### Node pools
 Properties within the `.nodePools` top-level object

helm/cluster-aws/files/etc/containerd/conf.d/registry-config.toml → helm/cluster-aws/files/etc/containerd/config.toml

@@ -1,3 +1,28 @@
+version = 2
+
+# recommended defaults from https://github.com/containerd/containerd/blob/main/docs/ops.md#base-configuration
+# set containerd as a subreaper on linux when it is not running as PID 1
+subreaper = true
+# set containerd's OOM score
+oom_score = -999
+disabled_plugins = []
+[plugins."containerd.runtime.v1.linux"]
+# shim binary name/path
+shim = "containerd-shim"
+# runtime binary name/path
+runtime = "runc"
+# do not use a shim when starting containers, saves on memory but
+# live restore is not supported
+no_shim = false
+
+[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
+# setting runc.options unsets parent settings
+runtime_type = "io.containerd.runc.v2"
+[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
+SystemdCgroup = true
+[plugins."io.containerd.grpc.v1.cri"]
+sandbox_image = "{{ .Values.internal.sandboxContainerImage.registry }}/{{ .Values.internal.sandboxContainerImage.name }}:{{ .Values.internal.sandboxContainerImage.tag }}"
+
 [plugins."io.containerd.grpc.v1.cri".registry]
   [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
   {{- range $host, $config := .Values.connectivity.containerRegistries }}

helm/cluster-aws/files/etc/sysctl.d/hardening.conf

@@ -0,0 +1,24 @@
+fs.inotify.max_user_watches = 16384
+fs.inotify.max_user_instances = 8192
+kernel.kptr_restrict = 2
+kernel.sysrq = 0
+net.ipv4.conf.all.log_martians = 1
+net.ipv4.conf.all.send_redirects = 0
+net.ipv4.conf.default.accept_redirects = 0
+net.ipv4.conf.default.log_martians = 1
+net.ipv4.tcp_timestamps = 0
+net.ipv6.conf.all.accept_redirects = 0
+net.ipv6.conf.default.accept_redirects = 0
+# Increased mmapfs because some applications, like ES, need higher limit to store data properly
+vm.max_map_count = 262144
+# Reserved to avoid conflicts with kube-apiserver, which allocates within this range
+net.ipv4.ip_local_reserved_ports=30000-32767
+net.ipv4.conf.all.rp_filter = 1
+net.ipv4.conf.all.arp_ignore = 1
+net.ipv4.conf.all.arp_announce = 2
+
+# These are required for the kubelet '--protect-kernel-defaults' flag
+# See https://github.com/giantswarm/giantswarm/issues/13587
+vm.overcommit_memory=1
+kernel.panic=10
+kernel.panic_on_oops=1

helm/cluster-aws/files/etc/systemd/timesyncd.conf

@@ -0,0 +1,2 @@
+[Time]
+NTP=169.254.169.123

helm/cluster-aws/templates/_bastion.tpl

@@ -41,7 +41,7 @@ ignition:
     additionalConfig: |
       systemd:
         units:
-        {{- include "flatcarKubeadmService" $ | nindent 8 }}
+        {{- include "flatcarSystemdUnits" $ | nindent 8 }}
 preKubeadmCommands:
 {{ include "flatcarKubeadmPreCommands" $  }}
 - systemctl restart sshd

helm/cluster-aws/templates/_control_plane.tpl

@@ -71,6 +71,19 @@ spec:
       kind: AWSMachineTemplate
       name: {{ include "resource.default.name" $ }}-control-plane-{{ include "hash" (dict "data" (include "controlplane-awsmachinetemplate-spec" $) "global" .) }}
   kubeadmConfigSpec:
+    format: ignition
+    ignition:
+      containerLinuxConfig:
+        additionalConfig: |
+          systemd:
+            units:
+            {{- include "flatcarSystemdUnits" $ | nindent 14 }}
+            {{- include "diskStorageSystemdUnits" $ | nindent 14 }}
+          storage:
+            filesystems:
+            {{- include "diskStorageConfig" $ | nindent 14 }}
+            directories:
+            {{- include "nodeDirectories" $ | nindent 14 }}
     clusterConfiguration:
       # Avoid accessibility issues (e.g. on private clusters) and potential future rate limits for the default `registry.k8s.io`
       imageRepository: docker.io/giantswarm
@@ -148,10 +161,9 @@ spec:
     files:
     {{- include "oidcFiles" . | nindent 4 }}
     {{- include "sshFiles" . | nindent 4 }}
-    {{- include "diskFiles" . | nindent 4 }}
     {{- include "irsaFiles" . | nindent 4 }}
     {{- include "kubeletConfigFiles" . | nindent 4 }}
-    {{- include "awsNtpFiles" . | nindent 4 }}
+    {{- include "nodeConfigFiles" . | nindent 4 }}
     {{- if .Values.connectivity.proxy.enabled }}{{- include "proxyFiles" . | nindent 4 }}{{- end }}
     {{- include "kubernetesFiles" . | nindent 4 }}
     {{- include "registryFiles" . | nindent 4 }}
@@ -167,9 +179,9 @@ spec:
           cloud-provider: external
           feature-gates: CronJobTimeZone=true
           healthz-bind-address: 0.0.0.0
-          node-ip: '{{ `{{ ds.meta_data.local_ipv4 }}` }}'
+          node-ip: ${COREOS_EC2_IPV4_LOCAL}
           v: "2"
-        name: '{{ `{{ ds.meta_data.local_hostname }}` }}'
+        name: ${COREOS_EC2_HOSTNAME}
         {{- if .Values.controlPlane.customNodeTaints }}
         {{- if (gt (len .Values.controlPlane.customNodeTaints) 0) }}
         taints:
@@ -186,7 +198,7 @@ spec:
         kubeletExtraArgs:
           cloud-provider: external
           feature-gates: CronJobTimeZone=true
-        name: '{{ `{{ ds.meta_data.local_hostname }}` }}'
+        name: ${COREOS_EC2_HOSTNAME}
         {{- if .Values.controlPlane.customNodeTaints }}
         {{- if (gt (len .Values.controlPlane.customNodeTaints) 0) }}
         taints:
@@ -198,14 +210,12 @@ spec:
         {{- end }}
         {{- end }}
     preKubeadmCommands:
-    {{- include "prepare-varLibKubelet-Dir" . | nindent 4 }}
-    {{- include "diskPreKubeadmCommands" . | nindent 4 }}
+    {{- include "flatcarKubeadmPreCommands" . | nindent 4 }}
     {{- include "sshPreKubeadmCommands" . | nindent 4 }}
     {{- if .Values.connectivity.proxy.enabled }}{{- include "proxyCommand" $ | nindent 4 }}{{- end }}
     postKubeadmCommands:
     {{- include "irsaPostKubeadmCommands" . | nindent 4 }}
     {{- include "kubeletConfigPostKubeadmCommands" . | nindent 4 }}
-    {{- include "awsNtpPostKubeadmCommands" . | nindent 4 }}
     users:
     {{- include "sshUsers" . | nindent 4 }}
   replicas: 3