Flatcar nodes #322

calvix · 2023-06-14T07:32:53Z

CP and worker nodes machines on flatcar
move several scripts into the proper configuration
adjust OS setting and hardening to match what we have in k8scloudconfig

It is partially breaking change as KubeadmControlPlane webhook does not allow change of the format from cloud-init to ignition, it can be worked around by shortly disabling the webhook and letting chart operator do the update of CRs, also the worker nodes became unavailable for a while as it takes a while until the ignition is rendered and in that time the operator tries to launch Flatcar OS and feeding it the cloud-init instead of ignition. But that will resolve after 5-10 minutes and workers will join the cluster

… into enable-flatcar-os

tityosbot · 2023-06-14T07:32:55Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

calvix · 2023-06-16T12:56:09Z

/run cluster-test-suites

calvix · 2023-06-16T12:58:42Z

helm/cluster-aws/files/etc/containerd/config.toml

+SystemdCgroup = true
+[plugins."io.containerd.grpc.v1.cri"]
+sandbox_image = "{{ .Values.internal.sandboxContainerImage.registry }}/{{ .Values.internal.sandboxContainerImage.name }}:{{ .Values.internal.sandboxContainerImage.tag }}"
+


to avoid a weird bug where containerd kills random pods - giantswarm/roadmap#1737
also configure sandbox pause container to use our registry instead of k8s gcr

calvix · 2023-06-16T12:59:03Z

helm/cluster-aws/files/etc/systemd/timesyncd.conf

@@ -0,0 +1,2 @@
+[Time]
+NTP=169.254.169.123


configure timesyncd instead of the ntp script

calvix · 2023-06-16T13:00:13Z

helm/cluster-aws/templates/_helpers.tpl

+    device: /dev/xvde
+    wipeFilesystem: true
+    label: kubelet
+    format: xfs


configure disks with ignition instead of bash script

calvix · 2023-06-16T13:07:56Z

/run cluster-test-suites

helm/cluster-aws/files/etc/containerd/config.toml

helm/cluster-aws/files/etc/sysctl.d/hardening.conf

helm/cluster-aws/templates/_helpers.tpl

calvix · 2023-06-20T09:17:25Z

/run cluster-test-suites

calvix · 2023-06-21T07:17:30Z

/run cluster-test-suites

github-actions · 2023-06-21T08:39:19Z

(helm/cluster-aws/ci/test-wc-minimal-values.yaml) rendered manifest diff

(file level)
  - two documents removed:
    ---
    # Source: cluster-aws/templates/list.yaml
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
    kind: AWSMachineTemplate
    metadata:
      name: test-wc-minimal-control-plane-04636bc2
      namespace: org-giantswarm
      labels:
        cluster.x-k8s.io/role: control-plane
        app: cluster-aws
        app.kubernetes.io/managed-by: Helm
        cluster.x-k8s.io/cluster-name: test-wc-minimal
        giantswarm.io/cluster: test-wc-minimal
        giantswarm.io/organization: test
        cluster.x-k8s.io/watch-filter: capi
        helm.sh/chart: cluster-aws-0.33.0
        application.giantswarm.io/team: hydra
        app.kubernetes.io/version: 0.33.0
    spec:
      template:
        metadata:
          labels:
            cluster.x-k8s.io/role: control-plane
            app: cluster-aws
            app.kubernetes.io/managed-by: Helm
            cluster.x-k8s.io/cluster-name: test-wc-minimal
            giantswarm.io/cluster: test-wc-minimal
            giantswarm.io/organization: test
            cluster.x-k8s.io/watch-filter: capi
            helm.sh/chart: cluster-aws-0.33.0
            application.giantswarm.io/team: hydra
        spec:
          ami: {}
          imageLookupBaseOS: ubuntu-20.04
          imageLookupFormat: capa-ami-{{.BaseOS}}-v{{.K8sVersion}}-gs
          imageLookupOrg: 706635527432
          cloudInit: {}
          instanceType: r6i.xlarge
          nonRootVolumes:
          - type: gp3
            deviceName: /dev/xvdc
            encrypted: true
            size: 100
          - type: gp3
            deviceName: /dev/xvdd
            encrypted: true
            size: 100
          - type: gp3
            deviceName: /dev/xvde
            encrypted: true
            size: 100
          rootVolume:
            type: gp3
            size: 120
          iamInstanceProfile: control-plane-test-wc-minimal
          sshKeyName: 
          subnet:
            filters:
            - name: "tag:kubernetes.io/cluster/test-wc-minimal"
              values:
              - shared
              - owned
            - name: "tag:sigs.k8s.io/cluster-api-provider-aws/role"
              values:
              - private
    # Source: cluster-aws/templates/list.yaml
    apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
    kind: KubeadmConfigTemplate
    metadata:
      labels:
        cluster.x-k8s.io/role: bastion
        app: cluster-aws
        app.kubernetes.io/managed-by: Helm
        cluster.x-k8s.io/cluster-name: test-wc-minimal
        giantswarm.io/cluster: test-wc-minimal
        giantswarm.io/organization: test
        cluster.x-k8s.io/watch-filter: capi
        helm.sh/chart: cluster-aws-0.33.0
        application.giantswarm.io/team: hydra
      name: test-wc-minimal-bastion-b2e9a9ff
      namespace: org-giantswarm
    spec:
      template:
        spec:
          format: ignition
          ignition:
            containerLinuxConfig:
              additionalConfig: |
                systemd:
                  units:
                  - name: kubeadm.service
                    dropins:
                    - name: 10-flatcar.conf
                      contents: |
                        [Unit]
                        # kubeadm must run after coreos-metadata populated /run/metadata directory.
                        Requires=coreos-metadata.service
                        After=coreos-metadata.service
                        [Service]
                        # Ensure kubeadm service has access to kubeadm binary in /opt/bin on Flatcar.
                        Environment=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/bin
                        # To make metadata environment variables available for pre-kubeadm commands.
                        EnvironmentFile=/run/metadata/*
                
          preKubeadmCommands:
          - "envsubst < /etc/kubeadm.yml > /etc/kubeadm.yml.tmp"
          - "mv /etc/kubeadm.yml.tmp /etc/kubeadm.yml"
          - "systemctl restart sshd"
          - "sleep infinity"
          files:
          - path: /etc/ssh/trusted-user-ca-keys.pem
            permissions: 0600
            encoding: base64
            content: c3NoLWVkMjU1MTkgQUFBQUMzTnphQzFsWkRJMU5URTVBQUFBSU00Y3ZaMDFmTG1POWNKYldVajdzZkYrTmhFQ2d5K0NsMGJhelNyWlg3c1UgdmF1bHQtY2FAdmF1bHQub3BlcmF0aW9ucy5naWFudHN3YXJtLmlvCg==
          - path: /etc/ssh/sshd_config
            permissions: 0600
            encoding: base64
            content: IyBVc2UgbW9zdCBkZWZhdWx0cyBmb3Igc3NoZCBjb25maWd1cmF0aW9uLgpTdWJzeXN0ZW0gc2Z0cCBpbnRlcm5hbC1zZnRwCkNsaWVudEFsaXZlSW50ZXJ2YWwgMTgwClVzZUROUyBubwpVc2VQQU0geWVzClByaW50TGFzdExvZyBubyAjIGhhbmRsZWQgYnkgUEFNClByaW50TW90ZCBubyAjIGhhbmRsZWQgYnkgUEFNCiMgTm9uIGRlZmF1bHRzICgjMTAwKQpDbGllbnRBbGl2ZUNvdW50TWF4IDIKUGFzc3dvcmRBdXRoZW50aWNhdGlvbiBubwpUcnVzdGVkVXNlckNBS2V5cyAvZXRjL3NzaC90cnVzdGVkLXVzZXItY2Eta2V5cy5wZW0KTWF4QXV0aFRyaWVzIDUKTG9naW5HcmFjZVRpbWUgNjAKQWxsb3dUY3BGb3J3YXJkaW5nIHllcwpBbGxvd0FnZW50Rm9yd2FyZGluZyB5ZXMKQ0FTaWduYXR1cmVBbGdvcml0aG1zIGVjZHNhLXNoYTItbmlzdHAyNTYsZWNkc2Etc2hhMi1uaXN0cDM4NCxlY2RzYS1zaGEyLW5pc3RwNTIxLHNzaC1lZDI1NTE5LHJzYS1zaGEyLTUxMixyc2Etc2hhMi0yNTYsc3NoLXJzYQpQb3J0IDMwMDAwClBvcnQgMjIK
          users:
          - name: giantswarm
            groups: sudo
            sudo: "ALL=(ALL) NOPASSWD:ALL"
    
  
    ---
    # Source: cluster-aws/templates/list.yaml
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
    kind: AWSMachineTemplate
    metadata:
      name: test-wc-minimal-control-plane-e573dc9a
      namespace: org-giantswarm
      labels:
        cluster.x-k8s.io/role: control-plane
        app: cluster-aws
        app.kubernetes.io/managed-by: Helm
        cluster.x-k8s.io/cluster-name: test-wc-minimal
        giantswarm.io/cluster: test-wc-minimal
        giantswarm.io/organization: test
        cluster.x-k8s.io/watch-filter: capi
        helm.sh/chart: cluster-aws-0.33.0
        application.giantswarm.io/team: hydra
        app.kubernetes.io/version: 0.33.0
    spec:
      template:
        metadata:
          labels:
            cluster.x-k8s.io/role: control-plane
            app: cluster-aws
            app.kubernetes.io/managed-by: Helm
            cluster.x-k8s.io/cluster-name: test-wc-minimal
            giantswarm.io/cluster: test-wc-minimal
            giantswarm.io/organization: test
            cluster.x-k8s.io/watch-filter: capi
            helm.sh/chart: cluster-aws-0.33.0
            application.giantswarm.io/team: hydra
        spec:
          ami: {}
          imageLookupBaseOS: flatcar-stable
          imageLookupFormat: capa-ami-{{.BaseOS}}-v{{.K8sVersion}}-gs
          imageLookupOrg: 706635527432
          cloudInit: {}
          instanceType: r6i.xlarge
          nonRootVolumes:
          - type: gp3
            deviceName: /dev/xvdc
            encrypted: true
            size: 100
          - type: gp3
            deviceName: /dev/xvdd
            encrypted: true
            size: 100
          - type: gp3
            deviceName: /dev/xvde
            encrypted: true
            size: 100
          rootVolume:
            type: gp3
            size: 120
          iamInstanceProfile: control-plane-test-wc-minimal
          sshKeyName: 
          subnet:
            filters:
            - name: "tag:kubernetes.io/cluster/test-wc-minimal"
              values:
              - shared
              - owned
            - name: "tag:sigs.k8s.io/cluster-api-provider-aws/role"
              values:
              - private
    # Source: cluster-aws/templates/list.yaml
    apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
    kind: KubeadmConfigTemplate
    metadata:
      labels:
        cluster.x-k8s.io/role: bastion
        app: cluster-aws
        app.kubernetes.io/managed-by: Helm
        cluster.x-k8s.io/cluster-name: test-wc-minimal
        giantswarm.io/cluster: test-wc-minimal
        giantswarm.io/organization: test
        cluster.x-k8s.io/watch-filter: capi
        helm.sh/chart: cluster-aws-0.33.0
        application.giantswarm.io/team: hydra
      name: test-wc-minimal-bastion-80ebd88e
      namespace: org-giantswarm
    spec:
      template:
        spec:
          format: ignition
          ignition:
            containerLinuxConfig:
              additionalConfig: |
                systemd:
                  units:
                  - name: kubereserved.slice
                    path: /etc/systemd/system/kubereserved.slice
                    content: |
                      [Unit]
                      Description=Limited resources slice for Kubernetes services
                      Documentation=man:systemd.special(7)
                      DefaultDependencies=no
                      Before=slices.target
                      Requires=-.slice
                      After=-.slice
                  - name: kubeadm.service
                    dropins:
                    - name: 10-flatcar.conf
                      contents: |
                        [Unit]
                        # kubeadm must run after coreos-metadata populated /run/metadata directory.
                        Requires=coreos-metadata.service
                        After=coreos-metadata.service
                        [Service]
                        # Ensure kubeadm service has access to kubeadm binary in /opt/bin on Flatcar.
                        Environment=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/bin
                        # To make metadata environment variables available for pre-kubeadm commands.
                        EnvironmentFile=/run/metadata/*
                  - name: containerd.service
                    enabled: true
                    contents: |
                    dropins:
                      - name: 10-change-cgroup.conf
                        contents: |
                          [Service]
                          CPUAccounting=true
                          MemoryAccounting=true
                          Slice=kubereserved.slice
                  - name: os-hardening.service
                    enabled: true
                    contents: |
                      [Unit]
                      Description=Apply os hardening
                      [Service]
                      Type=oneshot
                      ExecStartPre=-/bin/bash -c "gpasswd -d core rkt; gpasswd -d core docker; gpasswd -d core wheel"
                      ExecStartPre=/bin/bash -c "until [ -f '/etc/sysctl.d/hardening.conf' ]; do echo Waiting for sysctl file; sleep 1s;done;"
                      ExecStart=/usr/sbin/sysctl -p /etc/sysctl.d/hardening.conf
                      [Install]
                      WantedBy=multi-user.target
                  - name: audit-rules.service
                    enabled: true
                    dropins:
                    - name: 10-wait-for-containerd.conf
                      contents: |
                        [Service]
                        ExecStartPre=/bin/bash -c "while [ ! -f /etc/audit/rules.d/containerd.rules ]; do echo 'Waiting for /etc/audit/rules.d/containerd.rules to be written' && sleep 1; done"
                  - name: update-engine.service
                    enabled: false
                    mask: true
                  - name: locksmithd.service
                    enabled: false
                    mask: true
                
          preKubeadmCommands:
          - "envsubst < /etc/kubeadm.yml > /etc/kubeadm.yml.tmp"
          - "mv /etc/kubeadm.yml.tmp /etc/kubeadm.yml"
          - "systemctl restart sshd"
          - "sleep infinity"
          files:
          - path: /etc/ssh/trusted-user-ca-keys.pem
            permissions: 0600
            encoding: base64
            content: c3NoLWVkMjU1MTkgQUFBQUMzTnphQzFsWkRJMU5URTVBQUFBSU00Y3ZaMDFmTG1POWNKYldVajdzZkYrTmhFQ2d5K0NsMGJhelNyWlg3c1UgdmF1bHQtY2FAdmF1bHQub3BlcmF0aW9ucy5naWFudHN3YXJtLmlvCg==
          - path: /etc/ssh/sshd_config
            permissions: 0600
            encoding: base64
            content: IyBVc2UgbW9zdCBkZWZhdWx0cyBmb3Igc3NoZCBjb25maWd1cmF0aW9uLgpTdWJzeXN0ZW0gc2Z0cCBpbnRlcm5hbC1zZnRwCkNsaWVudEFsaXZlSW50ZXJ2YWwgMTgwClVzZUROUyBubwpVc2VQQU0geWVzClByaW50TGFzdExvZyBubyAjIGhhbmRsZWQgYnkgUEFNClByaW50TW90ZCBubyAjIGhhbmRsZWQgYnkgUEFNCiMgTm9uIGRlZmF1bHRzICgjMTAwKQpDbGllbnRBbGl2ZUNvdW50TWF4IDIKUGFzc3dvcmRBdXRoZW50aWNhdGlvbiBubwpUcnVzdGVkVXNlckNBS2V5cyAvZXRjL3NzaC90cnVzdGVkLXVzZXItY2Eta2V5cy5wZW0KTWF4QXV0aFRyaWVzIDUKTG9naW5HcmFjZVRpbWUgNjAKQWxsb3dUY3BGb3J3YXJkaW5nIHllcwpBbGxvd0FnZW50Rm9yd2FyZGluZyB5ZXMKQ0FTaWduYXR1cmVBbGdvcml0aG1zIGVjZHNhLXNoYTItbmlzdHAyNTYsZWNkc2Etc2hhMi1uaXN0cDM4NCxlY2RzYS1zaGEyLW5pc3RwNTIxLHNzaC1lZDI1NTE5LHJzYS1zaGEyLTUxMixyc2Etc2hhMi0yNTYsc3NoLXJzYQpQb3J0IDMwMDAwClBvcnQgMjIK
          users:
          - name: giantswarm
            groups: sudo
            sudo: "ALL=(ALL) NOPASSWD:ALL"
    
  

/data/registry-config.toml  (Secret/default/test-wc-minimal-registry-configuration)
  ± value change
    - W3BsdWdpbnMuImlvLmNvbnRhaW5lcmQuZ3JwYy52MS5jcmkiLnJlZ2lzdHJ5XQogIFtwbHVnaW5zLiJpby5jb250YWluZXJkLmdycGMudjEuY3JpIi5yZWdpc3RyeS5taXJyb3JzXQogICAgW3BsdWdpbnMuImlvLmNvbnRhaW5lcmQuZ3JwYy52MS5jcmkiLnJlZ2lzdHJ5Lm1pcnJvcnMuImRvY2tlci5pbyJdCiAgICAgIGVuZHBvaW50ID0gWyJodHRwczovL3JlZ2lzdHJ5LTEuZG9ja2VyLmlvIiwiaHR0cHM6Ly9naWFudHN3YXJtLmF6dXJlY3IuaW8iLF0KICAgIFtwbHVnaW5zLiJpby5jb250YWluZXJkLmdycGMudjEuY3JpIi5yZWdpc3RyeS5taXJyb3JzLiJ3aXRoLWF1dGguZXhhbXBsZS5jb20iXQogICAgICBlbmRwb2ludCA9IFsiaHR0cHM6Ly93aXRoLWF1dGguZXhhbXBsZS5jb20iLCJodHRwczovL3F1YXkuaW8iLF0KW3BsdWdpbnMuImlvLmNvbnRhaW5lcmQuZ3JwYy52MS5jcmkiLnJlZ2lzdHJ5LmNvbmZpZ3NdCiAgW3BsdWdpbnMuImlvLmNvbnRhaW5lcmQuZ3JwYy52MS5jcmkiLnJlZ2lzdHJ5LmNvbmZpZ3MuIndpdGgtYXV0aC5leGFtcGxlLmNvbSIuYXV0aF0KICAgICAgYXV0aCA9ICJaMmxoYm5SemQyRnliWEIxYkd3NllXSmpaR1ZtIgo=
    + dmVyc2lvbiA9IDIKCiMgcmVjb21tZW5kZWQgZGVmYXVsdHMgZnJvbSBodHRwczovL2dpdGh1Yi5jb20vY29udGFpbmVyZC9jb250YWluZXJkL2Jsb2IvbWFpbi9kb2NzL29wcy5tZCNiYXNlLWNvbmZpZ3VyYXRpb24KIyBzZXQgY29udGFpbmVyZCBhcyBhIHN1YnJlYXBlciBvbiBsaW51eCB3aGVuIGl0IGlzIG5vdCBydW5uaW5nIGFzIFBJRCAxCnN1YnJlYXBlciA9IHRydWUKIyBzZXQgY29udGFpbmVyZCdzIE9PTSBzY29yZQpvb21fc2NvcmUgPSAtOTk5CmRpc2FibGVkX3BsdWdpbnMgPSBbXQpbcGx1Z2lucy4iY29udGFpbmVyZC5ydW50aW1lLnYxLmxpbnV4Il0KIyBzaGltIGJpbmFyeSBuYW1lL3BhdGgKc2hpbSA9ICJjb250YWluZXJkLXNoaW0iCiMgcnVudGltZSBiaW5hcnkgbmFtZS9wYXRoCnJ1bnRpbWUgPSAicnVuYyIKIyBkbyBub3QgdXNlIGEgc2hpbSB3aGVuIHN0YXJ0aW5nIGNvbnRhaW5lcnMsIHNhdmVzIG9uIG1lbW9yeSBidXQKIyBsaXZlIHJlc3RvcmUgaXMgbm90IHN1cHBvcnRlZApub19zaGltID0gZmFsc2UKCltwbHVnaW5zLiJpby5jb250YWluZXJkLmdycGMudjEuY3JpIi5jb250YWluZXJkLnJ1bnRpbWVzLnJ1bmNdCiMgc2V0dGluZyBydW5jLm9wdGlvbnMgdW5zZXRzIHBhcmVudCBzZXR0aW5ncwpydW50aW1lX3R5cGUgPSAiaW8uY29udGFpbmVyZC5ydW5jLnYyIgpbcGx1Z2lucy4iaW8uY29udGFpbmVyZC5ncnBjLnYxLmNyaSIuY29udGFpbmVyZC5ydW50aW1lcy5ydW5jLm9wdGlvbnNdClN5c3RlbWRDZ3JvdXAgPSB0cnVlCltwbHVnaW5zLiJpby5jb250YWluZXJkLmdycGMudjEuY3JpIl0Kc2FuZGJveF9pbWFnZSA9ICJxdWF5LmlvL2dpYW50c3dhcm0vcGF1c2U6My45IgoKW3BsdWdpbnMuImlvLmNvbnRhaW5lcmQuZ3JwYy52MS5jcmkiLnJlZ2lzdHJ5XQogIFtwbHVnaW5zLiJpby5jb250YWluZXJkLmdycGMudjEuY3JpIi5yZWdpc3RyeS5taXJyb3JzXQogICAgW3BsdWdpbnMuImlvLmNvbnRhaW5lcmQuZ3JwYy52MS5jcmkiLnJlZ2lzdHJ5Lm1pcnJvcnMuImRvY2tlci5pbyJdCiAgICAgIGVuZHBvaW50ID0gWyJodHRwczovL3JlZ2lzdHJ5LTEuZG9ja2VyLmlvIiwiaHR0cHM6Ly9naWFudHN3YXJtLmF6dXJlY3IuaW8iLF0KICAgIFtwbHVnaW5zLiJpby5jb250YWluZXJkLmdycGMudjEuY3JpIi5yZWdpc3RyeS5taXJyb3JzLiJ3aXRoLWF1dGguZXhhbXBsZS5jb20iXQogICAgICBlbmRwb2ludCA9IFsiaHR0cHM6Ly93aXRoLWF1dGguZXhhbXBsZS5jb20iLCJodHRwczovL3F1YXkuaW8iLF0KW3BsdWdpbnMuImlvLmNvbnRhaW5lcmQuZ3JwYy52MS5jcmkiLnJlZ2lzdHJ5LmNvbmZpZ3NdCiAgW3BsdWdpbnMuImlvLmNvbnRhaW5lcmQuZ3JwYy52MS5jcmkiLnJlZ2lzdHJ5LmNvbmZpZ3MuIndpdGgtYXV0aC5leGFtcGxlLmNvbSIuYXV0aF0KICAgICAgYXV0aCA9ICJaMmxoYm5SemQyRnliWEIxYkd3NllXSmpaR1ZtIgo=
  

/spec/awsLaunchTemplate/imageLookupBaseOS  (AWSMachinePool/org-giantswarm/test-wc-minimal-def00)
  ± value change
    - ubuntu-20.04
    + flatcar-stable

/spec  (KubeadmConfig/org-giantswarm/test-wc-minimal-def00)
  + two map entries added:
    format: ignition
    ignition:
      containerLinuxConfig:
        additionalConfig: |
          systemd:
            units:
              - name: kubereserved.slice
                path: /etc/systemd/system/kubereserved.slice
                content: |
                  [Unit]
                  Description=Limited resources slice for Kubernetes services
                  Documentation=man:systemd.special(7)
                  DefaultDependencies=no
                  Before=slices.target
                  Requires=-.slice
                  After=-.slice
              - name: kubeadm.service
                dropins:
                - name: 10-flatcar.conf
                  contents: |
                    [Unit]
                    # kubeadm must run after coreos-metadata populated /run/metadata directory.
                    Requires=coreos-metadata.service
                    After=coreos-metadata.service
                    [Service]
                    # Ensure kubeadm service has access to kubeadm binary in /opt/bin on Flatcar.
                    Environment=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/bin
                    # To make metadata environment variables available for pre-kubeadm commands.
                    EnvironmentFile=/run/metadata/*
              - name: containerd.service
                enabled: true
                contents: |
                dropins:
                  - name: 10-change-cgroup.conf
                    contents: |
                      [Service]
                      CPUAccounting=true
                      MemoryAccounting=true
                      Slice=kubereserved.slice
              - name: os-hardening.service
                enabled: true
                contents: |
                  [Unit]
                  Description=Apply os hardening
                  [Service]
                  Type=oneshot
                  ExecStartPre=-/bin/bash -c "gpasswd -d core rkt; gpasswd -d core docker; gpasswd -d core wheel"
                  ExecStartPre=/bin/bash -c "until [ -f '/etc/sysctl.d/hardening.conf' ]; do echo Waiting for sysctl file; sleep 1s;done;"
                  ExecStart=/usr/sbin/sysctl -p /etc/sysctl.d/hardening.conf
                  [Install]
                  WantedBy=multi-user.target
              - name: audit-rules.service
                enabled: true
                dropins:
                - name: 10-wait-for-containerd.conf
                  contents: |
                    [Service]
                    ExecStartPre=/bin/bash -c "while [ ! -f /etc/audit/rules.d/containerd.rules ]; do echo 'Waiting for /etc/audit/rules.d/containerd.rules to be written' && sleep 1; done"
              - name: update-engine.service
                enabled: false
                mask: true
              - name: locksmithd.service
                enabled: false
                mask: true
          storage:
            directories:
              - path: /var/lib/kubelet
                mode: 0750
          
    
  

/spec/joinConfiguration/nodeRegistration/kubeletExtraArgs/node-ip  (KubeadmConfig/org-giantswarm/test-wc-minimal-def00)
  ± value change
    - {{ ds.meta_data.local_ipv4 }}
    + ${COREOS_EC2_IPV4_LOCAL}

/spec/joinConfiguration/nodeRegistration/name  (KubeadmConfig/org-giantswarm/test-wc-minimal-def00)
  ± value change
    - {{ ds.meta_data.local_hostname }}
    + ${COREOS_EC2_HOSTNAME}

/spec/preKubeadmCommands  (KubeadmConfig/org-giantswarm/test-wc-minimal-def00)
  - one list entry removed:
    - "/bin/test ! -d /var/lib/kubelet && (/bin/mkdir -p /var/lib/kubelet && /bin/chmod 0750 /var/lib/kubelet)"
    
  
  + two list entries added:
    - "envsubst < /etc/kubeadm.yml > /etc/kubeadm.yml.tmp"
    - "mv /etc/kubeadm.yml.tmp /etc/kubeadm.yml"
  

/spec/postKubeadmCommands  (KubeadmConfig/org-giantswarm/test-wc-minimal-def00)
  - one list entry removed:
    - "/bin/sh /opt/set-aws-ntp.sh"

/spec/files  (KubeadmConfig/org-giantswarm/test-wc-minimal-def00)
  - three list entries removed:
    - path: /lib/systemd/logind.conf.d/zzz-kubelet-graceful-shutdown.conf
      permissions: 0700
      encoding: base64
      content: W0xvZ2luXQojIGRlbGF5CkluaGliaXREZWxheU1heFNlYz0zMDAK
    - path: /etc/containerd/conf.d/registry-config.toml
      permissions: 0600
      contentFrom:
        secret:
          name: test-wc-minimal-registry-configuration
          key: registry-config.toml
    - path: /opt/set-aws-ntp.sh
      permissions: 0700
      encoding: base64
      content: IyEvYmluL2Jhc2gKCk5UUF9TRVJWRVI9IiQoY2F0IC9ldGMvY2hyb255L2Nocm9ueS5jb25mIDI+L2Rldi9udWxsIHwgZ3JlcCAxNjkuMjU0LjE2OS4xMjMpIgoKaWYgWyAteiAiJE5UUF9TRVJWRVIiIF0KdGhlbgogIGVjaG8gIk5UUCBzZXJ2ZXIgaXMgbm90IHNldCwgc2V0dGluZyBpdCB0byBBV1MgTlRQIFNlcnZlciIKICBlY2hvICJzZXJ2ZXIgMTY5LjI1NC4xNjkuMTIzIHByZWZlciBpYnVyc3QiID4+IC9ldGMvY2hyb255L2Nocm9ueS5jb25mCiAgc2VydmljZSBjaHJvbnkgZm9yY2UtcmVsb2FkCmZpCg==
    
  
  + four list entries added:
    - path: /etc/systemd/logind.conf.d/zzz-kubelet-graceful-shutdown.conf
      permissions: 0700
      encoding: base64
      content: W0xvZ2luXQojIGRlbGF5CkluaGliaXREZWxheU1heFNlYz0zMDAK
    - path: /etc/containerd/config.toml
      permissions: 0644
      contentFrom:
        secret:
          name: test-wc-minimal-registry-configuration
          key: registry-config.toml
    - path: /etc/systemd/timesyncd.conf
      permissions: 0644
      encoding: base64
      content: W1RpbWVdCk5UUD0xNjkuMjU0LjE2OS4xMjMK
    - path: /etc/sysctl.d/hardening.conf
      permissions: 0644
      encoding: base64
      content: ZnMuaW5vdGlmeS5tYXhfdXNlcl93YXRjaGVzID0gMTYzODQKZnMuaW5vdGlmeS5tYXhfdXNlcl9pbnN0YW5jZXMgPSA4MTkyCmtlcm5lbC5rcHRyX3Jlc3RyaWN0ID0gMgprZXJuZWwuc3lzcnEgPSAwCm5ldC5pcHY0LmNvbmYuYWxsLmxvZ19tYXJ0aWFucyA9IDEKbmV0LmlwdjQuY29uZi5hbGwuc2VuZF9yZWRpcmVjdHMgPSAwCm5ldC5pcHY0LmNvbmYuZGVmYXVsdC5hY2NlcHRfcmVkaXJlY3RzID0gMApuZXQuaXB2NC5jb25mLmRlZmF1bHQubG9nX21hcnRpYW5zID0gMQpuZXQuaXB2NC50Y3BfdGltZXN0YW1wcyA9IDAKbmV0LmlwdjYuY29uZi5hbGwuYWNjZXB0X3JlZGlyZWN0cyA9IDAKbmV0LmlwdjYuY29uZi5kZWZhdWx0LmFjY2VwdF9yZWRpcmVjdHMgPSAwCiMgSW5jcmVhc2VkIG1tYXBmcyBiZWNhdXNlIHNvbWUgYXBwbGljYXRpb25zLCBsaWtlIEVTLCBuZWVkIGhpZ2hlciBsaW1pdCB0byBzdG9yZSBkYXRhIHByb3Blcmx5CnZtLm1heF9tYXBfY291bnQgPSAyNjIxNDQKIyBSZXNlcnZlZCB0byBhdm9pZCBjb25mbGljdHMgd2l0aCBrdWJlLWFwaXNlcnZlciwgd2hpY2ggYWxsb2NhdGVzIHdpdGhpbiB0aGlzIHJhbmdlCm5ldC5pcHY0LmlwX2xvY2FsX3Jlc2VydmVkX3BvcnRzPTMwMDAwLTMyNzY3Cm5ldC5pcHY0LmNvbmYuYWxsLnJwX2ZpbHRlciA9IDEKbmV0LmlwdjQuY29uZi5hbGwuYXJwX2lnbm9yZSA9IDEKbmV0LmlwdjQuY29uZi5hbGwuYXJwX2Fubm91bmNlID0gMgoKIyBUaGVzZSBhcmUgcmVxdWlyZWQgZm9yIHRoZSBrdWJlbGV0ICctLXByb3RlY3Qta2VybmVsLWRlZmF1bHRzJyBmbGFnCiMgU2VlIGh0dHBzOi8vZ2l0aHViLmNvbS9naWFudHN3YXJtL2dpYW50c3dhcm0vaXNzdWVzLzEzNTg3CnZtLm92ZXJjb21taXRfbWVtb3J5PTEKa2VybmVsLnBhbmljPTEwCmtlcm5lbC5wYW5pY19vbl9vb3BzPTEK
    
  

/spec/machineTemplate/infrastructureRef/name  (KubeadmControlPlane/org-giantswarm/test-wc-minimal)
  ± value change
    - test-wc-minimal-control-plane-04636bc2
    + test-wc-minimal-control-plane-e573dc9a

/spec/kubeadmConfigSpec  (KubeadmControlPlane/org-giantswarm/test-wc-minimal)
  + two map entries added:
    format: ignition
    ignition:
      containerLinuxConfig:
        additionalConfig: |
          systemd:
            units:
              - name: kubereserved.slice
                path: /etc/systemd/system/kubereserved.slice
                content: |
                  [Unit]
                  Description=Limited resources slice for Kubernetes services
                  Documentation=man:systemd.special(7)
                  DefaultDependencies=no
                  Before=slices.target
                  Requires=-.slice
                  After=-.slice
              - name: kubeadm.service
                dropins:
                - name: 10-flatcar.conf
                  contents: |
                    [Unit]
                    # kubeadm must run after coreos-metadata populated /run/metadata directory.
                    Requires=coreos-metadata.service
                    After=coreos-metadata.service
                    [Service]
                    # Ensure kubeadm service has access to kubeadm binary in /opt/bin on Flatcar.
                    Environment=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/bin
                    # To make metadata environment variables available for pre-kubeadm commands.
                    EnvironmentFile=/run/metadata/*
              - name: containerd.service
                enabled: true
                contents: |
                dropins:
                  - name: 10-change-cgroup.conf
                    contents: |
                      [Service]
                      CPUAccounting=true
                      MemoryAccounting=true
                      Slice=kubereserved.slice
              - name: os-hardening.service
                enabled: true
                contents: |
                  [Unit]
                  Description=Apply os hardening
                  [Service]
                  Type=oneshot
                  ExecStartPre=-/bin/bash -c "gpasswd -d core rkt; gpasswd -d core docker; gpasswd -d core wheel"
                  ExecStartPre=/bin/bash -c "until [ -f '/etc/sysctl.d/hardening.conf' ]; do echo Waiting for sysctl file; sleep 1s;done;"
                  ExecStart=/usr/sbin/sysctl -p /etc/sysctl.d/hardening.conf
                  [Install]
                  WantedBy=multi-user.target
              - name: audit-rules.service
                enabled: true
                dropins:
                - name: 10-wait-for-containerd.conf
                  contents: |
                    [Service]
                    ExecStartPre=/bin/bash -c "while [ ! -f /etc/audit/rules.d/containerd.rules ]; do echo 'Waiting for /etc/audit/rules.d/containerd.rules to be written' && sleep 1; done"
              - name: update-engine.service
                enabled: false
                mask: true
              - name: locksmithd.service
                enabled: false
                mask: true
              - name: var-lib-etcd.mount
                enabled: true
                contents: |
                  [Unit] 
                  Description=etcd volume
                  DefaultDependencies=no
                  [Mount]
                  What=/dev/disk/by-label/etcd
                  Where=/var/lib/etcd
                  Type=xfs
                  [Install]
                  WantedBy=local-fs-pre.target
              - name: var-lib-kubelet.mount
                enabled: true
                contents: |
                  [Unit]
                  Description=kubelet volume
                  DefaultDependencies=no
                  [Mount]
                  What=/dev/disk/by-label/kubelet
                  Where=/var/lib/kubelet
                  Type=xfs
                  [Install]
                  WantedBy=local-fs-pre.target
              - name: var-lib-containerd.mount
                enabled: true
                contents: |
                  [Unit]
                  Description=containerd volume
                  DefaultDependencies=no
                  [Mount]
                  What=/dev/disk/by-label/containerd
                  Where=/var/lib/containerd
                  Type=xfs
                  [Install]
                  WantedBy=local-fs-pre.target
          storage:
            filesystems:
              - name: etcd
                mount:
                  device: /dev/xvdc
                  wipeFilesystem: true
                  label: etcd
                  format: xfs
              - name: containerd
                mount:
                  device: /dev/xvdd
                  wipeFilesystem: true
                  label: containerd
                  format: xfs
              - name: kubelet
                mount:
                  device: /dev/xvde
                  wipeFilesystem: true
                  label: kubelet
                  format: xfs
            directories:
              - path: /var/lib/kubelet
                mode: 0750
          
    
  

/spec/kubeadmConfigSpec/files  (KubeadmControlPlane/org-giantswarm/test-wc-minimal)
  - four list entries removed:
    - path: /opt/init-disks.sh
      permissions: 0700
      encoding: base64
      content: IyEvYmluL3NoCiMgY2hlY2sgZm9yIE5WTUUgZGlza3MKaWYgWyAiYGxzYmxrIHwgZ3JlcCBudm1lIHwgd2MgLWxgIiAtZ2UgNCBdOyB0aGVuCglFVENEX0RJU0s9Ii9kZXYvbnZtZTFuMSIKCUNPTlRBSU5FUkRfRElTSz0iL2Rldi9udm1lMm4xIgoJS1VCRUxFVF9ESVNLPSIvZGV2L252bWUzbjEiCgllY2hvIGZvdW5kIG52bWUgRElTS3MKZWxzZQoJRVRDRF9ESVNLPSIvZGV2L3h2ZGMiCiAgICAgICAgQ09OVEFJTkVSRF9ESVNLPSIvZGV2L3h2ZGQiCiAgICAgICAgS1VCRUxFVF9ESVNLPSIvZGV2L3h2ZGUiCmZpCgojIGluaXQgZXRjZCBkaXNrCm1rZnMueGZzICRFVENEX0RJU0sKbWtkaXIgLXAgL3Zhci9saWIvZXRjZAptb3VudCAkRVRDRF9ESVNLIC92YXIvbGliL2V0Y2QKCiMgaW5pdCBrdWJlbGV0IGRpc2sKbWtmcy54ZnMgJEtVQkVMRVRfRElTSwpta2RpciAtcCAvdmFyL2xpYi9rdWJlbGV0Cm1vdW50ICRLVUJFTEVUX0RJU0sgL3Zhci9saWIva3ViZWxldAoKIyBpbml0IGNvbnRhaW5lcmQgZGlzawpta2ZzLnhmcyAkQ09OVEFJTkVSRF9ESVNLCnN5c3RlbWN0bCBzdG9wIGNvbnRhaW5lcmQKcm0gLXJmIC92YXIvbGliL2NvbnRhaW5lcmQKbWtkaXIgLXAgL3Zhci9saWIvY29udGFpbmVyZAptb3VudCAkQ09OVEFJTkVSRF9ESVNLIC92YXIvbGliL2NvbnRhaW5lcmQKc3lzdGVtY3RsIHN0YXJ0IGNvbnRhaW5lcmQK
    - path: /lib/systemd/logind.conf.d/zzz-kubelet-graceful-shutdown.conf
      permissions: 0700
      encoding: base64
      content: W0xvZ2luXQojIGRlbGF5CkluaGliaXREZWxheU1heFNlYz0zMDAK
    - path: /opt/set-aws-ntp.sh
      permissions: 0700
      encoding: base64
      content: IyEvYmluL2Jhc2gKCk5UUF9TRVJWRVI9IiQoY2F0IC9ldGMvY2hyb255L2Nocm9ueS5jb25mIDI+L2Rldi9udWxsIHwgZ3JlcCAxNjkuMjU0LjE2OS4xMjMpIgoKaWYgWyAteiAiJE5UUF9TRVJWRVIiIF0KdGhlbgogIGVjaG8gIk5UUCBzZXJ2ZXIgaXMgbm90IHNldCwgc2V0dGluZyBpdCB0byBBV1MgTlRQIFNlcnZlciIKICBlY2hvICJzZXJ2ZXIgMTY5LjI1NC4xNjkuMTIzIHByZWZlciBpYnVyc3QiID4+IC9ldGMvY2hyb255L2Nocm9ueS5jb25mCiAgc2VydmljZSBjaHJvbnkgZm9yY2UtcmVsb2FkCmZpCg==
    - path: /etc/containerd/conf.d/registry-config.toml
      permissions: 0600
      contentFrom:
        secret:
          name: test-wc-minimal-registry-configuration
          key: registry-config.toml
    
  
  + four list entries added:
    - path: /etc/systemd/logind.conf.d/zzz-kubelet-graceful-shutdown.conf
      permissions: 0700
      encoding: base64
      content: W0xvZ2luXQojIGRlbGF5CkluaGliaXREZWxheU1heFNlYz0zMDAK
    - path: /etc/systemd/timesyncd.conf
      permissions: 0644
      encoding: base64
      content: W1RpbWVdCk5UUD0xNjkuMjU0LjE2OS4xMjMK
    - path: /etc/sysctl.d/hardening.conf
      permissions: 0644
      encoding: base64
      content: ZnMuaW5vdGlmeS5tYXhfdXNlcl93YXRjaGVzID0gMTYzODQKZnMuaW5vdGlmeS5tYXhfdXNlcl9pbnN0YW5jZXMgPSA4MTkyCmtlcm5lbC5rcHRyX3Jlc3RyaWN0ID0gMgprZXJuZWwuc3lzcnEgPSAwCm5ldC5pcHY0LmNvbmYuYWxsLmxvZ19tYXJ0aWFucyA9IDEKbmV0LmlwdjQuY29uZi5hbGwuc2VuZF9yZWRpcmVjdHMgPSAwCm5ldC5pcHY0LmNvbmYuZGVmYXVsdC5hY2NlcHRfcmVkaXJlY3RzID0gMApuZXQuaXB2NC5jb25mLmRlZmF1bHQubG9nX21hcnRpYW5zID0gMQpuZXQuaXB2NC50Y3BfdGltZXN0YW1wcyA9IDAKbmV0LmlwdjYuY29uZi5hbGwuYWNjZXB0X3JlZGlyZWN0cyA9IDAKbmV0LmlwdjYuY29uZi5kZWZhdWx0LmFjY2VwdF9yZWRpcmVjdHMgPSAwCiMgSW5jcmVhc2VkIG1tYXBmcyBiZWNhdXNlIHNvbWUgYXBwbGljYXRpb25zLCBsaWtlIEVTLCBuZWVkIGhpZ2hlciBsaW1pdCB0byBzdG9yZSBkYXRhIHByb3Blcmx5CnZtLm1heF9tYXBfY291bnQgPSAyNjIxNDQKIyBSZXNlcnZlZCB0byBhdm9pZCBjb25mbGljdHMgd2l0aCBrdWJlLWFwaXNlcnZlciwgd2hpY2ggYWxsb2NhdGVzIHdpdGhpbiB0aGlzIHJhbmdlCm5ldC5pcHY0LmlwX2xvY2FsX3Jlc2VydmVkX3BvcnRzPTMwMDAwLTMyNzY3Cm5ldC5pcHY0LmNvbmYuYWxsLnJwX2ZpbHRlciA9IDEKbmV0LmlwdjQuY29uZi5hbGwuYXJwX2lnbm9yZSA9IDEKbmV0LmlwdjQuY29uZi5hbGwuYXJwX2Fubm91bmNlID0gMgoKIyBUaGVzZSBhcmUgcmVxdWlyZWQgZm9yIHRoZSBrdWJlbGV0ICctLXByb3RlY3Qta2VybmVsLWRlZmF1bHRzJyBmbGFnCiMgU2VlIGh0dHBzOi8vZ2l0aHViLmNvbS9naWFudHN3YXJtL2dpYW50c3dhcm0vaXNzdWVzLzEzNTg3CnZtLm92ZXJjb21taXRfbWVtb3J5PTEKa2VybmVsLnBhbmljPTEwCmtlcm5lbC5wYW5pY19vbl9vb3BzPTEK
    - path: /etc/containerd/config.toml
      permissions: 0644
      contentFrom:
        secret:
          name: test-wc-minimal-registry-configuration
          key: registry-config.toml
    
  

/spec/kubeadmConfigSpec/initConfiguration/nodeRegistration/kubeletExtraArgs/node-ip  (KubeadmControlPlane/org-giantswarm/test-wc-minimal)
  ± value change
    - {{ ds.meta_data.local_ipv4 }}
    + ${COREOS_EC2_IPV4_LOCAL}

/spec/kubeadmConfigSpec/initConfiguration/nodeRegistration/name  (KubeadmControlPlane/org-giantswarm/test-wc-minimal)
  ± value change
    - {{ ds.meta_data.local_hostname }}
    + ${COREOS_EC2_HOSTNAME}

/spec/kubeadmConfigSpec/joinConfiguration/nodeRegistration/name  (KubeadmControlPlane/org-giantswarm/test-wc-minimal)
  ± value change
    - {{ ds.meta_data.local_hostname }}
    + ${COREOS_EC2_HOSTNAME}

/spec/kubeadmConfigSpec/preKubeadmCommands  (KubeadmControlPlane/org-giantswarm/test-wc-minimal)
  - two list entries removed:
    - "/bin/test ! -d /var/lib/kubelet && (/bin/mkdir -p /var/lib/kubelet && /bin/chmod 0750 /var/lib/kubelet)"
    - "/bin/sh /opt/init-disks.sh"
    
  
  + two list entries added:
    - "envsubst < /etc/kubeadm.yml > /etc/kubeadm.yml.tmp"
    - "mv /etc/kubeadm.yml.tmp /etc/kubeadm.yml"
  

/spec/kubeadmConfigSpec/postKubeadmCommands  (KubeadmControlPlane/org-giantswarm/test-wc-minimal)
  - one list entry removed:
    - "/bin/sh /opt/set-aws-ntp.sh"

/spec/template/spec/bootstrap/configRef/name  (MachineDeployment/org-giantswarm/test-wc-minimal-bastion)
  ± value change
    - test-wc-minimal-bastion-b2e9a9ff
    + test-wc-minimal-bastion-80ebd88e

primeroz · 2023-06-21T09:03:07Z

helm/cluster-aws/templates/_helpers.tpl

 {{- end -}}

-{{- define "sshPreKubeadmCommands" -}}
- systemctl restart sshd
+{{- define "diskStorageConfig" -}}


if you have not yet you might want to double check that the filesystems are actually being mounted

in capz we hit a bug with the way that the systemd mount units are generated see

Creating partitions with Ignition bootstrap format fails when Ignition 3.1 is used. kubernetes-sigs/cluster-api#7679

https://github.com/giantswarm/cluster-azure/blob/main/helm/cluster-azure/templates/_kcp.tpl#L72-L88

where essentially we needed to have both kubeadmConfigSpec.diskSetup and ignition.containerLinuxConfig.additionalConfig.storage to actually get the right volume mounted on the right path

pretty sure the filesystems are mounted, but I created custom systemd units for the mounts maybe that's the difference?
https://github.com/giantswarm/cluster-aws/pull/322/files#diff-72301810e9b39353e074da2c29eadb7e61d61ca28007476794b2a07103336500R174-R211

yeah that could be 👍 thanks for checking

calvix added 23 commits March 31, 2023 11:37

enable-flatcar-os

3194607

fix-nodes

77c773c

enable-flatcar-os

93e4af4

Merge branch 'master' into enable-flatcar-os

78f18a9

enable-flatcar-os

cbc9f65

Merge branch 'enable-flatcar-os' of github.com:giantswarm/cluster-aws…

28cb207

… into enable-flatcar-os

enable-flatcar-os

5a6ce8d

enable-flatcar-os

1382a58

enable-flatcar-os

706b978

enable-flatcar-os

b50b496

enable-flatcar-os

c0c2045

enable-flatcar-os

b88667e

flatcar-nodes

1c721da

enable-flatcar-os

e210fac

rollback

1fc7849

Merge branch 'master' into enable-flatcar-os

de90445

Merge branch 'enable-flatcar-os' into flatcar-nodes

49c0b8b

enable-flatcar-os

eb1418d

enable-flatcar-os

c9fd313

enable-flatcar-os

e4fdb01

enable-flatcar-os

fe7f7a2

Merge branch 'enable-flatcar-os' into flatcar-nodes

ead0893

flatcar-nodes

a92025b

calvix added 6 commits June 14, 2023 09:47

flatcar-nodes

7035bdb

flatcar-nodes

a0f28a4

enable-flatcar-os

210db8d

Merge branch 'enable-flatcar-os' into flatcar-nodes

6f03a3c

enable-flatcar-os

89de7e2

Merge branch 'enable-flatcar-os' into flatcar-nodes

6a45a32

calvix added 9 commits June 14, 2023 13:26

flatcar-nodes

f2f75d6

merge

6b54408

fix

5f8b3b8

flatcar-nodes

74dfc8c

registry

9a56218

sandbox-image-from-values

24d50b9

normalize

86c28af

flatcar-nodes

24890dc

audit

bf75eb4

calvix self-assigned this Jun 16, 2023

calvix marked this pull request as ready for review June 16, 2023 12:56

calvix requested a review from a team as a code owner June 16, 2023 12:56

calvix commented Jun 16, 2023

View reviewed changes

AndiDog reviewed Jun 19, 2023

View reviewed changes

calvix added 2 commits June 20, 2023 09:42

remove-comment

dc1b9b0

comment

5be86a6

fiunchinho approved these changes Jun 20, 2023

View reviewed changes

Merge branch 'master' into flatcar-nodes

c12e3d5

changelog-breaking-change

035f24c

primeroz reviewed Jun 21, 2023

View reviewed changes

AndiDog approved these changes Jun 21, 2023

View reviewed changes

calvix merged commit d47b3ee into master Jun 21, 2023

calvix deleted the flatcar-nodes branch June 21, 2023 10:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Flatcar nodes #322

Flatcar nodes #322

calvix commented Jun 14, 2023 •

edited

Loading

tityosbot commented Jun 14, 2023

calvix commented Jun 16, 2023

calvix Jun 16, 2023

calvix Jun 16, 2023

calvix Jun 16, 2023 •

edited

Loading

calvix commented Jun 16, 2023

calvix commented Jun 20, 2023

calvix commented Jun 21, 2023

github-actions bot commented Jun 21, 2023

primeroz Jun 21, 2023

calvix Jun 21, 2023

primeroz Jun 21, 2023

		@@ -0,0 +1,2 @@
		[Time]
		NTP=169.254.169.123

Flatcar nodes #322

Flatcar nodes #322

Conversation

calvix commented Jun 14, 2023 • edited Loading

tityosbot commented Jun 14, 2023

calvix commented Jun 16, 2023

calvix Jun 16, 2023

Choose a reason for hiding this comment

calvix Jun 16, 2023

Choose a reason for hiding this comment

calvix Jun 16, 2023 • edited Loading

Choose a reason for hiding this comment

calvix commented Jun 16, 2023

calvix commented Jun 20, 2023

calvix commented Jun 21, 2023

github-actions bot commented Jun 21, 2023

primeroz Jun 21, 2023

Choose a reason for hiding this comment

calvix Jun 21, 2023

Choose a reason for hiding this comment

primeroz Jun 21, 2023

Choose a reason for hiding this comment

calvix commented Jun 14, 2023 •

edited

Loading

calvix Jun 16, 2023 •

edited

Loading