feat: Support Flatcar OS (#225)

* flatcar support Add preKubeadmCommands for node name replacement Add flatcarDriver in poetry plugin list Enable required feature gate * Support os switch & Fix clusterclas patches * Update ignition Fix ignition ignoring error. Fix lint. Add comment about PATH overwrite in flatcar. * Build flatcar image Remove default os in image build cmd * Land clct openstack provider override * modify json patches for flatcar changed ignition config to match the one used by CAPO * modify CA secret type and metadata CAPI expects a CA secret of type "cluster.x-k8s.io/secret" instead of "kubernetes.io/tls". Also, the "cluster.x-k8s.io/cluster-name" label has been added, containing the cluster's stack id. * ci(flatcar): add image builds + functional tests * ci(flatcar): fix image dependency * ci: fix concurrency for new os * chore: fix concurrency * chore: bump image-builder to v0.1.19 * chore: undo getting-started docs * chore(flatcar): fix image builds * ci: fix image builds for ubuntu * docs: move to link to docs * chore: fix linters * docs: add flatcar info * ci: enable debugging * ci: use correct os_distro * ci: add GITHUB_TOKEN for CAPI installs * ci: fix image name * ci: restore functional testing for all versions --------- Co-authored-by: okozachenko1203 <okozachenko1203@gmail.com> Co-authored-by: Mohammed Naser <mnaser@vexxhost.com>
vexxhost · Oct 2, 2023 · ef4401f · ef4401f
1 parent 391cdac
commit ef4401f
Show file tree

Hide file tree

Showing 12 changed files with 273 additions and 87 deletions.
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
@@ -139,11 +139,17 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
+        os_distro:
+          - ubuntu
+          - flatcar
         kube:
           - v1.24.16
           - v1.25.12
           - v1.26.7
           - v1.27.4
+    concurrency:
+      group: build-images-${{ github.ref }}-${{ matrix.kube }}-${{ matrix.os_distro }}
+      cancel-in-progress: true
     steps:
       - name: Checkout project
         uses: actions/checkout@v3
@@ -166,29 +172,32 @@ jobs:
         timeout-minutes: 30
         run: |
           magnum-cluster-api-image-builder \
-            --operating-system ubuntu-2204 \
+            --operating-system ${{ matrix.os_distro }} \
             --version ${{ matrix.kube }}
         env:
           PACKER_GITHUB_API_TOKEN: ${{ secrets.GITHUB_TOKEN }}
 
       - name: Upload built image
         uses: actions/upload-artifact@v3
         with:
-          name: ubuntu-2204-kube-${{ matrix.kube }}.qcow2
-          path: ubuntu-2204-kube-${{ matrix.kube }}.qcow2
+          name: ${{ matrix.os_distro }}-kube-${{ matrix.kube }}.qcow2
+          path: ${{ matrix.os_distro }}-kube-${{ matrix.kube }}.qcow2
 
   functional:
     runs-on: ubuntu-latest-16-cores
     strategy:
       fail-fast: false
       matrix:
+        os_distro:
+          - ubuntu
+          - flatcar
         kube:
           - v1.24.16
           - v1.25.12
           - v1.26.7
           - v1.27.4
     concurrency:
-      group: ${{ github.ref }}-${{ matrix.kube }}
+      group: functional-${{ github.ref }}-${{ matrix.kube }}-${{ matrix.os_distro }}
       cancel-in-progress: true
     steps:
       - name: Checkout project
@@ -205,6 +214,8 @@ jobs:
 
       - name: Install Magnum with Cluster API
         run: ./hack/stack.sh
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
 
       # - name: Setup "tmate" session
       #   uses: mxschmitt/action-tmate@v3
@@ -214,21 +225,22 @@ jobs:
         uses: lewagon/wait-on-check-action@v1.3.1
         with:
           ref: ${{ github.event.pull_request.head.sha }}
-          check-name: "build-images (${{ matrix.kube }})"
+          check-name: "build-images (${{ matrix.os_distro }}, ${{ matrix.kube }})"
           repo-token: ${{ secrets.GITHUB_TOKEN }}
           wait-interval: 10
 
       - name: Download built image
         if: contains(github.event.pull_request.body, '/build-new-image')
         uses: actions/download-artifact@v3
         with:
-          name: ubuntu-2204-kube-${{ matrix.kube }}.qcow2
+          name: ${{ matrix.os_distro }}-kube-${{ matrix.kube }}.qcow2
 
       - name: Run functional tests
         run: |
           ./hack/run-functional-tests.sh
         env:
           BUILD_NEW_IMAGE: "${{ contains(github.event.pull_request.body, '/build-new-image') }}"
+          OS_DISTRO: "${{ matrix.os_distro }}"
           KUBE_TAG: "${{ matrix.kube }}"
           NODE_COUNT: 2
 

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -289,12 +289,10 @@
 
 ## [0.2.0](https://github.com/vexxhost/magnum-cluster-api/compare/v0.1.2...v0.2.0) (2022-11-16)
 
-
 ### Bug Fixes
 
 * added flux + node labels ([5f04d4e](https://github.com/vexxhost/magnum-cluster-api/commit/5f04d4ed8b00ba0d45cc59edd678bdf72679b00b))
 
-
 ### Miscellaneous Chores
 
 * release 0.2.0 ([9c8fe82](https://github.com/vexxhost/magnum-cluster-api/commit/9c8fe8252e61b43019a0c45d31284467ec99af15))

diff --git a/README.md b/README.md
@@ -23,11 +23,20 @@ published for the latest stable release of Kubernetes.
 You can find the pre-built images for the latest stable release of Kubernetes
 at the following URL:
 
+#### Ubuntu 22.04
+
 * [v1.23.17](https://object-storage.public.mtl1.vexxhost.net/swift/v1/a91f106f55e64246babde7402c21b87a/magnum-capi/ubuntu-2204-kube-v1.23.17.qcow2)
-* [v1.24.15](https://object-storage.public.mtl1.vexxhost.net/swift/v1/a91f106f55e64246babde7402c21b87a/magnum-capi/ubuntu-2204-kube-v1.24.15.qcow2)
-* [v1.25.11](https://object-storage.public.mtl1.vexxhost.net/swift/v1/a91f106f55e64246babde7402c21b87a/magnum-capi/ubuntu-2204-kube-v1.25.11.qcow2)
-* [v1.26.6](https://object-storage.public.mtl1.vexxhost.net/swift/v1/a91f106f55e64246babde7402c21b87a/magnum-capi/ubuntu-2204-kube-v1.26.6.qcow2)
-* [v1.27.3](https://object-storage.public.mtl1.vexxhost.net/swift/v1/a91f106f55e64246babde7402c21b87a/magnum-capi/ubuntu-2204-kube-v1.27.3.qcow2)
+* [v1.24.16](https://object-storage.public.mtl1.vexxhost.net/swift/v1/a91f106f55e64246babde7402c21b87a/magnum-capi/ubuntu-2204-kube-v1.24.16.qcow2)
+* [v1.25.12](https://object-storage.public.mtl1.vexxhost.net/swift/v1/a91f106f55e64246babde7402c21b87a/magnum-capi/ubuntu-2204-kube-v1.25.12.qcow2)
+* [v1.26.7](https://object-storage.public.mtl1.vexxhost.net/swift/v1/a91f106f55e64246babde7402c21b87a/magnum-capi/ubuntu-2204-kube-v1.26.7.qcow2)
+* [v1.27.4](https://object-storage.public.mtl1.vexxhost.net/swift/v1/a91f106f55e64246babde7402c21b87a/magnum-capi/ubuntu-2204-kube-v1.27.4.qcow2)
+
+#### Flatcar
+
+* [v1.24.16](https://object-storage.public.mtl1.vexxhost.net/swift/v1/a91f106f55e64246babde7402c21b87a/magnum-capi/flatcar-kube-v1.24.16.qcow2)
+* [v1.25.12](https://object-storage.public.mtl1.vexxhost.net/swift/v1/a91f106f55e64246babde7402c21b87a/magnum-capi/flatcar-kube-v1.25.12.qcow2)
+* [v1.26.7](https://object-storage.public.mtl1.vexxhost.net/swift/v1/a91f106f55e64246babde7402c21b87a/magnum-capi/flatcar-kube-v1.26.7.qcow2)
+* [v1.27.4](https://object-storage.public.mtl1.vexxhost.net/swift/v1/a91f106f55e64246babde7402c21b87a/magnum-capi/flatcar-kube-v1.27.4.qcow2)
 
 ### Building images
 
@@ -41,52 +50,5 @@ magnum-cluster-api-image-builder
 
 ## Testing & Development
 
-In order to be able to test and develop the `magnum-cluster-api` project, you
-will need to have an existing Magnum deployment.  You can use the following
-steps to be able to test and develop the project.
-
-1. Start up a DevStack environment with all Cluster API dependencies
-
-   ```bash
-   ./hack/stack.sh
-   ```
-
-1. Upload an image to use with Magnum and create cluster templates
-
-   ```bash
-   pushd /tmp
-   source /opt/stack/openrc
-   for version in v1.23.17 v1.24.15 v1.25.11 v1.26.6 v1.27.3; do \
-      curl -LO https://object-storage.public.mtl1.vexxhost.net/swift/v1/a91f106f55e64246babde7402c21b87a/magnum-capi/ubuntu-2204-kube-${version}.qcow2; \
-      openstack image create ubuntu-2204-kube-${version} --disk-format=qcow2 --container-format=bare --property os_distro=ubuntu --file=ubuntu-2204-kube-${version}.qcow2; \
-      openstack coe cluster template create \
-         --image $(openstack image show ubuntu-2204-kube-${version} -c id -f value) \
-         --external-network public \
-         --dns-nameserver 8.8.8.8 \
-         --master-lb-enabled \
-         --master-flavor m1.medium \
-         --flavor m1.medium \
-         --network-driver calico \
-         --docker-storage-driver overlay2 \
-         --coe kubernetes \
-         --label kube_tag=${version} \
-         k8s-${version};
-   done;
-   popd
-   ```
-
-1. Spin up a new cluster using the Cluster API driver
-
-   ```bash
-   openstack coe cluster create \
-     --cluster-template k8s-v1.25.11 \
-     --master-count 3 \
-     --node-count 2 \
-     k8s-v1.25.11
-   ```
-
-1. Once the cluster reaches `CREATE_COMPLETE` state, you can interact with it:
-
-   ```bash
-   eval $(openstack coe cluster config k8s-v1.25.11)
-   ```
+For more information on how to test, develop and contribute to the Cluster API
+driver for Magnum, refer to the [developer guide](https://vexxhost.github.io/magnum-cluster-api/developer/testing-and-development/).
diff --git a/docs/developer/testing-and-development.md b/docs/developer/testing-and-development.md
@@ -15,12 +15,14 @@ steps to be able to test and develop the project.
    ```bash
    pushd /tmp
    source /opt/stack/openrc
-   for version in v1.23.17 v1.24.15 v1.25.11 v1.26.6 v1.27.3; do \
-      curl -LO https://object-storage.public.mtl1.vexxhost.net/swift/v1/a91f106f55e64246babde7402c21b87a/magnum-capi/ubuntu-2204-kube-${version}.qcow2; \
-      openstack image create ubuntu-2204-kube-${version} --disk-format=qcow2 --container-format=bare --property os_distro=ubuntu --file=ubuntu-2204-kube-${version}.qcow2; \
+   export OS_DISTRO=ubuntu # you can change this to "flatcar" if you want to use Flatcar
+   for version in v1.24.16 v1.25.12 v1.26.7 v1.27.4; do \
+      [[ "${OS_DISTRO}" == "ubuntu" ]] && IMAGE_NAME="ubuntu-2204-kube-${version}" || IMAGE_NAME="flatcar-kube-${version}"; \
+      curl -LO https://object-storage.public.mtl1.vexxhost.net/swift/v1/a91f106f55e64246babde7402c21b87a/magnum-capi/${IMAGE_NAME}.qcow2; \
+      openstack image create ${IMAGE_NAME} --disk-format=qcow2 --container-format=bare --property os_distro=${OS_DISTRO} --file=${IMAGE_NAME}.qcow2; \
       openstack coe cluster template create \
-         --image $(openstack image show ubuntu-2204-kube-${version} -c id -f value) \
-         --external-network public \
+        --image $(openstack image show ${IMAGE_NAME} -c id -f value) \
+        --external-network public \
          --dns-nameserver 8.8.8.8 \
          --master-lb-enabled \
          --master-flavor m1.medium \
@@ -29,7 +31,7 @@ steps to be able to test and develop the project.
          --docker-storage-driver overlay2 \
          --coe kubernetes \
          --label kube_tag=${version} \
-         k8s-${version};
+       d   k8s-${version};
    done;
    popd
    ```

diff --git a/docs/user/getting-started.md b/docs/user/getting-started.md
@@ -47,11 +47,13 @@ dashboard, Terraform, Ansible or the Magnum API directly.
     them using the OpenStack CLI:
 
     ```bash
-    for version in v1.23.17 v1.24.15 v1.25.11 v1.26.6 v1.27.3; do \
-      curl -LO https://object-storage.public.mtl1.vexxhost.net/swift/v1/a91f106f55e64246babde7402c21b87a/magnum-capi/ubuntu-2204-kube-${version}.qcow2; \
-      openstack image create ubuntu-2204-kube-${version} --disk-format=qcow2 --container-format=bare --property os_distro=ubuntu --file=ubuntu-2204-kube-${version}.qcow2; \
+    export OS_DISTRO=ubuntu # you can change this to "flatcar" if you want to use Flatcar
+    for version in v1.24.16 v1.25.12 v1.26.7 v1.27.4; do \
+      [[ "${OS_DISTRO}" == "ubuntu" ]] && IMAGE_NAME="ubuntu-2204-kube-${version}" || IMAGE_NAME="flatcar-kube-${version}"; \
+      curl -LO https://object-storage.public.mtl1.vexxhost.net/swift/v1/a91f106f55e64246babde7402c21b87a/magnum-capi/${IMAGE_NAME}.qcow2; \
+      openstack image create ${IMAGE_NAME} --disk-format=qcow2 --container-format=bare --property os_distro=${OS_DISTRO} --file=${IMAGE_NAME}.qcow2; \
       openstack coe cluster template create \
-          --image $(openstack image show ubuntu-2204-kube-${version} -c id -f value) \
+          --image $(openstack image show ${IMAGE_NAME} -c id -f value) \
           --external-network public \
           --dns-nameserver 8.8.8.8 \
           --master-lb-enabled \
@@ -152,7 +154,7 @@ dashboard, Terraform, Ansible or the Magnum API directly.
     The next step is managing the network configuration of the cluster.  The
     required fields are:
 
-    * **Enable Load Balancer for Master Nodes**  
+    * **Enable Load Balancer for Master Nodes**
       This is required to be **enabled** for the Cluster API driver for Magnum
       to work properly.
 

diff --git a/hack/run-functional-tests.sh b/hack/run-functional-tests.sh
@@ -20,11 +20,15 @@
 
 source /opt/stack/openrc
 
+OS_DISTRO=${OS_DISTRO:-ubuntu}
 NODE_COUNT=${NODE_COUNT:-2}
 SONOBUOY_VERSION=${SONOBUOY_VERSION:-0.56.16}
 SONOBUOY_ARCH=${SONOBUOY_ARCH:-amd64}
 DNS_NAMESERVER=${DNS_NAMESERVER:-1.1.1.1}
 
+# Determine image name
+[[ "${OS_DISTRO}" == "ubuntu" ]] && IMAGE_NAME="ubuntu-2204-kube-${KUBE_TAG}" || IMAGE_NAME="flatcar-kube-${KUBE_TAG}";
+
 # If running inside GitHub Actions, use Azure's "168.63.129.16" for DNS
 if [[ -n "${GITHUB_ACTIONS}" ]]; then
   DNS_NAMESERVER=168.63.129.16
@@ -33,15 +37,14 @@ fi
 # If `BUILD_NEW_IMAGE` is true, then we use the provided artifact, otherwise
 # we download the latest promoted image.
 if [[ "${BUILD_NEW_IMAGE}" != "true" ]]; then
-  curl -LO https://object-storage.public.mtl1.vexxhost.net/swift/v1/a91f106f55e64246babde7402c21b87a/magnum-capi/ubuntu-2204-kube-${KUBE_TAG}.qcow2
+  curl -LO https://object-storage.public.mtl1.vexxhost.net/swift/v1/a91f106f55e64246babde7402c21b87a/magnum-capi/${IMAGE_NAME}.qcow2
 fi
 
 # Upload image to Glance
-IMAGE_NAME=ubuntu-2204-kube-${KUBE_TAG}
 openstack image create \
   --disk-format=qcow2 \
   --container-format=bare \
-  --property os_distro=ubuntu \
+  --property os_distro=${OS_DISTRO} \
   --file=${IMAGE_NAME}.qcow2 \
   ${IMAGE_NAME}
 

diff --git a/hack/stack.sh b/hack/stack.sh
@@ -131,6 +131,7 @@ sudo chmod +x /usr/local/bin/clusterctl
 
 # Initialize the `clusterctl` CLI
 export EXP_CLUSTER_RESOURCE_SET=true
+export EXP_KUBEADM_BOOTSTRAP_FORMAT_IGNITION=true #Used by the kubeadm bootstrap provider
 export CLUSTER_TOPOLOGY=true
 clusterctl init \
   --core cluster-api:${CAPI_VERSION} \

diff --git a/magnum_cluster_api/cmd/image_builder.py b/magnum_cluster_api/cmd/image_builder.py
@@ -38,12 +38,14 @@ def validate_version(_, __, value):
 
 
 @click.command()
+@click.pass_context
 @click.option(
     "--operating-system",
     show_default=True,
     default="ubuntu-2204",
-    type=click.Choice(["ubuntu-2004", "ubuntu-2204"]),
+    type=click.Choice(["ubuntu-2004", "ubuntu-2204", "flatcar"]),
     help="Operating system to build image for",
+    prompt="Operating system to build image for",
 )
 @click.option(
     "--version",
@@ -55,17 +57,25 @@ def validate_version(_, __, value):
 @click.option(
     "--image-builder-version",
     show_default=True,
-    default="bab3d4d",
+    default="v0.1.19",
     help="Image builder tag (or commit) to use for building image",
 )
-def main(operating_system, version, image_builder_version):
+def main(ctx: click.Context, operating_system, version, image_builder_version):
     ib_path = f"/tmp/image-builder-{image_builder_version}"
-    output = f"{operating_system}-kube-{version}"
+    output_path = f"{ib_path}/images/capi/output"
 
-    target = f"{ib_path}/images/capi/output/{output}/{output}"
-    if os.path.exists(target):
-        print(f"Image already exists: {target}")
-        return
+    # Scan the entire output directory recursively and stop if there is any file
+    # at all
+    for root, dirs, files in os.walk(output_path):
+        if files:
+            message = (
+                "There are files in the output directory which will cause the build to fail. "
+                "Please remove them before continuing.\n"
+            )
+            for file in files:
+                message += f"- {root}/{file}\n"
+
+            ctx.fail(message)
 
     click.echo("- Install QEMU packages")
     subprocess.run(
@@ -111,9 +121,11 @@ def main(operating_system, version, image_builder_version):
     click.echo("- Create customization file")
     kubernetes_series = ".".join(version.split(".")[0:2])
     customization = {
-        "kubernetes_deb_version": f"{version.replace('v', '')}-00",
+        "kubernetes_deb_version": f"{version.replace('v', '')}-1.1",
         "kubernetes_semver": f"{version}",
         "kubernetes_series": f"{kubernetes_series}",
+        # https://github.com/flatcar/Flatcar/issues/823
+        "ansible_user_vars": "oem_id=openstack",
     }
 
     # NOTE(mnaser): We use the latest tested daily ISO for Ubuntu 22.04 in order
@@ -178,5 +190,18 @@ def main(operating_system, version, image_builder_version):
             },
         )
 
+    # Try and detect the target image path since it can be different depending
+    # on the operating system, so we scan the output directory for the file
+    # that matches the pattern.
+    target = None
+    for root, dirs, files in os.walk(output_path):
+        if len(files) > 1:
+            ctx.fail(f"Unexpected number of files in {root}")
+        if files:
+            target = f"{root}/{files[0]}"
+
+    if target is None:
+        ctx.fail("Unable to detect target image")
+
     # Copy from the target to the current working directory
     os.rename(target, f"{operating_system}-kube-{version}.qcow2")
diff --git a/magnum_cluster_api/driver.py b/magnum_cluster_api/driver.py
@@ -386,3 +386,11 @@ def provides(self):
         return [
             {"server_type": "vm", "os": "ubuntu-focal", "coe": "kubernetes"},
         ]
+
+
+class FlatcarDriver(BaseDriver):
+    @property
+    def provides(self):
+        return [
+            {"server_type": "vm", "os": "flatcar", "coe": "kubernetes"},
+        ]