feat: Support Flatcar OS and custom NTP servers #44

okozachenko1203 · 2022-11-17T14:24:14Z

MAGNUM-2

mnaser · 2022-11-17T15:45:08Z

by doing this, we're fully committing to Flatcar and not running anything on Ubuntu.

I think the cleaner thing is that we should create a variable called operatingSystem in the ClusterClass, and then in there, it can be an enum of either ubuntu or flatcar, by default it will be set to ubuntu.

And then, we should create a patch, conditional on operatingSystem being equal to flatcar to make all the changes that are needed for Flatcar, that way we can support both.

Then, we will have to update the driver to detect os_distro and use the appropriate model. Also, I think that means we'll also need to update the image builder script to allow building flatcar images and publish them too.

okozachenko1203 · 2022-11-18T07:33:41Z

flatcar image here for testing

https://object-storage.public.mtl1.vexxhost.net/swift/v1/7d9c8dc3734742b8a329c22b5c78bab3/magnum-capi-driver/flatcar-3374.2.0-v1.25.3

mnaser

good progress, let's make sure to switch to patches on the ClusterClass as mentioned

Also, let's update title to be feat since this is a feature addition.

magnum_cluster_api/resources.py

mnaser

small nits, otherwise is this ready to land @okozachenko1203 -- shall we drop the WIP?

magnum_cluster_api/cmd/image_builder.py

okozachenko1203 · 2022-11-23T12:14:28Z

small nits, otherwise is this ready to land @okozachenko1203 -- shall we drop the WIP?

@mnaser
All patches are rendered exactly but now I am facing kcp unavailable issue. Seems kubernetes-sigs/image-builder#939 (comment) is not working. Btw, I need to dig more by jump into the machine. But key auth is not working with flatcar image. Created an issue to the upstream.
Flatcar images built by image-builder does not support key auth · Issue #1018 · kubernetes-sigs/image-builder

okozachenko1203 · 2022-11-25T11:06:07Z

small nits, otherwise is this ready to land @okozachenko1203 -- shall we drop the WIP?

@mnaser All patches are rendered exactly but now I am facing kcp unavailable issue. Seems kubernetes-sigs/image-builder#939 (comment) is not working. Btw, I need to dig more by jump into the machine. But key auth is not working with flatcar image. Created an issue to the upstream. Flatcar images built by image-builder does not support key auth · Issue #1018 · kubernetes-sigs/image-builder

I close this kubernetes-sigs/image-builder#1018 and land openstack provider specific override in this driver.

mnaser · 2022-11-25T15:16:55Z

magnum_cluster_api/cmd/image_builder.py

+        # https://github.com/flatcar/Flatcar/issues/823
+        "ansible_user_vars": "oem_id=openstack",


nice catch here.

There is a lot of hacks around this because we're using an ISO image, @okozachenko1203 is going to contribute to image-builder an openstack image so that it uses OpenStack to deploy and publish directly to Glance.

Once we have that in place, we should switch the image builder to use that instead. It'll drive down the dependencies and we can remove this after.

mnaser · 2022-11-25T15:45:17Z

magnum_cluster_api/cmd/image_builder.py

+        # https://github.com/flatcar/Flatcar/issues/823
+        "ansible_user_vars": "oem_id=openstack",


There is a lot of hacks around this because we're using an ISO image, @okozachenko1203 is going to contribute to image-builder an openstack image so that it uses OpenStack to deploy and publish directly to Glance.

Once we have that in place, we should switch the image builder to use that instead. It'll drive down the dependencies and we can remove this after.

mnaser · 2022-11-25T15:45:48Z

magnum_cluster_api/resources.py

+                                                          units:
+                                                          - name: coreos-metadata.service
+                                                            dropins:
+                                                            - name: 20-clct-provider-override.conf
+                                                              contents: |
+                                                                [Service]
+                                                                # Set Openstack as coreos-metadata provider
+                                                                Environment=COREOS_METADATA_OPT_PROVIDER=--provider=openstack-metadata
+                                                          - name: coreos-metadata-sshkeys@.service
+                                                            enabled: true
+                                                            dropins:
+                                                            - name: 20-clct-provider-override.conf
+                                                              contents: |
+                                                                [Service]
+                                                                # Set Openstack as coreos-metadata provider
+                                                                Environment=COREOS_METADATA_OPT_PROVIDER=--provider=openstack-metadata


we can remove this after moving to openstack built images.

mnaser · 2022-11-25T15:46:12Z

magnum_cluster_api/resources.py

+                                                          - name: coreos-metadata.service
+                                                            dropins:
+                                                            - name: 20-clct-provider-override.conf
+                                                              contents: |
+                                                                [Service]
+                                                                Environment=COREOS_METADATA_OPT_PROVIDER=--provider=openstack-metadata
+                                                          - name: coreos-metadata-sshkeys@.service
+                                                            enabled: true
+                                                            dropins:
+                                                            - name: 20-clct-provider-override.conf
+                                                              contents: |
+                                                                [Service]
+                                                                Environment=COREOS_METADATA_OPT_PROVIDER=--provider=openstack-metadata


we can remove this after moving to openstack built images

mnaser · 2022-11-25T15:47:51Z

magnum_cluster_api/resources.py

+                                                          - name: kubeadm.service
+                                                            dropins:
+                                                            - name: 10-flatcar.conf
+                                                              contents: |
+                                                                [Unit]
+                                                                # kubeadm must run after coreos-metadata populated /run/metadata directory.
+                                                                Requires=coreos-metadata.service
+                                                                After=coreos-metadata.service
+                                                                [Service]
+                                                                # Ensure kubeadm service has access to kubeadm binary in /opt/bin on Flatcar.
+                                                                Environment=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/bin
+                                                                # To make metadata environment variables available for pre-kubeadm commands.
+                                                                EnvironmentFile=/run/metadata/*
+                                                        """  # noqa: E501


this feels like it should live inside image-builder, so I think as a separate PR we should fix this?

The PATH modifying part should not be needed anymore since kubernetes-sigs/image-builder@f4864de, but I think dependence on coreos-metadata.service is platform-specific, so I don't think image-builder is a good place to put this.

mnaser · 2022-11-25T15:48:06Z

magnum_cluster_api/resources.py

+                                                          - name: kubeadm.service
+                                                            dropins:
+                                                            - name: 10-flatcar.conf
+                                                              contents: |
+                                                                [Unit]
+                                                                # kubeadm must run after coreos-metadata populated /run/metadata directory.
+                                                                Requires=coreos-metadata.service
+                                                                After=coreos-metadata.service
+                                                                [Service]
+                                                                # In Flatcar /usr is immutable, so image-builder puts the binaries in /opt/bin instead.
+                                                                # Ensure kubeadm service has access to kubeadm binary in /opt/bin on Flatcar.
+                                                                Environment=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/bin
+                                                                # To make metadata environment variables available for pre-kubeadm commands.
+                                                                EnvironmentFile=/run/metadata/*
+                                                        """  # noqa: E501


same for this

tormath1 · 2023-05-30T07:27:24Z

Hello, Flatcar is now supported by CAPO - did you give a try with Magnum recently ? @mnaser I see you're in the #cluster-api-openstack Slack channel, do not hesitate to ping me if you need a hand on this topic. :)

mnaser · 2023-05-30T13:45:54Z

I think perhaps maybe if there's some documentation on building images and how clusters are defined for a flatcar system.. that would be helpful

tormath1 · 2023-05-30T17:40:10Z

@mnaser there is this documentation on the CAPO handbook: https://cluster-api-openstack.sigs.k8s.io/clusteropenstack/configuration.html#ignition-based-images to build and use Flatcar images (and as you know there is this issue: kubernetes-sigs/cluster-api-provider-openstack#1502)

Mazorius · 2023-07-07T09:49:35Z

I am waiting for the NTP fix as on autoscaling it is not possible to always change the internal NTP server.

Can the NTP fix be extracted so it can be merged @okozachenko1203 ?

mnaser · 2023-10-13T14:21:57Z

@Mazorius you shouldn't have any ntp issues with the latest images as you'll be getting the NTP server from DHCP, and the Flatcar part has been merged so I'm closing thsi

okozachenko1203 added 3 commits November 18, 2022 01:22

fix: use flatcar

befc19b

Add preKubeadmCommands for node name replacement

fc3a172

Fix mdlint

dbc43b2

okozachenko1203 requested a review from mnaser November 17, 2022 15:14

Build flatcar image

fafcaac

okozachenko1203 changed the title ~~fix: use flatcar~~ [WIP] fix: use flatcar Nov 18, 2022

okozachenko1203 added 4 commits November 18, 2022 23:45

Add flatcardriver

51b67e2

Add flatcarDriver in poetry plugin list

9a4f259

Fix prekubeadmcommands format

65bdd68

Enable required feature gate and update ignition

62bd1f0

mnaser requested changes Nov 18, 2022

View reviewed changes

magnum_cluster_api/resources.py Outdated Show resolved Hide resolved

magnum_cluster_api/resources.py Outdated Show resolved Hide resolved

okozachenko1203 added 6 commits November 22, 2022 21:51

Support os switch

f38ad5d

Fix lint

facabee

Fix lint

11fe3c1

Fix lint

e18d24a

Fix fake8 lint

e2b6dea

Fix clusterclas patches

18acf68

okozachenko1203 changed the title ~~[WIP] fix: use flatcar~~ [WIP] feat: use flatcar Nov 22, 2022

Fix lint

2408d37

mnaser requested changes Nov 22, 2022

View reviewed changes

magnum_cluster_api/cmd/image_builder.py Show resolved Hide resolved

magnum_cluster_api/cmd/image_builder.py Outdated Show resolved Hide resolved

Remove default os in image build cmd

a04544e

okozachenko1203 added 2 commits November 24, 2022 19:59

Update ignition

1aa707d

Fix ignition ignoring error

a1dc6b9

okozachenko1203 changed the title ~~[WIP] feat: use flatcar~~ feat: Support Flatcar OS Nov 24, 2022

Fix lint

ded7eb4

okozachenko1203 requested a review from mnaser November 24, 2022 18:43

Add comment about PATH overwrite in flatcar

7c46bb5

okozachenko1203 added 3 commits November 25, 2022 06:04

Update TODO comment

0429d08

Land clct openstack provider override

e46788a

Remove comments

9e58cb0

Support ntp servers

35fbe03

okozachenko1203 changed the title ~~feat: Support Flatcar OS~~ feat: Support Flatcar OS and custom NTP servers Nov 25, 2022

okozachenko1203 added 2 commits November 26, 2022 00:11

Fix syntax

4e18f36

Fix condition for ntpserver length

5e0e344

mnaser reviewed Nov 25, 2022

View reviewed changes

mnaser requested changes Nov 25, 2022

View reviewed changes

t-lo mentioned this pull request Jan 4, 2023

Magnum support flatcar/Flatcar#922

Closed

1 task

stefan-chivu mentioned this pull request Sep 28, 2023

feat: Support Flatcar OS #225

Merged

mnaser closed this Oct 13, 2023

mnaser deleted the use-flatcar branch October 13, 2023 14:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Support Flatcar OS and custom NTP servers #44

feat: Support Flatcar OS and custom NTP servers #44

okozachenko1203 commented Nov 17, 2022 •

edited

Loading

mnaser commented Nov 17, 2022

okozachenko1203 commented Nov 18, 2022 •

edited

Loading

mnaser left a comment

mnaser left a comment

okozachenko1203 commented Nov 23, 2022 •

edited

Loading

okozachenko1203 commented Nov 25, 2022

mnaser Nov 25, 2022

mnaser Nov 25, 2022

mnaser Nov 25, 2022

mnaser Nov 25, 2022

mnaser Nov 25, 2022

mnaser Nov 25, 2022

invidian Jan 2, 2023

mnaser Nov 25, 2022

tormath1 commented May 30, 2023

mnaser commented May 30, 2023

tormath1 commented May 30, 2023

Mazorius commented Jul 7, 2023

mnaser commented Oct 13, 2023

		# https://github.com/flatcar/Flatcar/issues/823
		"ansible_user_vars": "oem_id=openstack",

feat: Support Flatcar OS and custom NTP servers #44

feat: Support Flatcar OS and custom NTP servers #44

Conversation

okozachenko1203 commented Nov 17, 2022 • edited Loading

mnaser commented Nov 17, 2022

okozachenko1203 commented Nov 18, 2022 • edited Loading

mnaser left a comment

Choose a reason for hiding this comment

mnaser left a comment

Choose a reason for hiding this comment

okozachenko1203 commented Nov 23, 2022 • edited Loading

okozachenko1203 commented Nov 25, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tormath1 commented May 30, 2023

mnaser commented May 30, 2023

tormath1 commented May 30, 2023

Mazorius commented Jul 7, 2023

mnaser commented Oct 13, 2023

okozachenko1203 commented Nov 17, 2022 •

edited

Loading

okozachenko1203 commented Nov 18, 2022 •

edited

Loading

okozachenko1203 commented Nov 23, 2022 •

edited

Loading