Flat car systemd is crashing after OS update version from Flatcar-stable-3760.2.0-hvm

[shankar-vng]

Description

Systemd along with all process targets seems to crash in AWS X86 flat car ami Flatcar-stable-3815.2.0-hvm until we force update back to version 3760.2.0

while looking at the comments on the issue, it seems the issue could be with the bootstrap config which i'm enclosing below

At this point, we are not sure whether the issue is originating from OS bootstrap config or from the ami itself

Impact

None of the systemd targets (including Containderd & kubelet) are running which takes down all of K8s infrastructure on the node

Environment and steps to reproduce

@t-lo I'm enclosing the boot Strap config which gets loaded as part of flat car OS bootstrapping which may be conflicting with flatcar OS ignition bootstrap process. Appreciate if you could point us to any conflicts that might be interfering with the systemd bootstrapping

osp-flatcar.yml.zip

Including Journal logs

Journal_log.zip

Expected behavior

Systemd units should get bootstrapped instead of crashing

Additional information

[tormath1]

As mentioned on Matrix, it seems to be related to the line 681 of the provided bootstrap configuration:

      - content:
          inline:
            data: |
              [Service]
              EnvironmentFile=/run/metadata/torcx
              Environment=CONTAINERD_CONFIG=/etc/containerd/config.toml
              ExecStart=
              ExecStart=/usr/bin/env PATH=${TORCX_BINDIR}:${PATH} ${TORCX_BINDIR}/containerd --config ${CONTAINERD_CONFIG}
        path: /etc/systemd/system/containerd.service.d/10-custom.conf
        permissions: 644

In the last Stable, Torcx has been removed in favor of systemd-sysext https://www.flatcar.org/releases#release-3815.2.0:

torcx was replaced by systemd-sysext in the OS image

You might be interested to update your bootstrap config (i.e remove the EnvironmentFile=/run/metadata/torcxand drop the ExecStart overriding)

More on that here: https://www.flatcar.org/blog/2023/12/extending-flatcar-say-goodbye-to-torcx-and-hello-to-systemd-sysext/

[shankar-vng]

Closing this issue, since this is custom bootstrap config issue.. Thank a ton @tormath1

[tormath1]

@shankar-vng this change has been in Beta since Jan 18, 2024 - we consider Beta enough robust to deploy a few Beta nodes in your workloads. This is a great way to contribute to Flatcar: helping team and users by identifying any issues before they land on Stable. It's valuable for both side.

[shankar-vng]

@tormath1 will do 🙇 & thanks for the tip!.. We will launch beta release in our test cluster before they are propagated to other environment