[Bug]: Process /usr/local/bin/k3s could not be executed while applying terraform script

[lukasholzer]

Description

When applying the config it fails with the following error that k3s could not be executed.

journalctl -xeu k3s.service
░░ Subject: Process /usr/local/bin/k3s could not be executed
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ The process /usr/local/bin/k3s could not be executed and failed.
░░
░░ The error number returned by this process is ERRNO.
Jan 03 09:21:01 k8s-stg-control-plane-yhs (k3s)[4055]: k3s.service: Failed at step EXEC spawning /usr/local/bin/k3s: Permission>
░░ Subject: Process /usr/local/bin/k3s could not be executed
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ The process /usr/local/bin/k3s could not be executed and failed.
░░
░░ The error number returned by this process is ERRNO.
Jan 03 09:21:01 k8s-stg-control-plane-yhs systemd[1]: k3s.service: Main process exited, code=exited, status=203/EXEC
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ An ExecStart= process belonging to unit k3s.service has exited.
░░
░░ The process' exit code is 'exited' and its exit status is 203.
Jan 03 09:21:01 k8s-stg-control-plane-yhs systemd[1]: k3s.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ The unit k3s.service has entered the 'failed' state with result 'exit-code'.
Jan 03 09:21:01 k8s-stg-control-plane-yhs systemd[1]: Failed to start Lightweight Kubernetes.
░░ Subject: A start job for unit k3s.service has failed
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ A start job for unit k3s.service has finished with a failure.
░░
░░ The job identifier is 18522 and the job result is failed.

Kube.tf file

module "kube-hetzner" {
  providers = {
    hcloud = hcloud
  }
  hcloud_token = data.doppler_secrets.this.map.hcloud_token
  
  source = "github.com/kube-hetzner/terraform-hcloud-kube-hetzner"
  
  ssh_public_key = file("~/.ssh/id_ed25519.pub")
  ssh_private_key = file("~/.ssh/id_ed25519")
  
  network_region = "eu-central" # change to `us-east` if location is ash

  control_plane_nodepools = [
    {
      name        = "control-plane",
      server_type = "cax11",
      location    = "nbg1",
      labels      = [],
      taints      = [],
      count       = 1
    },
  ]

  agent_nodepools = [
    {
      name        = "agent-small",
      server_type = "cax11",
      location    = "nbg1",
      labels      = [],
      taints      = [],
      count       = 2
    },
  ]

  load_balancer_type     = "lb11"
  load_balancer_location = "fsn1"

  automatically_upgrade_k3s = false
  automatically_upgrade_os = false

  cluster_name = "k8s-stg"

  extra_firewall_rules = [
    {
      description     = "Node Mailer SMTP inbounds"
      direction       = "in"
      protocol        = "tcp"
      port            = "587"
      source_ips      = ["0.0.0.0/0", "::/0"]
      destination_ips = [] # Won't be used for this rule
    },
    {
      description     = "Node Mailer SMTP outbounds"
      direction       = "out"
      protocol        = "tcp"
      port            = "587"
      source_ips      = [] # Won't be used for this rule
      destination_ips = ["0.0.0.0/0", "::/0"]
    },
  ]

  dns_servers = [
    "1.1.1.1",
    "8.8.8.8",
    "2606:4700:4700::1111",
  ]
}

output "kubeconfig" {
  value     = module.kube-hetzner.kubeconfig
  sensitive = true
}

Screenshots

module.kube-hetzner.random_password.k3s_token: Creating...
module.kube-hetzner.random_password.rancher_bootstrap[0]: Creating...
module.kube-hetzner.hcloud_placement_group.control_plane[0]: Creating...
module.kube-hetzner.hcloud_ssh_key.k3s[0]: Creating...
module.kube-hetzner.hcloud_placement_group.agent[0]: Creating...
module.kube-hetzner.hcloud_network.k3s[0]: Creating...
module.kube-hetzner.hcloud_load_balancer.cluster[0]: Creating...
module.kube-hetzner.hcloud_firewall.k3s: Creating...
module.kube-hetzner.random_password.k3s_token: Creation complete after 0s [id=none]
module.kube-hetzner.random_password.rancher_bootstrap[0]: Creation complete after 0s [id=none]
module.kube-hetzner.local_file.kustomization_backup[0]: Creating...
module.kube-hetzner.local_file.kustomization_backup[0]: Creation complete after 0s [id=05d497b44bb01d42ebcadd0a6b316ce3792e47c8]
module.kube-hetzner.hcloud_network.k3s[0]: Creation complete after 0s [id=3732117]
module.kube-hetzner.data.hcloud_network.k3s: Reading...
module.kube-hetzner.data.hcloud_network.k3s: Read complete after 0s [name=k8s-stg]
module.kube-hetzner.hcloud_network_subnet.agent[0]: Creating...
module.kube-hetzner.hcloud_network_subnet.control_plane[0]: Creating...
module.kube-hetzner.hcloud_placement_group.control_plane[0]: Creation complete after 0s [id=292709]
module.kube-hetzner.hcloud_placement_group.agent[0]: Creation complete after 0s [id=292708]
module.kube-hetzner.hcloud_ssh_key.k3s[0]: Creation complete after 1s [id=18217105]
module.kube-hetzner.hcloud_network_subnet.control_plane[0]: Creation complete after 1s [id=3732117-10.255.0.0/16]
module.kube-hetzner.module.control_planes["0-0-control-plane"].random_string.identity_file: Creating...
module.kube-hetzner.module.control_planes["0-0-control-plane"].random_string.server: Creating...
module.kube-hetzner.module.control_planes["0-0-control-plane"].random_string.identity_file: Creation complete after 0s [id=2ry70oik4vcbbspr68x5]
module.kube-hetzner.module.control_planes["0-0-control-plane"].random_string.server: Creation complete after 0s [id=htu]
module.kube-hetzner.module.control_planes["0-0-control-plane"].data.cloudinit_config.config: Reading...
module.kube-hetzner.module.control_planes["0-0-control-plane"].data.cloudinit_config.config: Read complete after 0s [id=3385786107]
module.kube-hetzner.hcloud_network_subnet.agent[0]: Creation complete after 1s [id=3732117-10.0.0.0/16]
module.kube-hetzner.module.agents["0-1-agent-small"].random_string.identity_file: Creating...
module.kube-hetzner.module.agents["0-0-agent-small"].random_string.server: Creating...
module.kube-hetzner.module.agents["0-0-agent-small"].random_string.identity_file: Creating...
module.kube-hetzner.module.agents["0-1-agent-small"].random_string.server: Creating...
module.kube-hetzner.module.agents["0-1-agent-small"].random_string.server: Creation complete after 0s [id=vhg]
module.kube-hetzner.module.agents["0-1-agent-small"].random_string.identity_file: Creation complete after 0s [id=2py4m6z9gmj759k3c61p]
module.kube-hetzner.module.agents["0-0-agent-small"].random_string.server: Creation complete after 0s [id=gzu]
module.kube-hetzner.module.agents["0-0-agent-small"].random_string.identity_file: Creation complete after 0s [id=tg8nyxmfd8q9ic3twv9r]
module.kube-hetzner.module.agents["0-0-agent-small"].data.cloudinit_config.config: Reading...
module.kube-hetzner.module.agents["0-1-agent-small"].data.cloudinit_config.config: Reading...
module.kube-hetzner.module.agents["0-1-agent-small"].data.cloudinit_config.config: Read complete after 0s [id=3044855441]
module.kube-hetzner.module.agents["0-0-agent-small"].data.cloudinit_config.config: Read complete after 0s [id=187302299]
module.kube-hetzner.hcloud_firewall.k3s: Creation complete after 1s [id=1188674]
module.kube-hetzner.module.agents["0-0-agent-small"].hcloud_server.server: Creating...
module.kube-hetzner.module.control_planes["0-0-control-plane"].hcloud_server.server: Creating...
module.kube-hetzner.module.agents["0-1-agent-small"].hcloud_server.server: Creating...
module.kube-hetzner.hcloud_load_balancer.cluster[0]: Creation complete after 3s [id=1621808]
module.kube-hetzner.module.control_planes["0-0-control-plane"].hcloud_server.server: Still creating... [10s elapsed]
module.kube-hetzner.module.agents["0-0-agent-small"].hcloud_server.server: Still creating... [10s elapsed]
module.kube-hetzner.module.agents["0-1-agent-small"].hcloud_server.server: Still creating... [10s elapsed]
module.kube-hetzner.module.agents["0-1-agent-small"].hcloud_server.server: Still creating... [20s elapsed]
module.kube-hetzner.module.control_planes["0-0-control-plane"].hcloud_server.server: Still creating... [20s elapsed]
module.kube-hetzner.module.agents["0-0-agent-small"].hcloud_server.server: Still creating... [20s elapsed]
module.kube-hetzner.module.agents["0-0-agent-small"].hcloud_server.server: Still creating... [30s elapsed]
module.kube-hetzner.module.agents["0-1-agent-small"].hcloud_server.server: Still creating... [30s elapsed]
module.kube-hetzner.module.control_planes["0-0-control-plane"].hcloud_server.server: Still creating... [30s elapsed]
module.kube-hetzner.module.control_planes["0-0-control-plane"].hcloud_server.server: Still creating... [40s elapsed]
module.kube-hetzner.module.agents["0-0-agent-small"].hcloud_server.server: Still creating... [40s elapsed]
module.kube-hetzner.module.agents["0-1-agent-small"].hcloud_server.server: Still creating... [40s elapsed]
module.kube-hetzner.module.agents["0-0-agent-small"].hcloud_server.server: Still creating... [50s elapsed]
module.kube-hetzner.module.agents["0-1-agent-small"].hcloud_server.server: Still creating... [50s elapsed]
module.kube-hetzner.module.control_planes["0-0-control-plane"].hcloud_server.server: Still creating... [50s elapsed]
module.kube-hetzner.module.control_planes["0-0-control-plane"].hcloud_server.server: Still creating... [1m0s elapsed]
module.kube-hetzner.module.agents["0-1-agent-small"].hcloud_server.server: Still creating... [1m0s elapsed]
module.kube-hetzner.module.agents["0-0-agent-small"].hcloud_server.server: Still creating... [1m0s elapsed]
module.kube-hetzner.module.agents["0-0-agent-small"].hcloud_server.server: Still creating... [1m10s elapsed]
module.kube-hetzner.module.control_planes["0-0-control-plane"].hcloud_server.server: Still creating... [1m10s elapsed]
module.kube-hetzner.module.agents["0-1-agent-small"].hcloud_server.server: Still creating... [1m10s elapsed]
module.kube-hetzner.module.agents["0-1-agent-small"].hcloud_server.server: Still creating... [1m20s elapsed]
module.kube-hetzner.module.agents["0-0-agent-small"].hcloud_server.server: Still creating... [1m20s elapsed]
module.kube-hetzner.module.control_planes["0-0-control-plane"].hcloud_server.server: Still creating... [1m20s elapsed]
module.kube-hetzner.module.agents["0-1-agent-small"].hcloud_server.server: Still creating... [1m30s elapsed]
module.kube-hetzner.module.control_planes["0-0-control-plane"].hcloud_server.server: Still creating... [1m30s elapsed]
module.kube-hetzner.module.agents["0-0-agent-small"].hcloud_server.server: Still creating... [1m30s elapsed]
module.kube-hetzner.module.agents["0-1-agent-small"].hcloud_server.server: Still creating... [1m40s elapsed]
module.kube-hetzner.module.agents["0-0-agent-small"].hcloud_server.server: Still creating... [1m40s elapsed]
module.kube-hetzner.module.control_planes["0-0-control-plane"].hcloud_server.server: Still creating... [1m40s elapsed]
module.kube-hetzner.module.control_planes["0-0-control-plane"].hcloud_server.server: Still creating... [1m50s elapsed]
module.kube-hetzner.module.agents["0-1-agent-small"].hcloud_server.server: Still creating... [1m50s elapsed]
module.kube-hetzner.module.agents["0-0-agent-small"].hcloud_server.server: Still creating... [1m50s elapsed]
module.kube-hetzner.module.agents["0-0-agent-small"].hcloud_server.server: Still creating... [2m0s elapsed]
module.kube-hetzner.module.agents["0-1-agent-small"].hcloud_server.server: Still creating... [2m0s elapsed]
module.kube-hetzner.module.control_planes["0-0-control-plane"].hcloud_server.server: Still creating... [2m0s elapsed]
module.kube-hetzner.module.control_planes["0-0-control-plane"].hcloud_server.server: Still creating... [2m10s elapsed]
module.kube-hetzner.module.agents["0-0-agent-small"].hcloud_server.server: Still creating... [2m10s elapsed]
module.kube-hetzner.module.agents["0-1-agent-small"].hcloud_server.server: Still creating... [2m10s elapsed]
module.kube-hetzner.module.agents["0-0-agent-small"].hcloud_server.server: Provisioning with 'local-exec'...
module.kube-hetzner.module.agents["0-0-agent-small"].hcloud_server.server (local-exec): (output suppressed due to sensitive value in config)
module.kube-hetzner.module.agents["0-0-agent-small"].hcloud_server.server: Provisioning with 'local-exec'...
module.kube-hetzner.module.agents["0-0-agent-small"].hcloud_server.server (local-exec): Executing: ["/bin/sh" "-c" "until ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes -o PubkeyAuthentication=yes -i /tmp/tg8nyxmfd8q9ic3twv9r -o ConnectTimeout=2 -p 22 root@128.140.75.14 true 2> /dev/null\ndo\n  echo \"Waiting for MicroOS to become available...\"\n  sleep 3\ndone\n"]
module.kube-hetzner.module.agents["0-0-agent-small"].hcloud_server.server: Provisioning with 'local-exec'...
module.kube-hetzner.module.agents["0-0-agent-small"].hcloud_server.server (local-exec): Executing: ["/bin/sh" "-c" "rm /tmp/tg8nyxmfd8q9ic3twv9r\n"]
module.kube-hetzner.module.agents["0-0-agent-small"].hcloud_server.server: Provisioning with 'remote-exec'...
module.kube-hetzner.module.agents["0-0-agent-small"].hcloud_server.server (remote-exec): Connecting to remote host via SSH...
module.kube-hetzner.module.agents["0-0-agent-small"].hcloud_server.server (remote-exec):   Host: 128.140.75.14
module.kube-hetzner.module.agents["0-0-agent-small"].hcloud_server.server (remote-exec):   User: root
module.kube-hetzner.module.agents["0-0-agent-small"].hcloud_server.server (remote-exec):   Password: false
module.kube-hetzner.module.agents["0-0-agent-small"].hcloud_server.server (remote-exec):   Private key: true
module.kube-hetzner.module.agents["0-0-agent-small"].hcloud_server.server (remote-exec):   Certificate: false
module.kube-hetzner.module.agents["0-0-agent-small"].hcloud_server.server (remote-exec):   SSH Agent: true
module.kube-hetzner.module.agents["0-0-agent-small"].hcloud_server.server (remote-exec):   Checking Host Key: false
module.kube-hetzner.module.agents["0-0-agent-small"].hcloud_server.server (remote-exec):   Target Platform: unix
module.kube-hetzner.module.agents["0-0-agent-small"].hcloud_server.server (remote-exec): Connected!
module.kube-hetzner.module.agents["0-1-agent-small"].hcloud_server.server: Provisioning with 'local-exec'...
module.kube-hetzner.module.agents["0-1-agent-small"].hcloud_server.server (local-exec): (output suppressed due to sensitive value in config)
module.kube-hetzner.module.agents["0-1-agent-small"].hcloud_server.server: Provisioning with 'local-exec'...
module.kube-hetzner.module.agents["0-1-agent-small"].hcloud_server.server (local-exec): Executing: ["/bin/sh" "-c" "until ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes -o PubkeyAuthentication=yes -i /tmp/2py4m6z9gmj759k3c61p -o ConnectTimeout=2 -p 22 root@49.13.194.11 true 2> /dev/null\ndo\n  echo \"Waiting for MicroOS to become available...\"\n  sleep 3\ndone\n"]
module.kube-hetzner.module.agents["0-0-agent-small"].hcloud_server.server (remote-exec): Automatic OS updates are disabled
module.kube-hetzner.module.agents["0-0-agent-small"].hcloud_server.server (remote-exec): Removed "/etc/systemd/system/timers.target.wants/transactional-update.timer".
module.kube-hetzner.module.agents["0-0-agent-small"].hcloud_server.server: Creation complete after 2m15s [id=41383581]
module.kube-hetzner.module.agents["0-0-agent-small"].hcloud_server_network.server: Creating...
module.kube-hetzner.module.agents["0-0-agent-small"].null_resource.registries: Creating...
module.kube-hetzner.module.agents["0-0-agent-small"].null_resource.zram: Creating...
module.kube-hetzner.module.agents["0-0-agent-small"].null_resource.registries: Provisioning with 'file'...
module.kube-hetzner.module.agents["0-0-agent-small"].null_resource.zram: Provisioning with 'file'...
module.kube-hetzner.module.agents["0-1-agent-small"].hcloud_server.server: Provisioning with 'local-exec'...
module.kube-hetzner.module.agents["0-1-agent-small"].hcloud_server.server (local-exec): Executing: ["/bin/sh" "-c" "rm /tmp/2py4m6z9gmj759k3c61p\n"]
module.kube-hetzner.module.agents["0-1-agent-small"].hcloud_server.server: Provisioning with 'remote-exec'...
module.kube-hetzner.module.agents["0-1-agent-small"].hcloud_server.server (remote-exec): Connecting to remote host via SSH...
module.kube-hetzner.module.agents["0-1-agent-small"].hcloud_server.server (remote-exec):   Host: 49.13.194.11
module.kube-hetzner.module.agents["0-1-agent-small"].hcloud_server.server (remote-exec):   User: root
module.kube-hetzner.module.agents["0-1-agent-small"].hcloud_server.server (remote-exec):   Password: false
module.kube-hetzner.module.agents["0-1-agent-small"].hcloud_server.server (remote-exec):   Private key: true
module.kube-hetzner.module.agents["0-1-agent-small"].hcloud_server.server (remote-exec):   Certificate: false
module.kube-hetzner.module.agents["0-1-agent-small"].hcloud_server.server (remote-exec):   SSH Agent: true
module.kube-hetzner.module.agents["0-1-agent-small"].hcloud_server.server (remote-exec):   Checking Host Key: false
module.kube-hetzner.module.agents["0-1-agent-small"].hcloud_server.server (remote-exec):   Target Platform: unix
module.kube-hetzner.module.agents["0-1-agent-small"].hcloud_server.server (remote-exec): Connecting to remote host via SSH...
module.kube-hetzner.module.agents["0-1-agent-small"].hcloud_server.server (remote-exec):   Host: 49.13.194.11
module.kube-hetzner.module.agents["0-1-agent-small"].hcloud_server.server (remote-exec):   User: root
module.kube-hetzner.module.agents["0-1-agent-small"].hcloud_server.server (remote-exec):   Password: false
module.kube-hetzner.module.agents["0-1-agent-small"].hcloud_server.server (remote-exec):   Private key: true
module.kube-hetzner.module.agents["0-1-agent-small"].hcloud_server.server (remote-exec):   Certificate: false
module.kube-hetzner.module.agents["0-1-agent-small"].hcloud_server.server (remote-exec):   SSH Agent: true
module.kube-hetzner.module.agents["0-1-agent-small"].hcloud_server.server (remote-exec):   Checking Host Key: false
module.kube-hetzner.module.agents["0-1-agent-small"].hcloud_server.server (remote-exec):   Target Platform: unix
module.kube-hetzner.module.control_planes["0-0-control-plane"].hcloud_server.server: Provisioning with 'local-exec'...
module.kube-hetzner.module.control_planes["0-0-control-plane"].hcloud_server.server (local-exec): (output suppressed due to sensitive value in config)
module.kube-hetzner.module.control_planes["0-0-control-plane"].hcloud_server.server: Provisioning with 'local-exec'...
module.kube-hetzner.module.control_planes["0-0-control-plane"].hcloud_server.server (local-exec): Executing: ["/bin/sh" "-c" "until ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes -o PubkeyAuthentication=yes -i /tmp/2ry70oik4vcbbspr68x5 -o ConnectTimeout=2 -p 22 root@195.201.27.53 true 2> /dev/null\ndo\n  echo \"Waiting for MicroOS to become available...\"\n  sleep 3\ndone\n"]
module.kube-hetzner.module.agents["0-0-agent-small"].null_resource.registries: Provisioning with 'remote-exec'...
module.kube-hetzner.module.agents["0-0-agent-small"].null_resource.registries (remote-exec): Connecting to remote host via SSH...
module.kube-hetzner.module.agents["0-0-agent-small"].null_resource.registries (remote-exec):   Host: 128.140.75.14
module.kube-hetzner.module.agents["0-0-agent-small"].null_resource.registries (remote-exec):   User: root
module.kube-hetzner.module.agents["0-0-agent-small"].null_resource.registries (remote-exec):   Password: false
module.kube-hetzner.module.agents["0-0-agent-small"].null_resource.registries (remote-exec):   Private key: true
module.kube-hetzner.module.agents["0-0-agent-small"].null_resource.registries (remote-exec):   Certificate: false
module.kube-hetzner.module.agents["0-0-agent-small"].null_resource.registries (remote-exec):   SSH Agent: true
module.kube-hetzner.module.agents["0-0-agent-small"].null_resource.registries (remote-exec):   Checking Host Key: false
module.kube-hetzner.module.agents["0-0-agent-small"].null_resource.registries (remote-exec):   Target Platform: unix
module.kube-hetzner.module.agents["0-0-agent-small"].null_resource.zram: Provisioning with 'file'...
module.kube-hetzner.module.agents["0-1-agent-small"].hcloud_server.server (remote-exec): Connected!
module.kube-hetzner.module.agents["0-0-agent-small"].null_resource.registries (remote-exec): Connected!
module.kube-hetzner.module.agents["0-0-agent-small"].null_resource.zram: Provisioning with 'file'...
module.kube-hetzner.module.agents["0-1-agent-small"].hcloud_server.server (remote-exec): Automatic OS updates are disabled
module.kube-hetzner.module.agents["0-1-agent-small"].hcloud_server.server (remote-exec): Removed "/etc/systemd/system/timers.target.wants/transactional-update.timer".
module.kube-hetzner.module.agents["0-0-agent-small"].null_resource.registries (remote-exec): No update required to the registries.yaml file
module.kube-hetzner.module.control_planes["0-0-control-plane"].hcloud_server.server: Provisioning with 'local-exec'...
module.kube-hetzner.module.control_planes["0-0-control-plane"].hcloud_server.server (local-exec): Executing: ["/bin/sh" "-c" "rm /tmp/2ry70oik4vcbbspr68x5\n"]
module.kube-hetzner.module.control_planes["0-0-control-plane"].hcloud_server.server: Provisioning with 'remote-exec'...
module.kube-hetzner.module.control_planes["0-0-control-plane"].hcloud_server.server (remote-exec): Connecting to remote host via SSH...
module.kube-hetzner.module.control_planes["0-0-control-plane"].hcloud_server.server (remote-exec):   Host: 195.201.27.53
module.kube-hetzner.module.control_planes["0-0-control-plane"].hcloud_server.server (remote-exec):   User: root
module.kube-hetzner.module.control_planes["0-0-control-plane"].hcloud_server.server (remote-exec):   Password: false
module.kube-hetzner.module.control_planes["0-0-control-plane"].hcloud_server.server (remote-exec):   Private key: true
module.kube-hetzner.module.control_planes["0-0-control-plane"].hcloud_server.server (remote-exec):   Certificate: false
module.kube-hetzner.module.control_planes["0-0-control-plane"].hcloud_server.server (remote-exec):   SSH Agent: true
module.kube-hetzner.module.control_planes["0-0-control-plane"].hcloud_server.server (remote-exec):   Checking Host Key: false
module.kube-hetzner.module.control_planes["0-0-control-plane"].hcloud_server.server (remote-exec):   Target Platform: unix
module.kube-hetzner.module.agents["0-0-agent-small"].null_resource.registries: Creation complete after 3s [id=6836646672541288771]
module.kube-hetzner.module.agents["0-0-agent-small"].null_resource.zram: Provisioning with 'remote-exec'...
module.kube-hetzner.module.agents["0-0-agent-small"].null_resource.zram (remote-exec): Connecting to remote host via SSH...
module.kube-hetzner.module.agents["0-0-agent-small"].null_resource.zram (remote-exec):   Host: 128.140.75.14
module.kube-hetzner.module.agents["0-0-agent-small"].null_resource.zram (remote-exec):   User: root
module.kube-hetzner.module.agents["0-0-agent-small"].null_resource.zram (remote-exec):   Password: false
module.kube-hetzner.module.agents["0-0-agent-small"].null_resource.zram (remote-exec):   Private key: true
module.kube-hetzner.module.agents["0-0-agent-small"].null_resource.zram (remote-exec):   Certificate: false
module.kube-hetzner.module.agents["0-0-agent-small"].null_resource.zram (remote-exec):   SSH Agent: true
module.kube-hetzner.module.agents["0-0-agent-small"].null_resource.zram (remote-exec):   Checking Host Key: false
module.kube-hetzner.module.agents["0-0-agent-small"].null_resource.zram (remote-exec):   Target Platform: unix
module.kube-hetzner.module.agents["0-1-agent-small"].hcloud_server.server: Creation complete after 2m18s [id=41383583]
module.kube-hetzner.module.agents["0-1-agent-small"].null_resource.registries: Creating...
module.kube-hetzner.module.agents["0-1-agent-small"].null_resource.zram: Creating...
module.kube-hetzner.module.agents["0-1-agent-small"].hcloud_server_network.server: Creating...
module.kube-hetzner.module.agents["0-1-agent-small"].null_resource.registries: Provisioning with 'file'...
module.kube-hetzner.module.agents["0-1-agent-small"].null_resource.zram: Provisioning with 'file'...
module.kube-hetzner.module.control_planes["0-0-control-plane"].hcloud_server.server (remote-exec): Connected!
module.kube-hetzner.module.agents["0-0-agent-small"].null_resource.zram (remote-exec): Connected!
module.kube-hetzner.module.control_planes["0-0-control-plane"].hcloud_server.server (remote-exec): Automatic OS updates are disabled
module.kube-hetzner.module.control_planes["0-0-control-plane"].hcloud_server.server (remote-exec): Removed "/etc/systemd/system/timers.target.wants/transactional-update.timer".
module.kube-hetzner.module.agents["0-1-agent-small"].null_resource.zram: Provisioning with 'file'...
module.kube-hetzner.module.agents["0-1-agent-small"].null_resource.registries: Provisioning with 'remote-exec'...
module.kube-hetzner.module.agents["0-1-agent-small"].null_resource.registries (remote-exec): Connecting to remote host via SSH...
module.kube-hetzner.module.agents["0-1-agent-small"].null_resource.registries (remote-exec):   Host: 49.13.194.11
module.kube-hetzner.module.agents["0-1-agent-small"].null_resource.registries (remote-exec):   User: root
module.kube-hetzner.module.agents["0-1-agent-small"].null_resource.registries (remote-exec):   Password: false
module.kube-hetzner.module.agents["0-1-agent-small"].null_resource.registries (remote-exec):   Private key: true
module.kube-hetzner.module.agents["0-1-agent-small"].null_resource.registries (remote-exec):   Certificate: false
module.kube-hetzner.module.agents["0-1-agent-small"].null_resource.registries (remote-exec):   SSH Agent: true
module.kube-hetzner.module.agents["0-1-agent-small"].null_resource.registries (remote-exec):   Checking Host Key: false
module.kube-hetzner.module.agents["0-1-agent-small"].null_resource.registries (remote-exec):   Target Platform: unix
module.kube-hetzner.module.agents["0-1-agent-small"].null_resource.registries (remote-exec): Connected!
module.kube-hetzner.module.control_planes["0-0-control-plane"].hcloud_server.server: Creation complete after 2m19s [id=41383585]
module.kube-hetzner.module.control_planes["0-0-control-plane"].hcloud_server_network.server: Creating...
module.kube-hetzner.module.control_planes["0-0-control-plane"].null_resource.zram: Creating...
module.kube-hetzner.module.control_planes["0-0-control-plane"].null_resource.registries: Creating...
module.kube-hetzner.module.control_planes["0-0-control-plane"].null_resource.zram: Provisioning with 'file'...
module.kube-hetzner.module.control_planes["0-0-control-plane"].null_resource.registries: Provisioning with 'file'...
module.kube-hetzner.module.agents["0-1-agent-small"].null_resource.zram: Provisioning with 'file'...
module.kube-hetzner.module.agents["0-0-agent-small"].null_resource.zram: Creation complete after 4s [id=7401652924668171131]
module.kube-hetzner.module.agents["0-1-agent-small"].null_resource.registries (remote-exec): No update required to the registries.yaml file
module.kube-hetzner.module.agents["0-1-agent-small"].null_resource.registries: Creation complete after 1s [id=3596557116113897097]
module.kube-hetzner.module.control_planes["0-0-control-plane"].null_resource.zram: Provisioning with 'file'...
module.kube-hetzner.module.agents["0-1-agent-small"].null_resource.zram: Provisioning with 'remote-exec'...
module.kube-hetzner.module.agents["0-1-agent-small"].null_resource.zram (remote-exec): Connecting to remote host via SSH...
module.kube-hetzner.module.agents["0-1-agent-small"].null_resource.zram (remote-exec):   Host: 49.13.194.11
module.kube-hetzner.module.agents["0-1-agent-small"].null_resource.zram (remote-exec):   User: root
module.kube-hetzner.module.agents["0-1-agent-small"].null_resource.zram (remote-exec):   Password: false
module.kube-hetzner.module.agents["0-1-agent-small"].null_resource.zram (remote-exec):   Private key: true
module.kube-hetzner.module.agents["0-1-agent-small"].null_resource.zram (remote-exec):   Certificate: false
module.kube-hetzner.module.agents["0-1-agent-small"].null_resource.zram (remote-exec):   SSH Agent: true
module.kube-hetzner.module.agents["0-1-agent-small"].null_resource.zram (remote-exec):   Checking Host Key: false
module.kube-hetzner.module.agents["0-1-agent-small"].null_resource.zram (remote-exec):   Target Platform: unix
module.kube-hetzner.module.control_planes["0-0-control-plane"].null_resource.registries: Provisioning with 'remote-exec'...
module.kube-hetzner.module.control_planes["0-0-control-plane"].null_resource.registries (remote-exec): Connecting to remote host via SSH...
module.kube-hetzner.module.control_planes["0-0-control-plane"].null_resource.registries (remote-exec):   Host: 195.201.27.53
module.kube-hetzner.module.control_planes["0-0-control-plane"].null_resource.registries (remote-exec):   User: root
module.kube-hetzner.module.control_planes["0-0-control-plane"].null_resource.registries (remote-exec):   Password: false
module.kube-hetzner.module.control_planes["0-0-control-plane"].null_resource.registries (remote-exec):   Private key: true
module.kube-hetzner.module.control_planes["0-0-control-plane"].null_resource.registries (remote-exec):   Certificate: false
module.kube-hetzner.module.control_planes["0-0-control-plane"].null_resource.registries (remote-exec):   SSH Agent: true
module.kube-hetzner.module.control_planes["0-0-control-plane"].null_resource.registries (remote-exec):   Checking Host Key: false
module.kube-hetzner.module.control_planes["0-0-control-plane"].null_resource.registries (remote-exec):   Target Platform: unix
module.kube-hetzner.module.agents["0-1-agent-small"].null_resource.zram (remote-exec): Connected!
module.kube-hetzner.module.control_planes["0-0-control-plane"].null_resource.registries (remote-exec): Connected!
module.kube-hetzner.module.control_planes["0-0-control-plane"].null_resource.zram: Provisioning with 'file'...
module.kube-hetzner.module.control_planes["0-0-control-plane"].null_resource.registries (remote-exec): No update required to the registries.yaml file
module.kube-hetzner.module.control_planes["0-0-control-plane"].null_resource.registries: Creation complete after 1s [id=10495896537894926]
module.kube-hetzner.module.control_planes["0-0-control-plane"].null_resource.zram: Provisioning with 'remote-exec'...
module.kube-hetzner.module.control_planes["0-0-control-plane"].null_resource.zram (remote-exec): Connecting to remote host via SSH...
module.kube-hetzner.module.control_planes["0-0-control-plane"].null_resource.zram (remote-exec):   Host: 195.201.27.53
module.kube-hetzner.module.control_planes["0-0-control-plane"].null_resource.zram (remote-exec):   User: root
module.kube-hetzner.module.control_planes["0-0-control-plane"].null_resource.zram (remote-exec):   Password: false
module.kube-hetzner.module.control_planes["0-0-control-plane"].null_resource.zram (remote-exec):   Private key: true
module.kube-hetzner.module.control_planes["0-0-control-plane"].null_resource.zram (remote-exec):   Certificate: false
module.kube-hetzner.module.control_planes["0-0-control-plane"].null_resource.zram (remote-exec):   SSH Agent: true
module.kube-hetzner.module.control_planes["0-0-control-plane"].null_resource.zram (remote-exec):   Checking Host Key: false
module.kube-hetzner.module.control_planes["0-0-control-plane"].null_resource.zram (remote-exec):   Target Platform: unix
module.kube-hetzner.module.agents["0-1-agent-small"].null_resource.zram: Creation complete after 2s [id=4921665980483020868]
module.kube-hetzner.module.control_planes["0-0-control-plane"].null_resource.zram (remote-exec): Connected!
module.kube-hetzner.module.control_planes["0-0-control-plane"].null_resource.zram: Creation complete after 2s [id=5021014374364330763]
module.kube-hetzner.module.control_planes["0-0-control-plane"].hcloud_server_network.server: Creation complete after 3s [id=41383585-3732117]
module.kube-hetzner.null_resource.first_control_plane: Creating...
module.kube-hetzner.null_resource.first_control_plane: Provisioning with 'file'...
module.kube-hetzner.module.agents["0-1-agent-small"].hcloud_server_network.server: Creation complete after 4s [id=41383583-3732117]
module.kube-hetzner.null_resource.first_control_plane: Provisioning with 'remote-exec'...
module.kube-hetzner.null_resource.first_control_plane (remote-exec): Connecting to remote host via SSH...
module.kube-hetzner.null_resource.first_control_plane (remote-exec):   Host: 195.201.27.53
module.kube-hetzner.null_resource.first_control_plane (remote-exec):   User: root
module.kube-hetzner.null_resource.first_control_plane (remote-exec):   Password: false
module.kube-hetzner.null_resource.first_control_plane (remote-exec):   Private key: true
module.kube-hetzner.null_resource.first_control_plane (remote-exec):   Certificate: false
module.kube-hetzner.null_resource.first_control_plane (remote-exec):   SSH Agent: true
module.kube-hetzner.null_resource.first_control_plane (remote-exec):   Checking Host Key: false
module.kube-hetzner.null_resource.first_control_plane (remote-exec):   Target Platform: unix
module.kube-hetzner.module.agents["0-0-agent-small"].hcloud_server_network.server: Creation complete after 7s [id=41383581-3732117]
module.kube-hetzner.null_resource.agent_config["0-0-agent-small"]: Creating...
module.kube-hetzner.null_resource.agent_config["0-1-agent-small"]: Creating...
module.kube-hetzner.null_resource.agent_config["0-0-agent-small"]: Provisioning with 'file'...
module.kube-hetzner.null_resource.agent_config["0-1-agent-small"]: Provisioning with 'file'...
module.kube-hetzner.null_resource.first_control_plane (remote-exec): Connected!
module.kube-hetzner.null_resource.agent_config["0-1-agent-small"]: Provisioning with 'remote-exec'...
module.kube-hetzner.null_resource.agent_config["0-1-agent-small"] (remote-exec): Connecting to remote host via SSH...
module.kube-hetzner.null_resource.agent_config["0-1-agent-small"] (remote-exec):   Host: 49.13.194.11
module.kube-hetzner.null_resource.agent_config["0-1-agent-small"] (remote-exec):   User: root
module.kube-hetzner.null_resource.agent_config["0-1-agent-small"] (remote-exec):   Password: false
module.kube-hetzner.null_resource.agent_config["0-1-agent-small"] (remote-exec):   Private key: true
module.kube-hetzner.null_resource.agent_config["0-1-agent-small"] (remote-exec):   Certificate: false
module.kube-hetzner.null_resource.agent_config["0-1-agent-small"] (remote-exec):   SSH Agent: true
module.kube-hetzner.null_resource.agent_config["0-1-agent-small"] (remote-exec):   Checking Host Key: false
module.kube-hetzner.null_resource.agent_config["0-1-agent-small"] (remote-exec):   Target Platform: unix
module.kube-hetzner.null_resource.agent_config["0-0-agent-small"]: Provisioning with 'remote-exec'...
module.kube-hetzner.null_resource.agent_config["0-0-agent-small"] (remote-exec): Connecting to remote host via SSH...
module.kube-hetzner.null_resource.agent_config["0-0-agent-small"] (remote-exec):   Host: 128.140.75.14
module.kube-hetzner.null_resource.agent_config["0-0-agent-small"] (remote-exec):   User: root
module.kube-hetzner.null_resource.agent_config["0-0-agent-small"] (remote-exec):   Password: false
module.kube-hetzner.null_resource.agent_config["0-0-agent-small"] (remote-exec):   Private key: true
module.kube-hetzner.null_resource.agent_config["0-0-agent-small"] (remote-exec):   Certificate: false
module.kube-hetzner.null_resource.agent_config["0-0-agent-small"] (remote-exec):   SSH Agent: true
module.kube-hetzner.null_resource.agent_config["0-0-agent-small"] (remote-exec):   Checking Host Key: false
module.kube-hetzner.null_resource.agent_config["0-0-agent-small"] (remote-exec):   Target Platform: unix
module.kube-hetzner.null_resource.first_control_plane (remote-exec): + /etc/cloud/rename_interface.sh
module.kube-hetzner.null_resource.agent_config["0-1-agent-small"] (remote-exec): Connected!
module.kube-hetzner.null_resource.agent_config["0-0-agent-small"] (remote-exec): Connected!
module.kube-hetzner.null_resource.agent_config["0-1-agent-small"] (remote-exec): Updated config.yaml detected, restart of k3s service required
module.kube-hetzner.null_resource.agent_config["0-1-agent-small"] (remote-exec): No active k3s or k3s-agent service found
module.kube-hetzner.null_resource.agent_config["0-1-agent-small"] (remote-exec): k3s service or k3s-agent service (re)started successfully
module.kube-hetzner.null_resource.agent_config["0-0-agent-small"] (remote-exec): Updated config.yaml detected, restart of k3s service required
module.kube-hetzner.null_resource.agent_config["0-0-agent-small"] (remote-exec): No active k3s or k3s-agent service found
module.kube-hetzner.null_resource.agent_config["0-0-agent-small"] (remote-exec): k3s service or k3s-agent service (re)started successfully
module.kube-hetzner.null_resource.agent_config["0-1-agent-small"]: Creation complete after 2s [id=6796646789642468754]
module.kube-hetzner.null_resource.agent_config["0-0-agent-small"]: Creation complete after 2s [id=7979394231836354164]
module.kube-hetzner.null_resource.first_control_plane: Still creating... [10s elapsed]
module.kube-hetzner.null_resource.first_control_plane (remote-exec): + mkdir -p /etc/rancher/k3s
module.kube-hetzner.null_resource.first_control_plane (remote-exec): + '[' -f /tmp/config.yaml ']'
module.kube-hetzner.null_resource.first_control_plane (remote-exec): + mv /tmp/config.yaml /etc/rancher/k3s/config.yaml
module.kube-hetzner.null_resource.first_control_plane (remote-exec): + chmod 0600 /etc/rancher/k3s/config.yaml
module.kube-hetzner.null_resource.first_control_plane (remote-exec): + '[' -e /etc/rancher/k3s/k3s.yaml ']'
module.kube-hetzner.null_resource.first_control_plane (remote-exec): + cat
module.kube-hetzner.null_resource.first_control_plane (remote-exec): + set -a
module.kube-hetzner.null_resource.first_control_plane (remote-exec): + source /etc/environment
module.kube-hetzner.null_resource.first_control_plane (remote-exec): + set +a
module.kube-hetzner.null_resource.first_control_plane (remote-exec): + cat
module.kube-hetzner.null_resource.first_control_plane (remote-exec): + cat
module.kube-hetzner.null_resource.first_control_plane (remote-exec): + timeout 180s /bin/sh -c 'while ! ping -c 1 1.1.1.1 >/dev/null 2>&1; do echo "Ready for k3s installation, waiting for a successful connection to the internet..."; sleep 5; done; echo Connected'
module.kube-hetzner.null_resource.first_control_plane (remote-exec): Connected
module.kube-hetzner.null_resource.first_control_plane (remote-exec): + curl -sfL https://get.k3s.io
module.kube-hetzner.null_resource.first_control_plane (remote-exec): + INSTALL_K3S_SKIP_START=true
module.kube-hetzner.null_resource.first_control_plane (remote-exec): + INSTALL_K3S_SKIP_SELINUX_RPM=true
module.kube-hetzner.null_resource.first_control_plane (remote-exec): + INSTALL_K3S_CHANNEL=v1.28
module.kube-hetzner.null_resource.first_control_plane (remote-exec): + INSTALL_K3S_EXEC='server '
module.kube-hetzner.null_resource.first_control_plane (remote-exec): + sh -
module.kube-hetzner.null_resource.first_control_plane (remote-exec): [INFO]  Finding release for channel v1.28
module.kube-hetzner.null_resource.first_control_plane (remote-exec): [INFO]  Using v1.28.5+k3s1 as release
module.kube-hetzner.null_resource.first_control_plane (remote-exec): [INFO]  Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.28.5+k3s1/sha256sum-arm64.txt
module.kube-hetzner.null_resource.first_control_plane (remote-exec): [INFO]  Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.28.5+k3s1/k3s-arm64
module.kube-hetzner.null_resource.first_control_plane (remote-exec): [INFO]  Verifying binary download
module.kube-hetzner.null_resource.first_control_plane (remote-exec): [INFO]  Installing k3s to /usr/local/bin/k3s
module.kube-hetzner.null_resource.first_control_plane (remote-exec): [INFO]  Skipping installation of SELinux RPM
module.kube-hetzner.null_resource.first_control_plane (remote-exec): [INFO]  Creating /usr/local/bin/kubectl symlink to k3s
module.kube-hetzner.null_resource.first_control_plane (remote-exec): [INFO]  Creating /usr/local/bin/crictl symlink to k3s
module.kube-hetzner.null_resource.first_control_plane (remote-exec): [INFO]  Creating /usr/local/bin/ctr symlink to k3s
module.kube-hetzner.null_resource.first_control_plane (remote-exec): [INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
module.kube-hetzner.null_resource.first_control_plane (remote-exec): [INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh
module.kube-hetzner.null_resource.first_control_plane (remote-exec): [INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env
module.kube-hetzner.null_resource.first_control_plane (remote-exec): [INFO]  systemd: Creating service file /etc/systemd/system/k3s.service
module.kube-hetzner.null_resource.first_control_plane (remote-exec): [INFO]  systemd: Enabling k3s unit
module.kube-hetzner.null_resource.first_control_plane (remote-exec): Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
module.kube-hetzner.null_resource.first_control_plane (remote-exec): + /sbin/semodule -v -i /usr/share/selinux/packages/k3s.pp
module.kube-hetzner.null_resource.first_control_plane (remote-exec): Attempting to install module '/usr/share/selinux/packages/k3s.pp':
module.kube-hetzner.null_resource.first_control_plane (remote-exec): libsemanage.semanage_direct_install_info: Overriding k3s module at lower priority 200 with module at priority 400.
module.kube-hetzner.null_resource.first_control_plane (remote-exec): Ok: return value of 0.
module.kube-hetzner.null_resource.first_control_plane (remote-exec): Committing changes:
module.kube-hetzner.null_resource.first_control_plane: Still creating... [20s elapsed]
module.kube-hetzner.null_resource.first_control_plane (remote-exec): Ok: transaction number 8.
module.kube-hetzner.null_resource.first_control_plane: Provisioning with 'remote-exec'...
module.kube-hetzner.null_resource.first_control_plane (remote-exec): Connecting to remote host via SSH...
module.kube-hetzner.null_resource.first_control_plane (remote-exec):   Host: 195.201.27.53
module.kube-hetzner.null_resource.first_control_plane (remote-exec):   User: root
module.kube-hetzner.null_resource.first_control_plane (remote-exec):   Password: false
module.kube-hetzner.null_resource.first_control_plane (remote-exec):   Private key: true
module.kube-hetzner.null_resource.first_control_plane (remote-exec):   Certificate: false
module.kube-hetzner.null_resource.first_control_plane (remote-exec):   SSH Agent: true
module.kube-hetzner.null_resource.first_control_plane (remote-exec):   Checking Host Key: false
module.kube-hetzner.null_resource.first_control_plane (remote-exec):   Target Platform: unix
module.kube-hetzner.null_resource.first_control_plane (remote-exec): Connected!
module.kube-hetzner.null_resource.first_control_plane (remote-exec): Job for k3s.service failed because the control process exited with error code.
module.kube-hetzner.null_resource.first_control_plane (remote-exec): See "systemctl status k3s.service" and "journalctl -xeu k3s.service" for details.
module.kube-hetzner.null_resource.first_control_plane (remote-exec): Job for k3s.service failed because the control process exited with error code.
module.kube-hetzner.null_resource.first_control_plane (remote-exec): See "systemctl status k3s.service" and "journalctl -xeu k3s.service" for details.
module.kube-hetzner.null_resource.first_control_plane (remote-exec): Waiting for the k3s server to start...
module.kube-hetzner.null_resource.first_control_plane: Still creating... [30s elapsed]
module.kube-hetzner.null_resource.first_control_plane (remote-exec): Job for k3s.service failed because the control process exited with error code.
module.kube-hetzner.null_resource.first_control_plane (remote-exec): See "systemctl status k3s.service" and "journalctl -xeu k3s.service" for details.
module.kube-hetzner.null_resource.first_control_plane (remote-exec): Waiting for the k3s server to start...
module.kube-hetzner.null_resource.first_control_plane: Still creating... [40s elapsed]
module.kube-hetzner.null_resource.first_control_plane (remote-exec): Job for k3s.service failed because the control process exited with error code.
module.kube-hetzner.null_resource.first_control_plane (remote-exec): See "systemctl status k3s.service" and "journalctl -xeu k3s.service" for details.
module.kube-hetzner.null_resource.first_control_plane (remote-exec): Waiting for the k3s server to start...
module.kube-hetzner.null_resource.first_control_plane (remote-exec): Job for k3s.service failed because the control process exited with error code.
module.kube-hetzner.null_resource.first_control_plane (remote-exec): See "systemctl status k3s.service" and "journalctl -xeu k3s.service" for details.
module.kube-hetzner.null_resource.first_control_plane (remote-exec): Waiting for the k3s server to start...
module.kube-hetzner.null_resource.first_control_plane: Still creating... [50s elapsed]
module.kube-hetzner.null_resource.first_control_plane (remote-exec): Job for k3s.service failed because the control process exited with error code.
module.kube-hetzner.null_resource.first_control_plane (remote-exec): See "systemctl status k3s.service" and "journalctl -xeu k3s.service" for details.
module.kube-hetzner.null_resource.first_control_plane (remote-exec): Waiting for the k3s server to start...
module.kube-hetzner.null_resource.first_control_plane (remote-exec): Job for k3s.service failed because the control process exited with error code.
module.kube-hetzner.null_resource.first_control_plane (remote-exec): See "systemctl status k3s.service" and "journalctl -xeu k3s.service" for details.
module.kube-hetzner.null_resource.first_control_plane (remote-exec): Waiting for the k3s server to start...
module.kube-hetzner.null_resource.first_control_plane: Still creating... [1m0s elapsed]
module.kube-hetzner.null_resource.first_control_plane (remote-exec): Job for k3s.service failed because the control process exited with error code.
module.kube-hetzner.null_resource.first_control_plane (remote-exec): See "systemctl status k3s.service" and "journalctl -xeu k3s.service" for details.
module.kube-hetzner.null_resource.first_control_plane (remote-exec): Waiting for the k3s server to start...
module.kube-hetzner.null_resource.first_control_plane (remote-exec): Job for k3s.service failed because the control process exited with error code.
module.kube-hetzner.null_resource.first_control_plane (remote-exec): See "systemctl status k3s.service" and "journalctl -xeu k3s.service" for details.
module.kube-hetzner.null_resource.first_control_plane (remote-exec): Waiting for the k3s server to start...
module.kube-hetzner.null_resource.first_control_plane: Still creating... [1m10s elapsed]
module.kube-hetzner.null_resource.first_control_plane (remote-exec): Job for k3s.service failed because the control process exited with error code.
module.kube-hetzner.null_resource.first_control_plane (remote-exec): See "systemctl status k3s.service" and "journalctl -xeu k3s.service" for details.
module.kube-hetzner.null_resource.first_control_plane (remote-exec): Waiting for the k3s server to start...
module.kube-hetzner.null_resource.first_control_plane (remote-exec): Job for k3s.service failed because the control process exited with error code.
module.kube-hetzner.null_resource.first_control_plane (remote-exec): See "systemctl status k3s.service" and "journalctl -xeu k3s.service" for details.
module.kube-hetzner.null_resource.first_control_plane (remote-exec): Waiting for the k3s server to start...
module.kube-hetzner.null_resource.first_control_plane: Still creating... [1m20s elapsed]
module.kube-hetzner.null_resource.first_control_plane (remote-exec): Job for k3s.service failed because the control process exited with error code.
module.kube-hetzner.null_resource.first_control_plane (remote-exec): See "systemctl status k3s.service" and "journalctl -xeu k3s.service" for details.
module.kube-hetzner.null_resource.first_control_plane (remote-exec): Waiting for the k3s server to start...
module.kube-hetzner.null_resource.first_control_plane (remote-exec): Job for k3s.service failed because the control process exited with error code.
module.kube-hetzner.null_resource.first_control_plane (remote-exec): See "systemctl status k3s.service" and "journalctl -xeu k3s.service" for details.
module.kube-hetzner.null_resource.first_control_plane (remote-exec): Waiting for the k3s server to start...
module.kube-hetzner.null_resource.first_control_plane: Still creating... [1m30s elapsed]
module.kube-hetzner.null_resource.first_control_plane (remote-exec): Job for k3s.service failed because the control process exited with error code.
module.kube-hetzner.null_resource.first_control_plane (remote-exec): See "systemctl status k3s.service" and "journalctl -xeu k3s.service" for details.
module.kube-hetzner.null_resource.first_control_plane (remote-exec): Waiting for the k3s server to start...
module.kube-hetzner.null_resource.first_control_plane (remote-exec): Job for k3s.service failed because the control process exited with error code.
module.kube-hetzner.null_resource.first_control_plane (remote-exec): See "systemctl status k3s.service" and "journalctl -xeu k3s.service" for details.
module.kube-hetzner.null_resource.first_control_plane (remote-exec): Waiting for the k3s server to start...
module.kube-hetzner.null_resource.first_control_plane: Still creating... [1m40s elapsed]
module.kube-hetzner.null_resource.first_control_plane (remote-exec): Job for k3s.service failed because the control process exited with error code.
module.kube-hetzner.null_resource.first_control_plane (remote-exec): See "systemctl status k3s.service" and "journalctl -xeu k3s.service" for details.
module.kube-hetzner.null_resource.first_control_plane (remote-exec): Waiting for the k3s server to start...
module.kube-hetzner.null_resource.first_control_plane (remote-exec): Job for k3s.service failed because the control process exited with error code.
module.kube-hetzner.null_resource.first_control_plane (remote-exec): See "systemctl status k3s.service" and "journalctl -xeu k3s.service" for details.
module.kube-hetzner.null_resource.first_control_plane (remote-exec): Waiting for the k3s server to start...
module.kube-hetzner.null_resource.first_control_plane: Still creating... [1m50s elapsed]
module.kube-hetzner.null_resource.first_control_plane (remote-exec): Job for k3s.service failed because the control process exited with error code.
module.kube-hetzner.null_resource.first_control_plane (remote-exec): See "systemctl status k3s.service" and "journalctl -xeu k3s.service" for details.
module.kube-hetzner.null_resource.first_control_plane (remote-exec): Waiting for the k3s server to start...
module.kube-hetzner.null_resource.first_control_plane (remote-exec): Job for k3s.service failed because the control process exited with error code.
module.kube-hetzner.null_resource.first_control_plane (remote-exec): See "systemctl status k3s.service" and "journalctl -xeu k3s.service" for details.
module.kube-hetzner.null_resource.first_control_plane (remote-exec): Waiting for the k3s server to start...
module.kube-hetzner.null_resource.first_control_plane: Still creating... [2m0s elapsed]
module.kube-hetzner.null_resource.first_control_plane (remote-exec): Job for k3s.service failed because the control process exited with error code.
module.kube-hetzner.null_resource.first_control_plane (remote-exec): See "systemctl status k3s.service" and "journalctl -xeu k3s.service" for details.
module.kube-hetzner.null_resource.first_control_plane (remote-exec): Waiting for the k3s server to start...
module.kube-hetzner.null_resource.first_control_plane (remote-exec): Job for k3s.service failed because the control process exited with error code.
module.kube-hetzner.null_resource.first_control_plane (remote-exec): See "systemctl status k3s.service" and "journalctl -xeu k3s.service" for details.
module.kube-hetzner.null_resource.first_control_plane (remote-exec): Waiting for the k3s server to start...
module.kube-hetzner.null_resource.first_control_plane: Still creating... [2m10s elapsed]
module.kube-hetzner.null_resource.first_control_plane (remote-exec): Job for k3s.service failed because the control process exited with error code.
module.kube-hetzner.null_resource.first_control_plane (remote-exec): See "systemctl status k3s.service" and "journalctl -xeu k3s.service" for details.
module.kube-hetzner.null_resource.first_control_plane (remote-exec): Waiting for the k3s server to start...
module.kube-hetzner.null_resource.first_control_plane (remote-exec): Job for k3s.service failed because the control process exited with error code.
module.kube-hetzner.null_resource.first_control_plane (remote-exec): See "systemctl status k3s.service" and "journalctl -xeu k3s.service" for details.
module.kube-hetzner.null_resource.first_control_plane (remote-exec): Waiting for the k3s server to start...
module.kube-hetzner.null_resource.first_control_plane: Still creating... [2m20s elapsed]
╷
│ Error: remote-exec provisioner error
│
│   with module.kube-hetzner.null_resource.first_control_plane,
│   on .terraform/modules/kube-hetzner/init.tf line 73, in resource "null_resource" "first_control_plane":
│   73:   provisioner "remote-exec" {
│
│ error executing "/tmp/terraform_541376020.sh": Process exited with status 124
╵
➜  tf git:(main) ✗

Platform

Mac

[sebboer]

I have the same problem since today. Came out of nowhere. It also happens when scaling up. If you start /usr/local/bin/k3s manually with the root user, the agent connects successfully to the cluster. The systemd service does not seem to get the right permissions.

[RalucaPenciuc]

Downgrading to previous versions of the module does not solve the issue

[sebboer]

Seems to be a problem with selinux.
Quick and dirty workaround:

module "kube-hetzner" {

  postinstall_exec = ["restorecon -v /usr/local/bin/k3s"]
  ...
}

[mnencia]

I confirm this bug and the workaround. I wonder what has changed...

[Frodothedwarf]

Had the same bug happen for me, the workaround with the postinstall_exec, did it.

[Silvest89]

It probably something to do with SELINUX and the OS updates.
@mysticaltech

[mysticaltech]

@Silvest89 Indeed, I will correct it tonight, see which SELinux rules are needed.

[mysticaltech]

Thankd for the debug folks, it's fixed in v2.11.4.