Using this playbook in AWS

[teamsignprompt]

Configuring a single master with 2 nodes in host.ini hangs at the task Enable and check K3s service for nodes. All machines are freshly installed VMs running on AWS, with unique hostnames and no local firewalls running.

Expected Behavior

Running the playbook should run all tasks through to completion, and worker nodes should be able to join the cluster successfully.

Current Behavior

As described above, and in steps to reproduce, running the playbook hangs on task Enable and check K3s service for the nodes, because the nodes cannot join the cluster.

Steps to Reproduce

1. Run command ansible-playbook site.yml inventory/bigpod/hosts.ini -K
2. Enter password at prompt.
3. Watch task each complete...
4. ...until TASK [k3s/node : Enable and check K3s service] (which hangs until doing a Ctrl+C to cancel)
5. SSH into master.
6. Run sudo kubectl get nodes
7. Observe that control-plane,master has a status of Ready, but no nodes are listed.
8. Exit from master.
9. SSH into one of the nodes.
10. Run command sudo systemctl status k3s-node.
11. Read numerous error messages failed to get CA certs: Get \"https://127.0.0.1:6444/cacerts\": read tcp 127.0.0.1:xxxxx->127.0.0.1:6444: read: connection reset by peer, where the port number xxxxx is different in each error message.
12. Run command sudo systemctl stop k3s-node
13. Run command sudo systemctl start k3s-node
14. Wait a long time for the service to start (and wait, and wait some more).
15. Give up waiting and run Ctrl-C to regain control of the command line.
16. Run command sudo systemctl status k3s-node again.
17. Observe the same error message about connection reset by peer, happening every 6 seconds.
18. Open file /etc/systemd/system/k3s-node.service
19. Copy the ExecStart command.
20. Run the command manually.
21. Observe the same error message about connection reset by peer.

Context (variables)

Operating system: Ubuntu 20.04
Ansible version: 2.12.10

Hardware (Master):

Total Memory: 2.0GiB
Total Storage: 20.0GiB
Cores: 1

Hardware (Nodes):

Total Memory: 1.0GiB
Total Storage: 10.0GiB
Cores: 1

Variables Used

all.yml

k3s_version: " v1.24.10+k3s1"
ansible_user: NA
systemd_dir: "/etc/systemd/system"

flannel_iface: "eth0"

apiserver_endpoint: "172.31.8.100"

k3s_token: "NA"

extra_server_args: >-
  {{ extra_args }}
  {{ '--node-taint node-role.kubernetes.io/master=true:NoSchedule' if k3s_master_taint else '' }}
  --tls-san {{ apiserver_endpoint }}
  --disable servicelb
  --disable traefik
extra_agent_args: >-
  {{ extra_args }}

kube_vip_tag_version: "v0.5.7"

metal_lb_speaker_tag_version: "v0.13.7"
metal_lb_controller_tag_version: "v0.13.7"

metal_lb_ip_range: "172.31.8.80-172.31.8.90"

Hosts

host.ini

[master]
172.31.8.144

[node]
172.31.2.207
172.31.12.34

[k3s_cluster:children]
master
node

Possible Solution

• I've checked the General Troubleshooting Guide

[djhunter67]

This has happened to me and my cluster of raspberry pi 4's w/ Ubuntu 22.04. My painstaking effort to solve this is to take all ten pi's external boot SSDs and put raspian os on them. I believe this failure is caused by an sftp transfer failure.

I changed the ansible config setting from 'smart' to 'true'. After hours of troubleshooting, I have to wipe all of the drives because they each had unique usernames. All my attempts to ansible new users have failed. Thus, new operating systems for everyone.

[djhunter67]

This has happened to me and my cluster of raspberry pi 4's w/ Ubuntu 22.04. My painstaking effort to solve this is to take all ten pi's external boot SSDs and put raspian os on them. I believe this failure is caused by an sftp transfer failure.

I changed the ansible config setting from 'smart' to 'true'. After hours of troubleshooting, I have to wipe all of the drives because they each had unique usernames. All my attempts to ansible new users have failed. Thus, new operating systems for everyone.

No joy. Not gonna get those hours back...

Fresh raspian install and change the config for ansible ssh back to 'smart'. Back to the 'sftp' failures and freeze at 'Enable and check service".

[teamsignprompt]

After some additional research perhaps my issue is related to MetalLB, because I am running on AWS and not bare metal?

[timothystewart6]

After some additional research perhaps my issue is related to MetalLB, because I am running on AWS and not bare metal?

That's most likely it. Metal LB simulates a cloud load balancer and in the cloud, you have a cloud load balancer :)

MetalLB says to just used EKS, so I assume that means no.

https://metallb.universe.tf/installation/clouds/

[teamsignprompt]

After some additional research perhaps my issue is related to MetalLB, because I am running on AWS and not bare metal?

That's most likely it. Metal LB simulates a cloud load balancer and in the cloud, you have a cloud load balancer :)

MetalLB says to just used EKS, so I assume that means no.

https://metallb.universe.tf/installation/clouds/

Makes perfect sense.