High CPU usage on fresh install makes K3S unusable immediately

[2fst4u]

Environmental Info:
K3s Version: v1.22.3+k3s1, and v1.21.5+k3s2 (latest and stable channels via install flags)

Node(s) CPU architecture, OS, and Version: Raspberry Pi 4 8GB - Linux k3s-node1 5.13.0-1010-raspi #12-Ubuntu SMP PREEMPT Tue Nov 9 15:30:22 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux

Cluster Configuration: single node. Single node is unusable before I even get to trying multiple nodes

Describe the bug: A fresh install is using an insanely high amount of CPU on my raspberry pi 4 8GB. To the point that kubectl commands are intermittently unavailable due to API being unresponsive. Sometimes they work, mostly they fail.

Steps To Reproduce:

• Installed K3s: curl -sfL https://get.k3s.io | INSTALL_K3S_CHANNEL=stable sh -s - server --disable traefik --cluster-init --token secret --kubelet-arg system-reserved=cpu=100m,memory=80Mi --kubelet-arg kube-reserved=cpu=200m,memory=500Mi

I found previously I needed to reserve compute otherwise everything would crash. I have tried with omission of the kubelet-args and the same occurs.

Expected behavior: K3S starts with less than 10% CPU usage

Actual behavior: CPU usage jumps to 300+% and stays consistently high.

Additional context / logs: happy to provide specific logs if you let me know what you're after.

Backporting

• Needs backporting to older releases

So some background, I have previously had a K3S cluster running with a single master and 2 workers on my 3 Raspberry Pi 4 cluster and it was going great. I had no issues and it worked as intended. The cluster failed catastrophically for unrelated reasons so I have rebuilt it from scratch.

I was using SD cards and initialised the cluster with etcd, which didn't wok well due to IO I presume. I then went through the process to boot the pis from SSD and that's where all nodes are running now (full SSD boot, no SDs in the pis at all)

With completely fresh installs of Ubuntu 21.10 and nothing else running, I tried to install K3S expecting it to work even better than it had before, however it's completely unresponsive. I thought this might be an etcd issue but no matter how I install K3S the same thing happens.

I have no idea what is different between this and my old install but now with what I thought would be a better system it is completely unusable from fresh and I am wracking my brains.

Here is a screenshot of htop to show what's happening:

CPU usages occasionally jumps down but generally stays up this high or higher.

Any help greatly appreciated.

[2fst4u]

As is always the case, I found the answer after hours of searching and finally reaching out for help.

#4234 (comment)

This appears to have resolved the issue. No idea which part of it specifically makes it work but it would be good if there was some sort of check for the element that fixes it, or a note in docs for raspberry pi users.

Nonetheless, problem solved and all is well.

[OneCricketeer]

Same happens on Debian and the linked command for Ubuntu does not work in Rasbian/DietPi because package isn't found

[mcmil]

I've had the same issue (high cpu usage) on Raspberry Pi OS and Raspberry Pi 2B. For me running the following commands helped (source):

sudo update-alternatives --set iptables /usr/sbin/iptables-legacy
sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy

[brandond]

@mcmil if your issue is related to iptables, you might check out https://docs.k3s.io/advanced#old-iptables-versions