containerd kept restarting - weird problem solved

[dwilliss]

This is my experience with the hope that it may help someone else with similar issues. Although I had a hard enough time finding any advice on the problems I was having that I doubt it :-)

I have been trying to follow the video about setting up k3s with Ansible for a couple of weeks now and it just wasn't working. The kube-vip would come up but only stay up for a few seconds. A ping would say destination unreachable for several seconds then it would come back for a few seconds.

It turns out that containerd was crashing because of timeouts in etcd.
Everything I found online said you needed to run the node on fast SSDs. I dismissed this because my nodes are running on VMs under Proxmox and the drives are on a zpool made up of 8 SSDs laid out in 2 RAIDZ1 VDEVs striped in RAID0

Surely that's fast, right? A little voice in the back of my head told me to confirm that. A bit more Googling and I found a tool called dbench which showed that I was averaging about 2MB/sec. That don't seem right.

A bit more Googling and I found the following suggestions:

zfs set sync=disabled zpool
zfs set atime=off zpool

The first command turns off synchronous writes. People will tell you this is a Bad Idea. Bah! The worst that could happen is that if the power goes out suddenly I'd lose the last several seconds of writes. That's what UPSs are for.
The second command disables keeping track of access time. With it on, every time you access a file, it updates some bit of metadata so that if you can pass a flag to ls to have it sort by most recently accessed. I never do that. Don't care.

After doing both of these commands, dbench gives me throughputs of 80-100 MB/sec.

Now my pool comes up just fine and is stable.

New problem: I just did a git pull on this repo and kube-vip isn't even trying to load. There are no logs for it at all now.
Although one thing I didn't try at this point was to run the reset.sh and deploy.sh scripts again to start over.

I found another YouTube video titled "Easy K3s Kubernetes tools WITH K3sup and Kube-VIP"
Following the section of his video about how to setup kube-vip got it running.

A ping of my VIP has been going for over 30 minutes now with no dropouts.

Next step: Rancher!