microK8s stalls on OCI E2.1.Micro instance

[worldofgeese]

Details of the instance are here under Compute. It's a 1 gb ram instance with 40 gigs of attached storage, which according to the issues I've read here, should fall within minimum system reqs.

1.19 never moves past the configure hook step when installing. 1.18, 1.18 refreshed to 1.19, 1.17, and 1.16 stall given a microk8s status or microk8s inspect command 90% of the time. I was able to get one inspect tarball out of this, which I've attached.

If anyone else wants to give this a try, Oracle Cloud offers no-catch always free compute resources. You can use their Terraform example here together with the relevant Terraform provider docs.

I'd love to receive an answer on either A. what the official minimum system requirements of microK8s are or B. what I can do to get a functional installation. It doesn't need more than dns and storage enabled (for the purposes of testing, I never enabled these).

inspection-report-20201027_170633.tar.gz

[balchua]

Imho 1Gb for kubernetes is too small. I've never been able to run kubernetes on a 1vcpu and 1Gb machine. Using kubeadm or MicroK8s. The smallest was 2Gb, but its crawling. I haven't even installed any workload.

[worldofgeese]

Imho 1Gb for kubernetes is too small.

K3s occupies this same space of small and lean k8s distros and does fine on a 1Gb vps.

For what it's worth I have microK8s running two deployments on a 2 gig AWS vps no problem.

But if it is the case that microK8s requires 2 gig minimum, I'd like to see a direct response from the team at Canonical updating the minimums.

[worldofgeese]

Pinging @ktsakalozos

[ktsakalozos]

Hi @worldofgeese,

In our docs we recommend at least 4GB of RAM. 4GB is not the minimum amount of memory you can get MicroK8s running with but considering we would like you to have a decent experience 4GB should be enough for kubernetes plus a basic workload.

If your goal is to just get MicroK8s running with as few resources as possible here are a few things you could try:

• Setup memory swap so the OS can swap out unused pages. Make sure you understand the implications of doing so [2].
• Use the 1.18/stable channel. The 1.19 will fail to configure if the API server does not start within 2 minutes from the installation time. See [3] on how to overcome this.

[1] https://microk8s.io/docs
[2] #1598 (comment)
[3] #1623 (comment)

[worldofgeese]

Hi @worldofgeese,

In our docs we recommend at least 4GB of RAM. 4GB is not the minimum amount of memory you can get MicroK8s running with but considering we would like you to have a decent experience 4GB should be enough for kubernetes plus a basic workload.

If your goal is to just get MicroK8s running with as few resources as possible here are a few things you could try:

• Setup memory swap so the OS can swap out unused pages. Make sure you understand the implications of doing so [2].
• Use the 1.18/stable channel. The 1.19 will fail to configure if the API server does not start within 2 minutes from the installation time. See [3] on how to overcome this.

[1] https://microk8s.io/docs
[2] #1598 (comment)
[3] #1623 (comment)

Hello, thank you for your reply. Under [2] I see nothing about the implications of enabling memory swap.

What I have gleaned from [3] is to run ​sudo snap install microk8s --classic --channel=latest/edge/slow-start. Is that correct?

[ktsakalozos]

You can read more on swap effects in https://discuss.kubernetes.io/t/swap-off-why-is-it-necessary/6879

Yes, you can try the --channel=latest/edge/slow-start channel to work around the installation failing during the configure

[worldofgeese]

You can read more on swap effects in https://discuss.kubernetes.io/t/swap-off-why-is-it-necessary/6879

Yes, you can try the --channel=latest/edge/slow-start channel to work around the installation failing during the configure

I started by making and activating a swapfile:

sudo fallocate -l 3G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
sudo sysctl vm.swappiness=10
sudo sysctl vm.vfs_cache_pressure=50

then installing the slow-start branch of microk8s 1.19

sudo snap install microk8s --classic --channel=latest/edge/slow-start

and enabled dns and storage with sudo microk8s.enable dns storage

Unfortunately, microk8s.status still reports microk8s is not running. Use microk8s inspect for a deeper inspection.

microk8s.inspect:

Inspecting Certificates
Inspecting services
  Service snap.microk8s.daemon-cluster-agent is running
  Service snap.microk8s.daemon-containerd is running
  Service snap.microk8s.daemon-apiserver is running
  Service snap.microk8s.daemon-apiserver-kicker is running
  Service snap.microk8s.daemon-control-plane-kicker is running
  Service snap.microk8s.daemon-proxy is running
  Service snap.microk8s.daemon-kubelet is running
  Service snap.microk8s.daemon-scheduler is running
  Service snap.microk8s.daemon-controller-manager is running
  Copy service arguments to the final report tarball
Inspecting AppArmor configuration
Gathering system information
  Copy processes list to the final report tarball
  Copy snap list to the final report tarball
  Copy VM name (or none) to the final report tarball
  Copy disk usage information to the final report tarball
  Copy memory usage information to the final report tarball
  Copy server uptime to the final report tarball
  Copy current linux distribution to the final report tarball
  Copy openSSL information to the final report tarball
  Copy network configuration to the final report tarball
Inspecting kubernetes cluster
  Inspect kubernetes cluster

Building the report tarball
  Report tarball is at /var/snap/microk8s/1758/inspection-report-20201030_120642.tar.gz

Report tarball attached. Please advise.
inspection-report-20201030_120642.tar.gz

[worldofgeese]

@ktsakalozos good morning 🌅. Just wanted to swing back to this since we had the weekend between us and see if anything could be learned from my attached logs?

[ktsakalozos]

Hi @worldofgeese

I see calls to the apiserver result in:

Unable to connect to the server: dial tcp 158.101.164.62:16443: connect: no route to host

The API server should be listening on port 16443 on all interfaces. The 158.101.164.62 seems to be the IP of the default interface, right? However, I cannot explain the "no route to host" error message. in the past on Oracle cloud we have seen we had some issues with the iptable rules injected by the cloud vendor. Can you see if you are affected by [1]?

[1] #854 (comment)

[worldofgeese]

I see in the issue linked to do the following for Oracle Cloud Infrastructure deployments:

sudo iptabled -D  INPUT -j REJECT --reject-with icmp-host-prohibited
sudo iptabled -D  FORWARD -j REJECT --reject-with icmp-host-prohibited

I needed to change the two instances of iptabled to iptables

sudo iptables -D  INPUT -j REJECT --reject-with icmp-host-prohibited
sudo iptables -D  FORWARD -j REJECT --reject-with icmp-host-prohibited

after which microk8s status reports microk8s is running!

Thank you @ktsakalozos for sticking with me through this one! 🙇