Slow Memory Leak over time with K3s HA Setup (Ubuntu 22.04 + K3s 1.25)

[nshores]

I've been battling a memory leak for some time now when using K3S in a multi-master HA setup. I'm running 3x nodes --

k3s-master-2 - Ubuntu 20.04 - (4vCPU 8GB Memory) - 5.4.0-162-generic - No memory leaks observed
k3s-master-5 - Ubuntu 22.04 (8vCPU 32GB Memory) - 5.15.0-75-generic - Leaks memory quickly (100mb per hour)
k3s-master-6 - Ubuntu 22.04 (10vCPU 16GB Memory) - 5.15.0-83-generic - Leaks memory slower (20-50mb per hour)

NAME           STATUS   ROLES                       AGE    VERSION
k3s-master-2   Ready    control-plane,etcd,master   321d   v1.25.13+k3s1
k3s-master-5   Ready    control-plane,etcd,master   105d   v1.25.13+k3s1
k3s-master-6   Ready    control-plane,etcd,master   36h    v1.25.13+k3s1

k3s systemd unit settings:

ExecStart=/usr/local/bin/k3s \
    server \
	'--disable' \
	'traefik' \
	'--disable' \
	'servicelb' \
	'server' \
	'--server' \
	'https://192.168.99.201:6443' 

All running in ESXI on NVME w/ 10gb networking. All machines have minimal workloads - but k3s-master-5 has the most at the moment. I'm running Longhorn 1.4.5 for storage.

For the past few weeks, I've been noticing a memory leak on the Ubuntu 22.04 nodes. It's particularly pronounced on one of them, And I've ruled out any running containers as being the cause. I see the "memory used"(node_memory_MemTotal_bytes - node_memory_MemFree_bytes- node_memory_Cached_bytes - node_memory_Buffers_bytes) rising, and "memory available" (node_memory_MemAvailable_bytes) falling.

On the node in question, I see high slab memory usage (SUnreclaim ) - rising over time,

Using smem I can see that the memory usage is coming from "Kernel Dynamic Memory" (Specifically non-cache) - AKA - not a userland process.

nshores@k3s-master-5:/var/lib/rancher/k3s$ sudo smem -w -k -t
Area                           Used      Cache   Noncache
firmware/hardware                 0          0          0
kernel image                      0          0          0
kernel dynamic memory         27.1G      26.1G       1.0G
userspace memory               3.9G     951.8M       3.0G
free memory                  347.9M     347.9M          0
----------------------------------------------------------
                              31.3G      27.3G       4.0G

slabtop shows high usage of kmalloc-256 that rises over time. My behavior follows this bug - but that's been fixed in the kernel I'm running - so i've ruled that out.

283552 283552 100%    0.25K   8861       32     70888K kmalloc-256

Stopping the k3s-service process stops the memory leak. It also releases the kernel memory I observed in use. Running k3s-killall.sh also clears further memory.

I suspect this is related to a process it's running, specifically etcd . I upgraded from 1.23 > 1.24 > 1.25 and the issue persisted. The odd thing is that it doesn't appear to affect ubuntu 20.04 , only 22.04 .

This has been driving me insane for many weeks, I'd appreciate any insight. I see multiple threads discussing memory + cpu utilization issues in general here - But none of them appear to be this exact issue.

[brandond]

Stopping the k3s-service process stops the memory leak. It however does not release the memory, only a restart of the node clears it. I suspect this is related to a process it's running, specifically etcd.

If stopping the k3s service doesn't free the memory, then whatever's going on is not a K3s issue. etcd is run in the main k3s process; if the memory isn't freed when that process is terminated then something is being leaked (or at least consumed and not freed) within the kernel itself.

Have you looked at overlayfs or other things like that as the cause of the growing memory utilization? If you run k3s-killall.sh to terminate all the pods and clean up the iptables rules, is the memory freed?

[nshores]

For reference, this is what k3s-master-5 eventually ends up looking like:

[nshores]

Some interesting feedback:

Before killing k3s.service

nshores@k3s-master-6:~$ sudo smem -w -k -t && echo $(date)
Area                           Used      Cache   Noncache
firmware/hardware                 0          0          0
kernel image                      0          0          0
kernel dynamic memory          2.8G       2.4G     407.2M
userspace memory               1.9G     473.5M       1.4G
free memory                   10.9G      10.9G          0
----------------------------------------------------------
                              15.6G      13.8G       1.8G

After stopping k3s.service

nshores@k3s-master-6:~$ sudo smem -w -k -t && echo $(date)
Area                           Used      Cache   Noncache
firmware/hardware                 0          0          0
kernel image                      0          0          0
kernel dynamic memory          2.8G       2.6G     225.7M
userspace memory             510.6M     235.8M     274.8M
free memory                   12.3G      12.3G          0
----------------------------------------------------------
                              15.6G      15.1G     500.5M

After running k3s-killall.sh

nshores@k3s-master-6:~$ sudo smem -w -k -t && echo $(date)
Area                           Used      Cache   Noncache
firmware/hardware                 0          0          0
kernel image                      0          0          0
kernel dynamic memory          3.0G       2.8G     214.6M
userspace memory             217.7M      96.6M     121.1M
free memory                   12.4G      12.4G          0
----------------------------------------------------------
                              15.6G      15.3G     335.6M
Thu Sep 7 12:04:21 PDT 2023

So, we do see that stopping k3s.service does free (some of) the kernel memory.

[brandond]

You might poke at the killall script, and disable by turns the bits that kill pods / unmount filesystems / clean up iptables rules, which one makes the most different in freeing memory.

[nshores]

Will do - But it looks like that (Running k3s-killall) only freed an additional 11MB of memory - stopping k3s.service seems to have freed the majority of the kernel memory (182mb) - this is what leads me to believe one of the components k3s runs is doing something that is using kernel memory, and never releasing it.

[brandond]

That's not really how things work though; userspace processes may leak their own memory, but one they are terminated the kernel is responsible for freeing all memory that they have allocated. If memory remains used, either there is a kernel bug, or the process is creating things (like iptables rules, namespaces, control groups, and so on) that exist outside the process memory space. If it is the latter, then it's just a matter of identifying what that is and whether or not it's something that we should be responsible for cleaning up.

[nshores]

After a little more troubleshooting and re-testing, I'd like to call out a change in my original post to clarify that, Stopping k3s.service and running k3s-killall.sh does clear all the memory that's in use, and makes my available amount go to what I expect - a reboot is not required.

[brandond]

sounds like its probably coming from iptables rules or overlayfs stuff then. You might try using this: https://docs.k3s.io/advanced?_highlight=prefer-bundled-bin#old-iptables-versions

[nshores]

I can confirm that i'm not being affected by the iptables bug (I'm on 1.8.7) - Can you give me some more information on the overlayfs problems? I'm aware of overlayfs growing in size in the land of docker, but I haven't seen anything about this in terms of kubernetes + containerd.

[brandond]

I'm not aware of any issues, but it seems like something that might consume some amount of memory for the kernel to track.

You might try the bundled binaries anyways, just to see if has any effect.

You're poking at stuff that is much lower level than I usually care to investigate, as long as K3s works I don't go poking into what the kernel is doing to make the underlying bits work, or how many resources that consumes.

[nshores]

Update on this issue, It's completely unrelated to K3S. Very hard to track down, but it's related to using ESXI to pass-through a IGPU on a Intel NUC --

https://www.reddit.com/r/Ubuntu/comments/kwyhr0/comment/jcyjj11/?utm_source=share&utm_medium=web2x&context=3

Disabling the integrated graphics card with the flag mentioned in the post above stops the mysterious memory leak.