[SCALING] weave-net pod CrashLoopBackOff #3593
Comments
|
Increasing memory requests and limits (from 200Mi in kops manifest) prevents pods from crashing. Interesting to note memory profile shows below 50MB consumption |
It can be hard to catch the heap at its largest. If you set |
|
Thanks @bboreham for the suggestion. With |
|
So that says 50MB was the real minimum and GC allowed the heap to grow to about 100MB before collecting. |
|
Should have mentioned it. This sample was taken when I left kops weave manfiest defaults unmodified resources:
requests:
cpu: 50m
memory: 200Mi
limits:
memory: 200Mi |
|
OK, it's plausible that overheads and fragmentation pushed it briefly beyond 200Mi. High memory usage in
|
Based on the memory profile of the weave container in our cluster (see #3659 (comment)), we've opted to increase the memory limit to 300MB. We're using Kops by the way, so the default is 200MB. To note, we never got to 100 nodes but we grew quickly from around 30 to 80 at which point lots of weave pods got OOM'd. |
|
@itskingori please share pprof heap output for any nodes where you observed 200MB was not sufficient enough for a cluster < 100 nodes. I did launch clusters with 100 nodes with kops numerous times for scaling tests, never faced any issue with defaults. Note that this issue is reported for cluster sizes > 150, beyond 150 nodes there is increase in memory requirement and 200M is too low resulting in OOM |
Unfortunately we haven't scaled again to 80 ... 👇
I ran this on a pod at 144MB right now and got weave-net-7mnzr.mem.zip ... |
Noted. Would node size cause some variance i.e. maybe because run many pods per node? |
@murali-reddy Just nudging you on this. I'm wondering if node-type and number of pods on the node would introduce some variance to your tests? We're running 30GB/60GB nodes and we have quite a number of pods running on them. I woke up to our production cluster at 64 nodes and you can see the memory edging towards 200MB (which is why we increased to 300MB). It didn't get to 80 which 💣-ed us last time (see #3659 (comment)). I think at 80 we’d be at if not more than 200MB.
|
|
There's no particular reason for it to be sensitive to node size. Also please state the version of Weave Net in use which is essential to interpret the profile - even better please open a new issue which will prompt you to enter such information. |
|
@bboreham Noted. That said, I've spotted a rogue weave pod (see cc: @murali-reddy
The version: /home/weave # ./weave --local version
weave 2.5.1 |
|
@bboreham This might be interesting ... looking at the logs you can see the same pod has a higher occurrence of these errors ...
Something definitely is wrong with |
|
Colleague of @itskingori here. 👋Thought I'd share this set of logs too as it's related to many of the other issues that have been popping up:
Adding the latest heap again: weave-net-7mnzr.mem.zip |
|
@bboreham We've tried to get as much information about the state as we can so that you have as much information as you can to do on ... Here the latest memory dump just in case: weave-net-7mnzr.mem.zip. We're up to 216MB ...
|
|
thanks @itskingori and @zacblazic for the data I am suspecting there is mismatch between external view of memory consumption of the process RSS/resident size and what go profiler view of heap. Even with the new pprof heap output its only consuming 38MB. I am no expert on GO GC, but setting GOGC to lesser value from default might help. Need to check if that helps. Also going by the memory usage growth of |
Yes indeed. But it gives us time to spot it and remedy. I discovered this at 201MB growing slowly, so with the old limit it would have gotten OOM'd already. It also seems that in growing memory is a symptom and not the cause in that when a weave pod is in this state, it's memory keeps growing and it's cpu also increases just a little bit. Also, to note, I should have pointed that the peer-list seems to have a missing node i.e. it has 41 entries and the rest have 42. Don't know if that means anything but I suspect the counts should be the same. |
|
There were many inefficiencies fixed in 2.6, and also a longstanding leak fixed in 2.6.5, so I'll close this. |
What you expected to happen?
On empty cluster with no workload traffic (dataplane traffic) with just Weave-net control plane traffic weave-net pods should scale to 100's and even thousands of nodes
What happened?
While running scaling tests one of the symptom noticed was some of the weave-net pods go into CrashLoopBackOff. This can happen on any cluster beyond 100 nodes (CONN_LIMIT set to larger value than default 100)
Some pods were explicitly marked as OOM killed will open separate issue for that.
How to reproduce it?
Launch cluster with more than 100 nodes with CONN_LIMIT set accordingly)
Anything else we need to know?
Versions:
Logs:
Weave net logs indicate no error there is abrupt
KilledmessageKubelet logs (and pod describe) does not indicate reason
Memory and CPU profiles
mem-profile-150nodes.pdf
cpu-profile-150nodes.pdf
The text was updated successfully, but these errors were encountered: