Bug 1881208: add alert for system.slice rss going over system-reserved #2033
Conversation
openshift-ci-robot
added
the
do-not-merge/work-in-progress
|
I think we want to get a number that accounts for pages that we aren't able to evict from cri-o on drop caches. Also, the alert could be at 90 - 95% of the reservation. |
| rules: | ||
| - alert: SystemMemoryExceedsReservation | ||
| expr: | | ||
| container_memory_rss{id="/system.slice"} > 1073741824 |
There was a problem hiding this comment.
Hrm, this needs to come from the reservation the system makes, which is exposed as a metric
There was a problem hiding this comment.
constant should be kube_node_status_capacity{... mem} - kube_node_status_allocatable{... mem}, and you need to figure out how to make that number show up in the message somehow
There was a problem hiding this comment.
is there anything on a node outside of system and kubepods? should the first value be (container_memory_rss{id="/"} - container_memory_rss{id="/kubepods.slice"}) > (capacity - allocatable)
There was a problem hiding this comment.
all the kernel threads are in /. no userspace process though.
sjenning
force-pushed
the
system-reservation-alert
branch
2 times, most recently
from
1cffb76
to
19b86d0
Compare
sjenning
changed the title
openshift-ci-robot
removed
the
do-not-merge/work-in-progress
sjenning
force-pushed
the
system-reservation-alert
branch
from
19b86d0
to
5ed6876
Compare
| labels: | ||
| severity: warning | ||
| annotations: | ||
| message: "System memory usage of {{ $value | humanize }} on {{ $labels.node }} exceeds 90% of the reservation" |
There was a problem hiding this comment.
Messages should be acitonable. So you need to recommend next steps. I would suggest playing around with something like the following additions:
Reserved memory ensures system processes can function even when the node is fully allocated and protects against workload out of memory events impacting the proper functioning of the node. The reservation may be increased (link to docs) when running nodes with high numbers of pods.
Or similar
There was a problem hiding this comment.
@smarterclayton better now?
sjenning
force-pushed
the
system-reservation-alert
branch
from
5ed6876
to
92fceb1
Compare
|
/retest |
1 similar comment
|
/retest |
|
/lgtm Attach the bug and this is gtg for 4.6 |
sjenning
changed the title
openshift-ci-robot
added
bugzilla/severity-high
|
@sjenning: This pull request references Bugzilla bug 1881208, which is invalid:
Comment In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
/bugzilla refresh |
openshift-ci-robot
added
the
bugzilla/valid-bug
|
@sjenning: This pull request references Bugzilla bug 1881208, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker. 3 validation(s) were run on this bug
In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
openshift-ci-robot
removed
the
bugzilla/invalid-bug
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: kikisdeliveryservice, sjenning, smarterclayton The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci-robot
added
the
approved
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
@sjenning: The following test failed, say
Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
@sjenning: All pull requests linked via external trackers have merged: Bugzilla bug 1881208 has been moved to the MODIFIED state. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
@smarterclayton @mrunalp @nee1esh @derekwaynecarr
xref https://issues.redhat.com/browse/OCPNODE-402