Skip to content

Latest commit

 

History

History
41 lines (33 loc) · 1.29 KB

README.md

File metadata and controls

41 lines (33 loc) · 1.29 KB

sysOOM

Send details to alertmanager when OOM event occurs

  "labels": {
        "memcg": "host",
        "process": "java",
        "hostname": "wiki2",
        "instance": "10.0.0.4",
        "severity": "warning",
        "alertname": "SysOOM",
        "process_rss": "165564kB",
        "process_list": "systemd-logind getty systemd-network systemd systemd-udevd java rsyslogd"
      }

Basic use

  • Start sysOOM, replace 10.0.0.3 with your own alertmanager ip

    docker run -p 5149:5149/udp --add-host alertmanager:10.0.0.3 -d --restart=always --name sysoom druggo/sysoom

  • Forward kernel log to sysOOM, replace 10.0.0.4 with sysOOM container's host ip and reload rsyslog

    echo ':syslogtag, startswith, "kernel" @10.0.0.4:5149' > /etc/rsyslog.d/41-kernel.conf

  • Test OOM

    systemd-run --user --scope -p MemoryMax=1K vi /var/log/syslog

Suppress node-exporter's NodeOOM

  • Ensure the syslog hostname matches the node-exporter hostname label

  • Add inhibit_rules to alertmanager.yml so that NodeOOM alert will supressed when SysOOM is fired

    - "equal":
  - "hostname"
  - "severity"
  "source_matchers":
  - "alertname = SysOOM"
  "target_matchers":
  - "alertname = NodeOOM"