Skip to content
This repository has been archived by the owner on Aug 31, 2022. It is now read-only.

Miss some Backoff events #115

Closed
sonots opened this issue Feb 26, 2021 · 0 comments · Fixed by #145
Closed

Miss some Backoff events #115

sonots opened this issue Feb 26, 2021 · 0 comments · Fixed by #145

Comments

@sonots
Copy link

sonots commented Feb 26, 2021

I am trying to capture some Backoff warning events like:

$ kubectl logs -f -n kube-support event-exporter-69645c7bb4-cmtjz | grep my-app | grep Warning | tee log.jsonl
$ cat log.jsonl | head -1 | jq .
{
  "metadata": {
    "name": "my-app-5fc464b465-t4z6r.1667424602513628",
    "namespace": "my-app",
    "selfLink": "/api/v1/namespaces/my-app/events/my-app-5fc464b465-t4z6r.1667424602513628",
    "uid": "48226d93-b2c7-4f26-8e7b-f60da9bfb335",
    "resourceVersion": "114058053",
    "creationTimestamp": "2021-02-26T09:30:59Z"
  },
  "reason": "BackOff",
  "message": "Back-off restarting failed container",
  "source": {
    "component": "kubelet",
    "host": "xxxxxx.ap-northeast-1.compute.internal"
  },
  "firstTimestamp": "2021-02-26T09:30:59Z",
  "lastTimestamp": "2021-02-26T09:30:59Z",
  "count": 1,
  "type": "Warning",
  "eventTime": null,
  "reportingComponent": "",
  "reportingInstance": "",
  "involvedObject": {
    "kind": "Pod",
    "namespace": "my-app",
    "name": "my-app-5fc464b465-t4z6r",
    "uid": "11f86797-ded9-4e1f-be1e-582b4e8ebde8",
    "apiVersion": "v1",
    "resourceVersion": "114058003",
    "fieldPath": "spec.containers{my-app}",
    "labels": {
      "app": "my-app",
      "env": "dev",
      "pod-template-hash": "5fc464b465",
      "run": "my-app"
    }
  }
}

However, I found that Backoff logs of count 4 to 10 were missing.

$ grep BackOff log.jsonl | jq '{count: .count, creationTimestamp: .metadata.creationTimestamp, firstTimestamp:  .firstTimestamp, lastTimestamp: .lastTimestamp}'
{
  "count": 1,
  "creationTimestamp": "2021-02-26T09:30:59Z",
  "firstTimestamp": "2021-02-26T09:30:59Z",
  "lastTimestamp": "2021-02-26T09:30:59Z"
}
{
  "count": 2,
  "creationTimestamp": "2021-02-26T09:30:59Z",
  "firstTimestamp": "2021-02-26T09:30:59Z",
  "lastTimestamp": "2021-02-26T09:31:00Z"
}
{
  "count": 3,
  "creationTimestamp": "2021-02-26T09:30:59Z",
  "firstTimestamp": "2021-02-26T09:30:59Z",
  "lastTimestamp": "2021-02-26T09:31:01Z"
}
{
  "count": 11,
  "creationTimestamp": "2021-02-26T09:36:05Z",
  "firstTimestamp": "2021-02-26T09:33:54Z",
  "lastTimestamp": "2021-02-26T09:36:05Z"
}

Any clues on this issue? Or, is this expected behavior of kubernetes-event-exporter?

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
1 participant