Touchup kettle readme, add docs/oss-oncall-log.md #8801
Conversation
k8s-ci-robot
added
area/kettle
approved
docs/oss-oncall-log.md
Outdated
|
|
||
| "Oh yeah, I remember having to deal with that a while back... what was it again?" | ||
|
|
||
| - Issue: https://github.com/kubernetes/test-infra/issues/8703 |
There was a problem hiding this comment.
Instead of just linking the issue here, can we give it a more descriptive link text eg: Velodrome is out-of-date?
There was a problem hiding this comment.
I feel like that's additional friction to what (after some thinking overnight) I am concerned is too manual a way to flag issues, but yeah let's give it a shot.
|
minor nit, otherwise lgtm. Feel free to remove hold anytime. /lgtm |
k8s-ci-robot
added
the
do-not-merge/hold
k8s-ci-robot
added
lgtm
kettle/README.md
Outdated
|
|
||
| ```sh | ||
| make get-cluster-credentials | ||
| kubectl logs $(kubectl get pods -o jsonpath='{.items[?(@.metadata.labels.app=="kettle")].metadata.name}') |
There was a problem hiding this comment.
kubectl logs -l app=kettle works
kettle/README.md
Outdated
| #### Replace pods | ||
|
|
||
| ```sh | ||
| kubectl delete pod/$( kubectl get pods -o jsonpath='{.items[?(@.metadata.labels.app=="kettle")].metadata.name}' ) |
There was a problem hiding this comment.
I'm assuming this should be kubectl delete pods -l app=kettle
kettle/README.md
Outdated
|
|
||
| If deployment.yaml changes, run 'make push deploy'. | ||
| - Occasionally data from Kettle stops updating, we suspect this is due to the pod running out of disk space ([#8800](https://github.com/kubernetes/test-infra/issues/8800)). If this happens, [restart kettle](#restarting) |
There was a problem hiding this comment.
Not running out of disk space-- we suspect this is from a transient hang when contacting GCS.
kettle/README.md
Outdated
| You can watch the pod startup and collect data from various GCS buckets by looking at its logs | ||
|
|
||
| ```sh | ||
| kubectl logs $(kubectl get pods -o jsonpath='{.items[?(@.metadata.labels.app=="kettle")].metadata.name}') |
There was a problem hiding this comment.
kubectl logs -f $(kubectl get pod -l app=kettle -oname)
kettle/README.md
Outdated
|
|
||
| buckets.yaml is automatically fetched from GitHub. | ||
| It might take a couple of hours to be fully functional and start updating BigQuery. You can always go back to [Gubernator BigQuery page](https://bigquery.cloud.google.com/table/k8s-gubernator:build.all?pli=1&tab=details) and check to see if data collection has resumed. Backfill should happen automatically. |
| - k8s-gubernator BigQuery table is out of date | ||
| - Remediation: | ||
| - Kettle froze and needed to be restarted | ||
| - See [Kettle's README](/kettle/README.md) |
There was a problem hiding this comment.
- added an alert on velodrome when no rows have been added to the DB in the last 6 hours http://velodrome.k8s.io/dashboard/db/bigquery-metrics?orgId=1&panelId=12&fullscreen
There was a problem hiding this comment.
I added a note to this firing under symptoms
Instead of dropping nits in another PR, I'm suggesting via my own - Trim kettle's readme down - Move restarting instructions into kettle's readme - Add known issues section to kettle's readme, link to root cause issue - Add docs/oss-oncall-log.md as an attempt to try logging a tl;dr of oss oncall issues we've had to deal with - Incorpoorates review feedback
spiffxp
force-pushed
the
touchup-kettle-troubleshooting
branch
from
2940dbe to
b725738
Compare
|
/lgtm |
k8s-ci-robot
added
the
lgtm
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: AishSundar, rmmh, spiffxp The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
/hold cancel |
k8s-ci-robot
removed
the
do-not-merge/hold
Instead of dropping nits in another PR, I'm suggesting via my own
of oss oncall issues we've had to deal with
/area kettle
/cc @AishSundar @mithrav