spike: log aggregation and JSON logs

[adamdecaf]

What were you trying to do?

Log aggregation is the process of collecting plaintext (or structured) logs from applications, databases, etc and capturing them into a unified system. There are options available (ELK - elasticsearch, logstash, kafka/kibana, oklog (recently deprecated)).

Aggregated logs are much easier to reason about and allows further automation built on logs.

What other options are available?

[adamdecaf]

FYI, I've been working on adding loki which is a super new, experimental log aggregator. We'll see how it turns out in a few months.

Configs:

• https://github.com/moov-io/infra/blob/master/lib/infra/19-loki.yml
• https://github.com/moov-io/infra/blob/master/lib/infra/20-promtail.yml

Docs / Notes

• https://github.com/grafana/loki/blob/master/docs/operations.md

Grfana dashboard:

• https://infra.moov.io/grafana/d/GqCm-ksiz/loki-and-pomtail?refresh=1m&orgId=1&from=now-24h&to=now

A cool thing is their logcli, which lets you grep logs from the cluster (infra.moov.io/loki) locally on a machine.

$ logcli --addr https://infra.moov.io/loki query '{job != "" }' 

2018-12-17T19:25:48Z {__filename__="/var/log/pods/b0ef4470-022f-11e9-82a3-42010a8000cf/loki/0.log", app="loki", instance="loki-65645bfc9f-pnb6t", job="infra/loki", namespace="infra", pod_template_hash="2120169759", stream="stderr"} level=debug ts=2018-12-17T19:25:48.652599602Z caller=gokit.go:29 msg="POST /api/prom/push (204) 4.185521ms"

2018-12-17T19:25:46Z {__filename__="/var/log/pods/cca368fe-01c8-11e9-9753-42010a80002c/nginx/0.log", app="grafana", instance="grafana-f85bbf798-zcdqv", job="infra/grafana", namespace="infra", pod_template_hash="941669354", stream="stdout"} 10.60.40.200 - - [17/Dec/2018:19:25:46 +0000] "GET /metrics HTTP/1.1" 200 13 "-" "Prometheus/2.6.0" "-"

$ logcli --addr https://infra.moov.io/loki query '{ job = "apps/auth" }'  af20742000b6a3b97faa7b418e0338389a96a3e3 

Common labels: {__filename__="/var/log/pods/72b6547e-0224-11e9-82a3-42010a8000cf/auth/0.log", app="auth", instance="auth-7c79cc7c49-xvgbf", job="apps/auth", namespace="apps", pod_template_hash="3735773705", stream="stderr"}

2018-12-17T19:33:12Z {}                                                                                                   ts=2018-12-17T19:33:12.96075701Z caller=http.go:164 checkAuth="method=GET, path=/auth/check, status=200, took=28.492348ms, requestId=af20742000b6a3b97faa7b418e0338389a96a3e3"

This is powerful for developers and something I wished was available from an ELK stack.

[adamdecaf]

Right now it's behind the oauth2-proxy auth and would need something else auth-wise. We'll likely have to add certificate based auth into it.

[adamdecaf]

If you're auth'd loading https://infra.moov.io/loki/api/prom/label works.

[adamdecaf]

Loki offers some dashboards and alerts we could pull in (with work from #22) in https://github.com/grafana/loki/tree/master/production/loki-mixin, but right I've got an open PR to fix the /metrics handler with a path prefix.