Add Loki log collection service to the monitoring stack

[tzach]

Loki is a Prometheus-inspired logging service, which works well with Grafana. It is still very new but can be a simple, out of the box solution for log collection.

[avikivity]

Before loki (which is alpha), we should add support for ordinary rsyslog. This provides a central place for logs instead of having to go to each server for its logs. rsyslog is available on all distributions and is integrated with syslog.

[tzach]

Loki is now in GA, and provide good integration with Grafana.

[amnonh]

@tzach @avikivity @slivne I would like to try and integrate loki,
it uses an agent that runs on the host and can read form journalctl
can we try to integrate it and see the effect?

[avikivity]

Why do people insist on agents?

Is there no way for loki to use the rsyslog protocol?

[amnonh]

Loki is a push-based server the agents push towards it.
I found an experimental adaptor for rsyslog that can push the logs into loki. It's un-official but we can give it a try.

[avikivity]

Please do, and open an issue for them to interoperate with rsyslog.

[amnonh]

it's the other way around, Loki is the passive side here, it's up to rsyslog to be able to push to loki

[avikivity]

No, rsyslog sends the data in its own format. Loki should listen on the rsyslog socket and accept it.

rsyslog is about 7,812 years old, you can't expect it to send data on the loki protocol.

[amnonh]

it has an extension mechanism to support different targets