New enhancment proposal: forward-to-Loki

Proposal to add loki forwarding to the ClusterLogForwarder.

enhancements/cluster-logging/forward-to-loki.md

@@ -0,0 +1,220 @@
+# Forward to Loki
+
+## Release Signoff Checklist
+
+- [X] Enhancement is `implementable`
+- [X] Design details are appropriately documented from clear requirements
+- [X] Test plan is defined
+- [ ] Operational readiness criteria is defined
+- [ ] Graduation criteria for dev preview, tech preview, GA
+- [ ] User-facing documentation is created in [openshift-docs](https://github.com/openshift/openshift-docs/)
+
+## Summary
+
+Add a new output type to the `ClusterLogForwarder` API to forward logs to a Loki instance.
+An important part of using Loki is choosing the right _Loki labels_ to define log streams.
+We present a default set of Loki labels, and explain how the choice of labels affect performance and queries.
+
+## Motivation
+
+Loki will become our default log store in the near future.
+There are also several use cases for forwarding to an external Loki instance.
+
+### Goals
+
+- Forward logs to a Loki instance
+- Choose a good default set of Loki labels.
+- Provide configuration to control the choice of Loki labels
+- Describe how expected query patterns correspond to Loki labels and JSON payload
+
+### Non-Goals
+
+The following may be covered in future proposals, but are out of scope here:
+
+- Query tools.
+- Optimizing/reducing the JSON payload format.
+- Alternate payload formats.
+- Use of collectors other than `fluentd` (e.g. promtail, fluentbit)
+
+## Proposal
+
+### Summary of Loki labels
+
+Loki and Elasticsearch take different approaches to indexing and search.
+Elasticsearch does content parsing and indexing _during ingest_ to optimize the _query_ path.
+Loki _optimizes ingest_ by using simpl key-value `labels` during ingest.
+Content parsing happens at _query_ time, to move work out of the ingest path.
+
+The benefit is faster ingest, which reduces the risk of dropping logs.
+The trade-off is that need to carefully choose what to use as labels.
+
+For more details:
+- [Guide to labels in loki](https://grafana.com/blog/2020/08/27/the-concise-guide-to-labels-in-loki/)
+- [Out-of-order streams issue]([https://github.com/grafana/loki/issues/1544)
+
+To summarize the key points:
+- Each unique combination of labels defines an ordered Loki _stream_.
+  - Too many streams degrades performance.
+  - _Cardinality_ refers to the number of unique label combinations.
+- The cardinality of `kubernetes.pod_name` was found to be
+  - too high in clusters with _very_ rapid turn-over of short-lived pods.
+  - acceptable in other (possibly more typical) clusters with high log data rates.
+- Minimize the _total number_ of labels per stream
+  - There is a limit of 30 max, fewer labels == better performance.
+- Log streams must be _ordered by collection timestamp_.
+  - Must not allow distinct nodes with separate clocks to contribute to a single stream.
+
+All logging meta-data will still be included in log records.
+Labels do not need to _identify the source of logs_, only to _partition the search space_.
+At query time labels reduce the search space, then Loki uses log content to complete the search.
+
+**Note**: Loki [label names][] must match the regex `[a-zA-Z_:][a-zA-Z0-9_:]*`.
+Meta-data names are translated to Loki labels by replacing illegal characters with '_'.
+
+Nested JSON objects are referenced by converting their JSON path to a label names.
+For example, if `kubernetes.namespace_name` is a label, then this JSON log record:
+
+``` json
+{"kubernetes.namespace_name":"foo","kubernetes":{"labels":{"foo":"bar"}}
+```
+would match this LogQL filter: 
+
+``` logql
+{ kubernetes_namespace_name="foo" } | json | kubernetes_labels_foo == "bar"
+```
+### Default Loki Labels
+
+The proposed default label set is:
+
+- `cluster`: for muti-cluster deployments (see Open Questions).
+- `kubernetes.host`: aka node, ensures ordered streams.
+- `kubernetes.namespace_name`: likely to be most common known factor in a search.
+- `kubernetes.container_name`: low cardinality, usually well-known
+
+`kubernetes.pod_name` is _not_ included. It is potentially high cardinality.
+`kubernetes.namespace_name` can be used to constrain the search for a named pod.
+
+The API below allows the user to add labels that are relevant for their application.
+
+### Proposed API
+
+Add a new `loki` output type to the `ClusterLogForwarder` API:
+
+``` yaml
+- name: myLokiOutput
+  type: loki
+  url: ...
+  secret: ...
+```
+
+`url` and `secret` have the usual meaning with regard to TLS and certificates.
+The `secret` may also contain `username` and `password` fields for Loki.
+
+The following optional fields are Loki-specific:
+
+- `labelKeys`: ([]string, optional)  A list of meta-data keys to add to the label set.\
+  Keys are translated to Loki labels: `kubernetes.labels.foo` => `kubernetes_labels_foo`,
+  both the dotted and underscored forms are allowed in the list.
+
+_**TODO**: Link to data model documentation for available meta-data keys._
+
+_**TODO**: Review Loki client configuration for any other fields._
+
+### User Stories
+
+#### See all logs from a namespace
+
+``` logql
+{ kubernetes_namespace_name="mynamespace"}
+```
+
+#### See logs from a specific container in a named Pod
+
+``` logql
+{ kubernetes_namespace_name="mynamespace", kubernetes_container_name="mycontainer" } |= kubernetes_pod_name == "mypod"
+```
+
+#### See logs from a labelled application
+
+Using the default configuration:
+
+``` logql
+{ } |= kubernetes_labels_app == "myapp""
+```
+
+By adding `labelKeys: [ kubernetes.labels.app ]` to the Loki output configuration this
+becomes a faster label query:
+
+``` logql
+{ kubernetes_labels_app="myapp" }
+```
+
+#### Example Queries From The Field
+
+These are real queries taken from use in the field and translated. 
+This deployment makes heavy use of the "app" and "deploymentconfig" k8s  labels, so
+we assume this configuration:
+
+``` yaml
+- name: myLokiOutput
+  type: loki
+  url: ...
+  secret: ...
+  labelKeys: [ kubernetes.labels.app, kubernetes.labels.deploymentconfig ]
+```
+
+``` logql
+{cluster="dsaas",kubernetes_labels_app="f8notification"} |= 'level:"error"'
+
+{cluster="dsaas",kubernetes_labels_app="keycloak"} |=  message:"*503 Service Unavailable*"
+
+{cluster="rh-idev", kubernetes_labels_deploymentconfig="bayesian-pgbouncer"} !~ “client close request|coreapi tls=no|server_lifetime|new connection to server|client unexpected eof|LOG Stats|server idle timeout|unclean server”
+
+{cluster="rh-idev", kubernetes_labels_deploymentconfig="wwwopenshifio"} |= message:"Connection reset by peer"
+```
+
+### Risks and Mitigations
+
+#### Cardinality explosions for large scale clusters
+
+_**TODO**: get numbers from DPTP experiment_
+
+## Design Details
+
+### Open Questions
+
+#### Pod Name
+
+Conflicting evidence for including this. Needs measurement of the impact of including it,
+and the query response time of find a pod using the namespace label.
+
+#### Cluster Name
+
+We need a name to identify the cluster, but k8s clusters lack a well-defined name.
+Options:
+- Authority part (host:port) of the API URL - easy to read, network-unique.
+- Cluster UID, hard to read, not user-friendly.
+- User defined name in ClusterLogForwarder config - explicit config not good for multi-cluster
+- What does DPTP/OSD use?
+
+[See this discussion](https://groups.google.com/g/kubernetes-sig-architecture/c/mVGobfD4TpY/m/nkdbkX1iBwAJ).
+
+### Test Plan
+
+- Unit and e2e tests
+- Measure throughput/latency on busy cluster, should exceed Elasticsearch.
+- Find breaking point for log loss in stress test, should exceed Elasticsearch.
+- Measure query response on large data, must be acceptable compared to Elasticsearch.
+
+### Graduation Criteria
+
+Acceptable performance & passing stress tests.
+
+### Upgrade / Downgrade Strategy
+
+_**TODO** Migrating from Elasticsearch_
+
+## Implementation History
+
+
+[label names]: https://prometheus.io/docs/concepts/data_model/#metric-names-and-labels