[Loki] Whitelisting automatic Kubernetes stream labels

[vladimirfx]

Is your feature request related to a problem? Please describe.

Loki strongly recommends using stable label values for stream labels. But automatic Kubernetes labels nearly always include some runtime hashes such as controller_revision_hash, pod_template_hash etc. That makes using of k8s auto labels fatal for Loki instance in long run because of infinitely growing label values.

Describe the solution you'd like
It would be nice to have functionality similar to record_modifier filter to include/exclude/whitelist stream labels.

[edsiper]

@vladimirfx what about the option remove_keys ?, it should do the job.

[vladimirfx]

@vladimirfx what about the option remove_keys ?, it should do the job.

Do you mean record_modifier filter? We are trying to implement this but this issue not about lossy modification of record but about filtering of Loki stream labels. Loki is very sensitive to stream labels especially in context of chunks ordering.

[edsiper]

Loki output plugin has a configuration property called remove_keys, e.g:

[OUTPUT]
    name        loki
    match       *
    remove_keys $key1, $key2, $key3['sub']['aaa']
    ...

[vladimirfx]

I see this in code but not in docs. Thanks I've try it!
This removes key from record or from stream labels?

[vladimirfx]

Unfortunately remove_keys not remove nested keys. Neither works:

Remove_keys $kubernetes['labels']['controller-revision-hash']

Remove_keys $kubernetes['labels']['controller_revision_hash']

Remove_keys kubernetes['labels']['controller_revision_hash']

Remove_keys kubernetes.labels.controller_revision_hash

Record:

{
"log":"2021-02-25T17:04:05.843785213Z stdout F 2021-02-25 17:04:05,838 INFO: no action.  i am the leader with the lock",
"kubernetes":{
"pod_name":"search-city-db-0",
"namespace_name":"default",
"pod_id":"67df6c12-053a-4268-b63b-fa00b304af25",
"labels":{
"application":"postgresql",
"cluster-name":"search-city-db",
"controller-revision-hash":"search-city-db-56bfc889bb",
"spilo-role":"master","statefulset.kubernetes.io/pod-name":"search-city-db-0"},
"annotations":{
"status":"{\"conn_url\":\"postgres://10.42.1.185:5432/postgres\",\"api_url\":\"http://10.42.1.185:8008/patroni\",\"state\":\"running\",\"role\":\"master\",\"version\":\"2.0.1\",\"xlog_location\":2415919104,\"timeline\":112}"},
"container_name":"postgres",
"docker_id":"c22b8b4bb4258f5a54aa7573aa9ac4bd055d47bf5d7d03c1c428f356d7c6acdf",
"container_hash":"registry.opensource.zalan.do/acid/spilo-13@sha256:843c659bc9c430caa81e1af1fb23c7287e0acb3c9c80288f656fcf2c82b49cc2",
"container_image":"registry.opensource.zalan.do/acid/spilo-13:2.0-p4"
}
}

[edsiper]

please share your full configuration.

[vladimirfx]

flbconf.zip

[vladimirfx]

Anyway if Remove_keys will work this particular issue is not about destructive change of record but about stream labels.

What do you think about Whitelist_label for effective stream labels list?

Whitelist_label app_kubernetes_io_name
Whitelist_label app_kubernetes_io_part_of
Whitelist_label namespace_name
Whitelist_label pod_name

If you ok I've try to implement it.

[vladimirfx]

Related Loki/Promtail ticket - grafana/loki#3390

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.

[vladimirfx]

Issue closed because no one can tell does this proposal makes sense or not. Not to implement it but just ok or nok.
That sad but the project looks semi-dead...