[operator] Add support for discover_service_name configuration override in LokiStack CR

[jasaltvik]

Is your feature request related to a problem? Please describe.
We are in the process of migrating from Elasticsearch to Loki in OpenShift Logging (using the Loki Operator). Initially, the discover_service_name configuration option was set by default, but it was eventually disabled as none of the labels were set to populate the service_name correctly (causing it to be set to unknown_service, which is not helpful). However, we have been overriding the application label to ensure the service_name have been detected correctly. In addition, when using Grafana 11.3 and up, the Explore Logs app is installed by default, which depends on the service name being set to be useful. We would really appreciate a way for us to enable the service name detection again so that we can create a better default log experience for the developers using Loki via Grafana.

Describe the solution you'd like
It would be nice to be able to override this via a parameter in the LokiStack CR in some way.

Describe alternatives you've considered
I'm not sure if it is possible to override this in any way when using the operator. I see that there is a CLI flag that potentially could be used. Appreciate any input on this.

Additional context

• PR for disabling service name discovery: fix(operator): Disable automatic discovery of service name #14506
• Red Hat Jira issues for disabling:
  • https://issues.redhat.com/browse/LOG-6254
  • https://issues.redhat.com/browse/LOG-6256
  • https://issues.redhat.com/browse/LOG-6295
• Grafana 11.3 changelog (explore logs app): https://grafana.com/docs/grafana/latest/whatsnew/whats-new-in-v11-3/#the-explore-logs-plugin-is-installed-by-default
• Limits config seems to include the override: https://grafana.com/docs/loki/latest/configure/#limits_config

# If no service_name label exists, Loki maps a single label from the configured
# list to service_name. If none of the configured labels exist in the stream,
# label is set to unknown_service. Empty list disables setting the label.
# CLI flag: -validation.discover-service-name
[discover_service_name: <list of strings> | default = [service app application app_name name app_kubernetes_io_name container container_name k8s_container_name component workload job k8s_job_name]]

[JStickler]

This is already possible and documented in the recently updated Labels topic.

"The default list of resource attributes to store as labels can be configured using default_resource_attributes_as_index_labels under the distributor’s otlp_config. You can set global limits using limits_config.otlp_config."

[jasaltvik]

This is already possible and documented in the recently updated Labels topic.

"The default list of resource attributes to store as labels can be configured using default_resource_attributes_as_index_labels under the distributor’s otlp_config. You can set global limits using limits_config.otlp_config."

@JStickler Thanks for the answer. That's interesting, I totally missed these docs. In this case we're using Vector for log shipping. So if I understand this correctly, we could achieve this with https://loki-operator.dev/docs/api.md/#loki-grafana-com-v1-OTLPSpec? I was also looking at https://grafana.com/docs/loki/latest/get-started/labels/#default-labels-for-all-users. Could I also inject service_name as a Loki label in the Vector config, and it would be used automatically?

Typically adding something like

[sinks.output_loki.labels]
service_name = "{{kubernetes.labels.app_kubernetes_io_name}}"

We were previously using application for this when discover_service_name was enabled

[sinks.output_loki.labels]
application = "{{kubernetes.labels.app_kubernetes_io_name}}"

Thanks for any feedback on this.

[jasaltvik]

This is already possible and documented in the recently updated Labels topic.
"The default list of resource attributes to store as labels can be configured using default_resource_attributes_as_index_labels under the distributor’s otlp_config. You can set global limits using limits_config.otlp_config."

@JStickler Thanks for the answer. That's interesting, I totally missed these docs. In this case we're using Vector for log shipping. So if I understand this correctly, we could achieve this with https://loki-operator.dev/docs/api.md/#loki-grafana-com-v1-OTLPSpec? I was also looking at https://grafana.com/docs/loki/latest/get-started/labels/#default-labels-for-all-users. Could I also inject service_name as a Loki label in the Vector config, and it would be used automatically?

Typically adding something like

[sinks.output_loki.labels]
service_name = "{{kubernetes.labels.app_kubernetes_io_name}}"

We were previously using application for this when discover_service_name was enabled

[sinks.output_loki.labels]
application = "{{kubernetes.labels.app_kubernetes_io_name}}"

Thanks for any feedback on this.

Ok, so I just tried adding

[sinks.output_loki.labels]
service_name = "{{kubernetes.labels.app_kubernetes_io_name}}"

as a an extra label in the Vector config, and that works. Not sure how the OTLP stuff relates to this. Anyways, thanks for the feedback @JStickler. Closing

[JStickler]

@jasaltvik, not surprised you missed the docs, it was a recent update. Glad I could help!