resource labels missing in logs, post migrate to otel collector 0.73

[kadhamecha-conga]

Component(s)

exporter/loki

What happened?

hi team,

we facing issue when we upgrade from otel 0.47 to 0.73. all config we make compatible with 0.73.
we have certain labels such as service.name and service.namespace attached to pods and we set resource attributes by fetching it from pod .
metrics and traces are having these labels . but same labels are missing for logs.

can you please help here ?

thanks.

Collector version

0.73

Environment information

Environment

kubernetes

OpenTelemetry Collector configuration

resource:
        attributes:
        - action: insert
          key: k8s.cluster.name
          value: test-cluster
        - action: insert
          key: deployment.environment
          value: env1
        - action: insert
          from_attribute: k8s.pod.labels.service.namespace
          key: service.namespace
        - action: insert
          from_attribute: k8s.pod.name
          key: service.instance.id
        - action: upsert
          from_attribute: k8s.pod.labels.service.name
          key: service.name

    service:
      pipelines:
        logs:
          exporters:
          - otlphttp
          processors:
          - memory_limiter
          - batch
          - resource
          receivers:
          - filelog
        metrics:
          exporters:
          - otlphttp/metrics
          processors:
          - memory_limiter
          - batch
          - resource
          receivers:
          - prometheus
          - otlp
        traces:
          exporters:
          - otlphttp
          processors:
          - memory_limiter
          - batch
          - resource
          receivers:
          - otlp

Log output

No response

Additional context

No response

[github-actions]

Pinging code owners:

• exporter/loki: @gramidt @gouthamve @jpkrohling @mar4uk

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[mar4uk]

Could you please share the full config? Do you have hints configured?

[kadhamecha-conga]

hi @mar4uk ,

we haven't configured hints here.

full config:

    exporters:
      logging: {}
      otlphttp:
        endpoint: test-endpoint.test.com
      otlphttp/metrics:
        endpoint: test-endpoint.test.com
        retry_on_failure:
          initial_interval: 1s
          max_elapsed_time: 60s
          max_interval: 10s
        sending_queue:
          num_consumers: 1
          queue_size: 500
        timeout: 5s
    processors:
      batch:
        timeout: 10s
      k8sattributes:
        extract:
          labels:
          - key: app
          - key: release
          - from: pod
            key: service.namespace
            tag_name: service.namespace
          metadata:
          - k8s.pod.name
          - k8s.node.name
          - k8s.namespace.name
          - k8s.deployment.name
          - k8s.pod.start_time
        filter:
          node_from_env_var: KUBE_NODE_NAME
      k8sattributes/pod_uid:
        extract:
          labels:
          - key: app
          - key: release
          - from: pod
            key: service.namespace
            tag_name: service.namespace
          metadata:
          - k8s.pod.name
          - k8s.node.name
          - k8s.namespace.name
          - k8s.deployment.name
          - k8s.pod.start_time
        filter:
          node_from_env_var: KUBE_NODE_NAME
        pod_association:
        - from: resource_attribute
          name: k8s.pod.uid
      memory_limiter:
        check_interval: 5s
        limit_mib: 1638
        limit_percentage: 80
        spike_limit_mib: 512
        spike_limit_percentage: 25
      resource:
        attributes:
        - action: insert
          key: k8s.cluster.name
          value: test-cluster
        - action: insert
          key: deployment.environment
          value: env1
        - action: insert
          from_attribute: k8s.pod.labels.service.namespace
          key: service.namespace
        - action: insert
          from_attribute: k8s.pod.name
          key: service.instance.id
        - action: upsert
          from_attribute: k8s.deployment.name
          key: service.name
        - action: upsert
          from_attribute: k8s.pod.labels.service.name
          key: service.name
    receivers:
      filelog:
        include:
        - /var/log/pods/*/*/*.log
        include_file_name: false
        include_file_path: true
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318
    service:
      pipelines:
        logs:
          exporters:
          - otlphttp
          processors:
          - memory_limiter
          - k8sattributes/pod_uid
          - batch
          - resource
          receivers:
          - filelog
          - otlp
        metrics:
          exporters:
          - otlphttp/metrics
          processors:
          - memory_limiter
          - k8sattributes
          - batch
          - resource
          receivers:
          - prometheus
          - otlp
        traces:
          exporters:
          - otlphttp
          processors:
          - memory_limiter
          - k8sattributes
          - batch
          - resource
          receivers:
          - otlp

let me know, if you need more info.

[mar4uk]

Thank you for sending the config. To create stream labels in Loki you need to configure hints in the collector config using resource attributes processor (refer to the loki exporter readme "Labels" section).

Or, if you don't want those resource attributes to be indexed labels, you can use a parser (| logfmt or | json depending on the format of logs) in Loki query line. It will derive fields from the body and you will be able to query it like:

| json | resources_deployment_environment=`env1`

[kadhamecha-conga]

hi @mar4uk ,

how can we set logs labels using hints? can you give us example?

when we add below config in resource processor , it is setting loki.resource.labels : service.name in log resource attributes.

resource:
attributes:
- action: insert
key: loki.resource.labels
value: service.name

we need service.name as resource attributes with value from pod label like service.name = example-app in logs , can you please guide here ?

thanks.

[mar4uk]

The config looks correct, it should add label service_name = example-app in Loki.

it is setting loki.resource.labels : service.name in log resource attributes

This is correct.

Could you please share what your log record looks like in Loki?

[kadhamecha-conga]

log record looks like with resource attributes as

[kadhamecha-conga]

hi @mar4uk ,

is below thing is not supported in 0.73 otel version for logs , to insert it in resource attributes?

        - action: insert
          from_attribute: k8s.pod.labels.service.namespace
          key: service.namespace
        - action: insert
          from_attribute: k8s.pod.labels.service.name
          key: service.name

[mar4uk]

hmm.. I don't see k8s.pod.labels.service.namespace and k8s.pod.labels.service.name resource attributes in the example of loki log record you sent earlier.
They are missing, I think that is the reason why service.namespace and service.name resource attributes are not set. Maybe k8sattributesprocessor had some changes?

Could you please add a logging exporter with a verbosity detailed option? and send the log record you see in the collector stdout?

exporters:
   logging:
      verbosity: detailed

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• exporter/loki: @gramidt @gouthamve @jpkrohling @mar4uk

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[jpkrohling]

Please feel free to reopen if there's still something to be addressed here.