Kubernetes logs are not enriched with namespace annotations via add_resource_metadata

[mantipin]

• Version: 7.11.0
• Operating System: Ubuntu
• Discuss Forum URL: https://discuss.elastic.co/t/add-resource-metadata-namespace-include-annotations-doesnt-work-anymore-in-7-10-x/261878
• Steps to Reproduce:
  Hello, trying to enrich my logs with namespace annotations, mentioned here - https://www.elastic.co/guide/en/beats/filebeat/current/configuration-autodiscover-hints.html#_namespace_defaults
  And nothing happens - no kind of annotations from namespace.

Config:

    name: ${NODE_NAME}
    filebeat.registry.flush: 5s
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          hints.enabled: true
          add_resource_metadata:
            namespace:
              include_annotations: ["testing"]
    output.file:
        path: "/tmp/filebeat"
        filename: filebeat
    http.enabled: true
    http.port: 5066

gives us:

{
   "@timestamp":"2021-03-02T10:30:51.913Z",
   "@metadata":{
      "beat":"filebeat",
      "type":"_doc",
      "version":"7.11.0"
   },
   "log":{
      "offset":366702,
      "file":{
         "path":"/var/lib/docker/containers/dd4491afb59cec4c8839bae9e8841750137bc7803c5158cefdd708df144ff4b7/dd4491afb59cec4c8839bae9e8841750137bc7803c5158cefdd708df144ff4b7-json.log"
      }
   },
   "stream":"stderr",
   "message":"2021-03-02 10:30:51.913173 I | etcdserver/api/etcdhttp: /health OK (status code 200)",
   "input":{
      "type":"container"
   },
   "kubernetes":{
      "namespace":"kube-system",
      "labels":{
         "tier":"control-plane",
         "component":"etcd"
      },
      "container":{
         "name":"etcd",
         "image":"k8s.gcr.io/etcd:3.4.13-0"
      },
      "node":{
         "name":"minikube"
      },
      "namespace_uid":"deb1a47a-6931-45cb-83d3-5945db682cb1",
      "namespace_labels":{
         "blabla":"blabla"
      },
      "pod":{
         "name":"etcd-minikube",
         "uid":"fe041f6c-9a0b-4111-871f-df74feff5e91"
      }
   },
   "container":{
      "image":{
         "name":"k8s.gcr.io/etcd:3.4.13-0"
      },
      "runtime":"docker",
      "id":"dd4491afb59cec4c8839bae9e8841750137bc7803c5158cefdd708df144ff4b7"
   },
   "ecs":{
      "version":"1.6.0"
   },
   "host":{
      "name":"minikube"
   },
   "agent":{
      "name":"minikube",
      "type":"filebeat",
      "version":"7.11.0",
      "hostname":"filebeat-kclr9",
      "ephemeral_id":"88808314-092b-4e1c-81fc-b45c120189ca",
      "id":"23eb3d8e-4828-4c79-b48b-69de731520df"
   }
}

namespace's yaml:

apiVersion: v1
kind: Namespace
metadata:
  annotations:
    testing: testing
  creationTimestamp: "2021-02-10T07:55:52Z"
  labels:
    blabla: blabla
  name: kube-system
  resourceVersion: "21374"
  uid: deb1a47a-6931-45cb-83d3-5945db682cb1
spec:
  finalizers:
  - kubernetes
status:
  phase: Active

Annotations from namespaces worked last time on 7.6.0 version, as I tested.

[mantipin]

And, by the way, there is even no kubernetes.node.hostname, mentioned here - #22189

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[tetianakravchenko]

hi @mantipin,
this issue should be already fixed by #27045 and should be available in 7.15

[tetianakravchenko]

I will close this issue as it should be fixed, feel free to reopen it if needed.