Libbeat add_kubernetes_metadata processor: Missing fields replicaset.name, etc. in fields.yml

[alakahakai]

Address issue #11133:

• Libbeat add_kubernetes_metadata processor: Add Kubernetes replica-set map to fields.yml

Now testing works

test_fileset_file_0_coredns (test_xpack_modules.XPackTest) ... ok
test_fileset_file_1_coredns (test_xpack_modules.XPackTest) ... ok

---

Ran 2 tests in 3.653s

OK

[ruflin]

You mention in #11133 that replicaset.name is missing but here you introduce an object replicaset. Why not directly introduce the field?

The other part which is not clear to me is why our tests on CI are not failing. Where do you have the message from that you posted in #11133

General note: If in your PR's you use the URL of issues / pr's instead of {pull}11134[11134] it will link them automatically together and will make it easier to navigate. Also you can use keywords in your PR's to close issues directly: https://help.github.com/en/articles/closing-issues-using-keywords

[alakahakai]

You mention in #11133 that replicaset.name is missing but here you introduce an object replicaset. Why not directly introduce the field?

There might be other fields that will be added by the processor. I am simply following the existing practice where it uses map for labels and other fields. See below:

        - name: labels
          type: object
          description: >
            Kubernetes labels map
        - name: annotations
          type: object
          description: >
            Kubernetes annotations map

Regarding why the CI test is not failing, I assume that it is not testing k8s deployment with anything or filebeat deployed as daemonset.

[alakahakai]

jenkins, test this

[ruflin]

lables and tags are special fields where the users add their own subfields and is why we have this kind of mapping. I don't this applies here as we are in control of the fields and should document each fields. Lets add specifically replicaset.name if we know this is missing.

In the original issue you posted:

raise Exception("Key '{}' found in event is not documented!".format(key))
Exception: Key 'kubernetes.replicaset.name' found in event is not documented!

So it seems somehow you got this error?

[alakahakai]

I can see this in libbeat/kibana/testdata/extensive/fields.yml:

        - name: replicaset
          type: group
          description: >
            kubernetes replica set metrics
          fields:
            - name: name
              type: keyword
              description: >
                Kubernetes replica set name
            - name: replicas
              type: group
              description: >
                Kubernetes replica set paused status
              fields:
                - name: available
                  type: long
                  description: >
                    The number of replicas per ReplicaSet
                - name: desired
                  type: long
                  description: >
                    The number of replicas per ReplicaSet
                - name: ready
                  type: long
                  description: >
                    The number of ready replicas per ReplicaSet
                - name: observed
                  type: long
                  description: >
                    The generation observed by the ReplicaSet controller
                - name: labeled
                  type: long
                  description: >
                    The number of fully labeled replicas per ReplicaSet

Why are they not included in all fields?

Since the field is not added by filebeat coredns module, we should find out whether there are more fields like this, so that the problem does not come back later when some other fields appear under replicaset.

[alakahakai]

Because of time constraints, I am going to just add the missing replicaset.name field for now. But it must be worth looking into why the replicaset fields are missing.

[ruflin]

jenkins, test this

[ruflin]

PR LGTM. Let's get it in as it already solves part of the problem.

Thanks for the investigation. The fields you linked above are from a test suite but someone must have copied them from somewhere where we potentially will need them. Could you open a follow up Github issue for this so we don't forget about tracking the missing fields?

[alakahakai]

@ruflin Sure, thanks

Opened issue #11222

[jsoriano]

Sorry, but I cannot follow 🤔

kubernetes.replicaset fields are not created by anything in libbeat, they are only created by kubernetes state_replicaset metricset, and they are documented there, why are we adding this to all beats?

[alakahakai]

It is added by the libbeat add_kubernetes_metadata processor. Please take a look at the issue #11133

[alakahakai]

It seems to be added here
Looks like we need to add statefulset.name and deployment.name as well. Also, metadata_test.go does not cover either one.

[ruflin]

LGTM.

@jsoriano Could you also have a quick look if that is what you expected?

[jsoriano]

It seems to be added here
Looks like we need to add statefulset.name and deployment.name as well. Also, metadata_test.go does not cover either one.

Oh I see, good catch.

But now these fields are defined in two places for metricbeat, should we remove them from the kubernetes metricsets?

[ruflin]

@jsoriano Looks like they are removed here? https://github.com/elastic/beats/pull/11134/files#diff-caf37f855a3380a66e838f4e45e7777dL7

[alakahakai]

@jsoriano Please let me know if the PR looks good to you. I do have another PR #11200 that are kinda blocked by this. Thanks!

[alakahakai]

I assume that this is fine to go now. Thanks.