Add complete k8s metadata through composable provider

[ChrsMark]

What does this PR do?

Add all k8s metadata via the provider.

Why is it important?

To support full metadata access like in Beats.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

TODOs

• Verify that short-living pods are handled properly.

How to test this PR locally

1. Enable a static autodiscovery rule in Agent's config:

providers.kubernetes:
  scope: node
  kube_config: /Users/chrismark/.kube/config
  node: "kind-control-plane"
  cleanup_timeout: 360s
  resources:
    pod:
      enabled: true

inputs:
- name: redis
  type: redis/metrics
  use_output: default
  meta:
    package:
      name: redis
      version: 0.3.6
  data_stream:
    namespace: default
  streams:
    - data_stream:
        dataset: redis.info
        type: metrics
      metricsets:
        - info
      hosts:
        - '${kubernetes.pod.ip}:${kubernetes.container.port}'
      idle_timeout: 20s
      maxconn: 10
      network: tcp
      period: 10s
      condition: ${kubernetes.labels.app} == 'redis' AND ${kubernetes.container.port_name} == 'web'

2. Deploy a Redis pod that triggers the condition:

apiVersion: v1
kind: Pod
metadata:
  name: redis
  labels:
    role: main
    app: redis
  annotations:
    # this had no meaning
    co.elastic.metrics.redis.XYZ/module: redis
spec:
  hostNetwork: true
  dnsPolicy: ClusterFirstWithHostNet
  containers:
    - name: redis
      image: redis:latest
      ports:
        - name: web
          containerPort: 6379
          protocol: TCP

3. Run agent's inspect command to verify if an input is created ./elastic-agent -c ./elastic-agent.yml inspect output -o default
4. Verify that an input is created and metadata are added properly via the processors, for example:

metricbeat:
  modules:
  - hosts:
    - 172.18.0.3:6379
    idle_timeout: 20s
    index: metrics-redis.info-default
    maxconn: 10
    meta:
      package:
        name: redis
        version: 0.3.6
    metricsets:
    - info
    module: redis
    name: redis
    network: tcp
    period: 10s
    processors:
    - add_fields:
        fields:
          id: cda93b8385201eaf725532e908e4a4c55895273d2a2cb57700b3cf48732b2de3
          image:
            name: redis:latest
          runtime: containerd
        target: container
    - add_fields:
        fields:
          container:
            name: redis
          labels:
            app: redis
            role: main
          namespace: default
          node:
            name: kind-control-plane
          pod:
            ip: 172.18.0.3
            name: redis
            uid: 365e8d21-a81d-44e2-acbf-95deb8a57730
        target: kubernetes
    - add_fields:
        fields:
          cluster:
            name: kind-kind
            url: https://127.0.0.1:50740
        target: orchestrator
    - add_fields:
        fields:
          dataset: redis.info
          namespace: default
          type: metrics
        target: data_stream
    - add_fields:
        fields:
          dataset: redis.info
        target: event
    - add_fields:
        fields:
          id: 32806c0a-4f04-499c-9427-e6e24e5f6035
          snapshot: false
          version: 8.0.0
        target: elastic_agent
    - add_fields:
        fields:
          id: 32806c0a-4f04-499c-9427-e6e24e5f6035
        target: agent

Using annotations (Note: annotations are not exposed to meta fields by default but are exposed in variable resolution mechanism)

Specify a condition based on an annotation like condition: ${kubernetes.annotations.level} == 'production' AND ${kubernetes.container.port_name} == 'web' and verify that input is created again. (note: tune Redis Pod manifest accordingly)

Short-living pods

Define a log input with condition based on container's name:

      - name: container-log
        type: logfile
        use_output: default
        meta:
          package:
            name: log
            version: 0.4.6
        data_stream:
          namespace: default
        streams:
          - data_stream:
              dataset: generic
            symlinks: true
            condition: ${kubernetes.container.name} == 'hello'
            paths:
              - /var/log/containers/*${kubernetes.container.id}.log

Deploy elastic-agent and verify that it collects logs for the following short-living pod:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: mytarge
  labels:
    app: mytarget
spec:
  schedule: "*/1 * * * *"
  failedJobsHistoryLimit: 10
  successfulJobsHistoryLimit: 20
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: hello
            image: busybox
            args:
            - /bin/sh
            - -c
            - date; echo Hello from the Kubernetes cluster
          restartPolicy: OnFailure

Short living Init containers are covered

---
apiVersion: v1
kind: Pod
metadata:
  name: mytarget
  labels:
    app: test
spec:
  initContainers:
    - name: test-init
      image: ubuntu:latest
      command:
        - bash
        - -c
        - |
          #!/bin/bash
          echo "$(date): started the INIT process"
          sleep 10
          echo "$(date): sleeping 10 INIT seconds"
  containers:
    - name: test
      image: ubuntu:latest
      command:
        - bash
        - -c
        - |
          #!/bin/bash
          echo "$(date): started the process"

          while :
          do
                 echo "$(date): sleeping 5 seconds"
                 sleep 5
          done

Ensure logs are captured even if Pod is marked for deletion

---
apiVersion: v1
kind: Pod
metadata:
  name: mytarget2
  labels:
    app: test
spec:
  containers:
    - name: test
      image: ubuntu:latest
      command:
        - bash
        - -c
        - |
          #!/bin/bash
          trap "{ echo '$(date): SIGTERM triggered'; sleep 3;echo '$(date): Bye bye'; sleep 5; echo '$(date): bye'; sleep 5; exit 1; }" SIGINT SIGTERM
          echo "$(date): started the process"

          while :
          do
                 echo "$(date): sleeping 5 seconds"
                 sleep 5
          done

Related issues

• Closes Add complete k8s metadata through composable provider #24495, [Agent] Support labels dedot in k8s provider #27019

Notes for the reviewer

Most of the functionality is ported from libbeat/autodiscover/providers/kubernetes and libbeat/common/kubernetes/metadata.

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2021-09-20T08:56:23.919+0000

• Duration: 144 min 25 sec

• Commit: e702b31

Test stats 🧪

Test	Results
Failed	0
Passed	54015
Skipped	5327
Total	59342

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	54015
Skipped	5327
Total	59342

[ChrsMark]

/test

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[ChrsMark]

I'm opening this for an early review. I will need to add/tune tests as well as do extensive manual testing, however an early review would be more than welcome so as to spot any possible foundational issues early on.

[ChrsMark]

Heads-up on this:

Unit tests were tuned accordingly and I'm running several manual testing scenarios to verify nothing is broken. I'm planning to work on adding e2e tests right after this one is merged (issue: elastic/e2e-testing#1090)

I have particularly tried to "port" the logic from the old autodiscovery feature to this one and try to cover cases like short-living jobs etc.

Review parts:

• @jsoriano it would be super helpful if you could review the discovery lifecycle start/stop events , proper termination etc since you had worked on this recently :)
• @blakerouse could you review this patch from the Agent's framework perspective?
• @exekias one more review since you initially shipped the k8s provider :) ?
• @MichaelKatsoulis the provider will not add kubernetes.container.image but you can verify the ECS parts, and feel free to review the patch in general.

Maybe in the same release we can tune the metadata part (in follow-ups) in order to be aligned with the outcomes of #13911 and more specifically #16483 and #14875

[blakerouse]

Can this just be the default? Do we need this configurable now?

[ChrsMark]

I'm not sure about the history of these settings. Maybe some users had requests for these? @jsoriano @exekias do you think we could remove those settings and have dedoting as an always-on feature?

[blakerouse]

If undocumented and deprecated, why add it? Being this is all new to Elastic Agent, it could be the time to remove it.

[ChrsMark]

You are right this one is not exposed so it should be removed.

[ChrsMark]

Actually this touches code that is used by beats too. I will remove it in follow up PR target 8.0

[ChrsMark]

PR: #28006

[blakerouse]

Where is IncludeLabels and ExcludeLabels used? I was not able to find them in the diff.

[ChrsMark]

These are passed and used deeper in meta Generators, which is already existent codebase.

[jsoriano]

@jsoriano it would be super helpful if you could review the discovery lifecycle start/stop events , proper termination etc since you had worked on this recently :)

This part looks good (waiting for e2e tests for confirmation 🙂), added only some thoughts, nothing really blocking.

[jsoriano]

Nit.

Suggested change

[jsoriano]

Nit. Move beats imports to the last group of imports.

[jsoriano]

There are two places now where we collect all the containers in the pod and their statuses (the other here), this can be error-prone in future refactors. I wonder if this could be unified.

[ChrsMark]

Fair point. I will tune it! :)

[ChrsMark]

@jsoriano @blakerouse I tested this extensively (manually) and I think is good to go. Would you mind giving it another review?

Next steps will be to add 2e2 tests (in this iteration/release) and verify that we are aligned with #13911 (cc @MichaelKatsoulis ).

[ChrsMark]

/test

[blakerouse]

Overall this looks good to me.

I would prefer to see the dedot feature just set to default and the options to turn if off removed. But I will leave that up to you to determine.