Fix log collection with hints when missing container ports or annotations #2386

ChrsMark · 2023-03-22T07:10:21Z

What does this PR do?

This PR enhances hints mechanism for logging use cases by adding the following:

emit mapping for all of the containers that belong to a discovered Pod and not only those that have defined Ports. This is needed so as to ensure that logs from all containers will be collected.
if no hints are detected then emit a generic_logs mapping so as to start collecting logs for all the discovered containers even if those are not annotated properly. This can be disabled with hints.default_container_logs: false

How to test this PR manaully

Multiple containers case

Define the following Agent config:

providers:
  kubernetes:
    kube_config: /home/chrismark/.kube/config
    node: "kind-control-plane"
    hints.enabled: true

inputs:
  - name: filestream-redis
    type: filestream
    use_output: default
    streams:
      - condition: ${kubernetes.hints.redis.log.enabled} == true or ${kubernetes.hints.redis.enabled} == true
        data_stream:
          dataset: redis.log
          type: logs
        exclude_files:
          - .gz$
        exclude_lines:
          - ^\s+[\-`('.|_]
        parsers:
          - container:
              format: auto
              stream: ${kubernetes.hints.redis.log.stream|'all'}
        paths:
          - /var/log/containers/*${kubernetes.hints.container_id}.log
        prospector:
          scanner:
            symlinks: true
        tags:
          - redis-log

Run a target pod with hints annotations:

apiVersion: v1
kind: Pod
metadata:
  name: redis
  annotations:
    co.elastic.hints/package: redis
    co.elastic.hints/data_streams: info, log
    co.elastic.hints/host: '${kubernetes.pod.ip}:6379'
    co.elastic.hints/info.period: 1m
    co.elastic.hints/info.password: myred1sp@ss
  labels:
    k8s-app: redis
    app: redis
spec:
  initContainers:
  - name: init-container42
    image: busybox
    command: ["echo","I am init-conatiner"]
  containers:
  - image: redis
    imagePullPolicy: IfNotPresent
    name: redis42
    ports:
    - name: redis
      containerPort: 6379
      protocol: TCP
    command:
      - redis-server
      - "--requirepass 'myred1sp@ss'"

Execute the inspect command to produce to populate the inputs:
./elastic-agent inspect -v --variables --variables-wait 2strue

The expected outcome will be 2 filestream inputs, one per container, ie:

...
inputs:
- id: kubernetes-ccd46902-cd37-4180-8fd3-9cfeec4ebc38.init-container42
  name: filestream-redis
  processors:
  - add_fields:
      fields:
        id: 97dafa6d905c4679e24207a3ed2b7e08ed9f1dd5d3af5e77807fa124e4a5f72e
        image:
          name: busybox
        runtime: containerd
      target: container
  - add_fields:
      fields:
        container:
          name: init-container42
        labels:
          app: redis
          k8s-app: redis
        namespace: default
        namespace_labels:
          kubernetes_io/metadata_name: default
        namespace_uid: a5692398-e32f-4332-979f-97f6f040faa3
        node:
          hostname: kind-control-plane
          labels:
            beta_kubernetes_io/arch: amd64
            beta_kubernetes_io/os: linux
            kubernetes_io/arch: amd64
            kubernetes_io/hostname: kind-control-plane
            kubernetes_io/os: linux
            node-role_kubernetes_io/control-plane: ""
            node_kubernetes_io/exclude-from-external-load-balancers: ""
          name: kind-control-plane
          uid: 1e3c36d2-f3e7-4d1f-ba66-75f72102cf22
        pod:
          ip: 10.244.0.7
          name: redis
          uid: ccd46902-cd37-4180-8fd3-9cfeec4ebc38
      target: kubernetes
  - add_fields:
      fields:
        cluster:
          name: kind-kind
          url: https://127.0.0.1:35435
      target: orchestrator
  streams:
  - data_stream:
      dataset: redis.log
      type: logs
    exclude_files:
    - .gz$
    exclude_lines:
    - ^\s+[\-`('.|_]
    parsers:
    - container:
        format: auto
        stream: all
    paths:
    - /var/log/containers/*97dafa6d905c4679e24207a3ed2b7e08ed9f1dd5d3af5e77807fa124e4a5f72e.log
    prospector:
      scanner:
        symlinks: true
    tags:
    - redis-log
  type: filestream
  use_output: default
- id: kubernetes-ccd46902-cd37-4180-8fd3-9cfeec4ebc38.redis42
  name: filestream-redis
  processors:
  - add_fields:
      fields:
        id: 64b605be45dfe470234994f282efb59332a43bae6e21d9f2876f59d7b24ebcad
        image:
          name: redis
        runtime: containerd
      target: container
  - add_fields:
      fields:
        container:
          name: redis42
        labels:
          app: redis
          k8s-app: redis
        namespace: default
        namespace_labels:
          kubernetes_io/metadata_name: default
        namespace_uid: a5692398-e32f-4332-979f-97f6f040faa3
        node:
          hostname: kind-control-plane
          labels:
            beta_kubernetes_io/arch: amd64
            beta_kubernetes_io/os: linux
            kubernetes_io/arch: amd64
            kubernetes_io/hostname: kind-control-plane
            kubernetes_io/os: linux
            node-role_kubernetes_io/control-plane: ""
            node_kubernetes_io/exclude-from-external-load-balancers: ""
          name: kind-control-plane
          uid: 1e3c36d2-f3e7-4d1f-ba66-75f72102cf22
        pod:
          ip: 10.244.0.7
          name: redis
          uid: ccd46902-cd37-4180-8fd3-9cfeec4ebc38
      target: kubernetes
  - add_fields:
      fields:
        cluster:
          name: kind-kind
          url: https://127.0.0.1:35435
      target: orchestrator
  streams:
  - data_stream:
      dataset: redis.log
      type: logs
    exclude_files:
    - .gz$
    exclude_lines:
    - ^\s+[\-`('.|_]
    parsers:
    - container:
        format: auto
        stream: all
    paths:
    - /var/log/containers/*64b605be45dfe470234994f282efb59332a43bae6e21d9f2876f59d7b24ebcad.log
    prospector:
      scanner:
        symlinks: true
    tags:
    - redis-log
  type: filestream
  use_output: default
...

Generic logs collection for all non annotated containers

Use the following agent config:

providers:
  kubernetes:
    kube_config: /home/chrismark/.kube/config
    node: "kind-control-plane"
    hints.enabled: true
    #hints.default_container_logs: false

inputs:
  - name: filestream-generic
    id: hints-container-logs-${kubernetes.hints.container_id}
    type: filestream
    use_output: default
    streams:
      - condition: ${kubernetes.hints.generic_logs.container_logs.enabled} == true
        data_stream:
          dataset: kubernetes.container_logs
          type: logs
        exclude_files: [ ]
        exclude_lines: [ ]
        parsers:
          - container:
              format: auto
              stream: ${kubernetes.hints.redis.generic_logs.stream|'all'}
        paths:
          - /var/log/containers/*${kubernetes.hints.container_id}.log
        prospector:
          scanner:
            symlinks: true
        tags: [ ]
    data_stream.namespace: default

Execute the inspect command to populate the inputs:
./elastic-agent inspect -v --variables --variables-wait 2strue

The expected outcome will be filestream inputs for all the discovered containers, one per container.

Uncomment the hints.default_container_logs: false and run again to see that no generic log inputs are populated.

Signed-off-by: ChrsMark <chrismarkou92@gmail.com>

mergify · 2023-03-22T07:10:56Z

This pull request does not have a backport label. Could you fix it @ChrsMark? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v./d./d./d is the label to automatically backport to the 8./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

Signed-off-by: ChrsMark <chrismarkou92@gmail.com>

elasticmachine · 2023-03-22T07:22:25Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-03-23T05:39:33.812+0000
Duration: 22 min 1 sec

Test stats 🧪

Test	Results
Failed	0
Passed	5015
Skipped	15
Total	5030

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages.
run integration tests : Run the Elastic Agent Integration tests.
run end-to-end tests : Generate the packages and run the E2E Tests.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

elasticmachine · 2023-03-22T07:22:34Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	98.387% (`61/62`)	👍
Files	69.34% (`147/212`)	👍
Classes	68.473% (`278/406`)	👍
Methods	53.605% (`855/1595`)	👎 -0.034
Lines	38.906% (`9497/24410`)	👎 -0.038
Conditionals	100.0% (`0/0`)	💚

gizas · 2023-03-22T16:01:38Z

I tried to test locally but seems that hints emmition is not working?

❯ ./elastic-agent inspect -v --variables --variables-wait 2s -c elastic-agent.yml
inputs: []
providers:
  kubernetes:
    hints:
      enabled: true
    kube_config: /Users/andreasgkizas/.kube/config
    node: pas-control-plane

❯ cat elastic-agent.yml
providers:
  kubernetes:
    kube_config: /Users/andreasgkizas/.kube/config
    node: "pas-control-plane"
    hints.enabled: true

inputs:
  - name: filestream-redis
    type: filestream
    use_output: default
    streams:
      - condition: ${kubernetes.hints.redis.log.enabled} == true or ${kubernetes.hints.redis.enabled} == true
        data_stream:
          dataset: redis.log
          type: logs
        exclude_files:
          - .gz$
        exclude_lines:
          - ^\s+[\-`('.|_]
        parsers:
          - container:
              format: auto
              stream: ${kubernetes.hints.redis.log.stream|'all'}
        paths:
          - /var/log/containers/*${kubernetes.hints.container_id}.log
        prospector:
          scanner:
            symlinks: true
        tags:
          - redis-log

Also an observation: Is this accurate: - /var/log/containers/*${kubernetes.hints.container_id}.log ?

Signed-off-by: ChrsMark <chrismarkou92@gmail.com>

gizas · 2023-03-23T12:07:32Z

Thanks @ChrsMark for the extra eyes and help to troubleshoot!

I needed a bigger wait period
./elastic-agent inspect -c config_sample2.yaml -v --variables --variables-wait 5s

New config_sample2.yaml :

providers:
  kubernetes:
    kube_config: /Users/andreasgkizas/.kube/config
    node: "pas-control-plane"
    hints.enabled: true

inputs:
  - name: filestream-generic
    id: hints-container-logs-${kubernetes.hints.container_id}
    type: filestream
    use_output: default
    streams:
      - condition: ${kubernetes.hints.generic_logs.container_logs.enabled} == true
        data_stream:
          dataset: kubernetes.hints.container_logs
          type: logs
        exclude_files: [ ]
        exclude_lines: [ ]
        parsers:
          - container:
              format: auto
              stream: ${kubernetes.hints.generic_logs.stream|'all'}
        paths:
          - /var/log/containers/*${kubernetes.hints.container_id}.log
        prospector:
          scanner:
            symlinks: true
        tags: [ ]
    data_stream.namespace: default

This way we get all the logs from cluster:
output_sample.txt

Fix hints generation when missing container ports

e6cf972

Signed-off-by: ChrsMark <chrismarkou92@gmail.com>

mergify bot assigned ChrsMark Mar 22, 2023

mergify bot added the backport-skip label Mar 22, 2023

ChrsMark requested review from a team, gsantoro and MichaelKatsoulis and removed request for a team March 22, 2023 07:11

Extract common functionality into function

445d145

Signed-off-by: ChrsMark <chrismarkou92@gmail.com>

framsouza mentioned this pull request Mar 22, 2023

elastic-agent standalone not collecting logs from multiples containers at the same pod #2375

Open

ChrsMark requested a review from gizas March 23, 2023 03:31

ChrsMark marked this pull request as ready for review March 23, 2023 03:32

ChrsMark requested a review from a team as a code owner March 23, 2023 03:32

ChrsMark requested review from michalpristas and michel-laterman and removed request for a team March 23, 2023 03:32

Add generic_logs fallback to collect logs from all container by default

876dcea

Signed-off-by: ChrsMark <chrismarkou92@gmail.com>

ChrsMark requested a review from a team as a code owner March 23, 2023 05:39

ChrsMark changed the title ~~Fix hints generation when missing container ports~~ Fix log collection with hints when missing container ports or annotations Mar 23, 2023

gizas approved these changes Mar 23, 2023

View reviewed changes

MichaelKatsoulis approved these changes Mar 24, 2023

View reviewed changes

michalpristas approved these changes Mar 28, 2023

View reviewed changes

ChrsMark added v8.8.0 8.8-candidate labels Mar 28, 2023

ChrsMark merged commit cdd839d into elastic:main Mar 28, 2023

gizas mentioned this pull request Jul 3, 2023

Adding generic_logs template for hints autodiscovery #2981

Merged

7 tasks

ChrsMark mentioned this pull request Jul 19, 2023

[Elastic Agent ] Support of processors functionality with hints autodiscovery #2959

Closed

ChrsMark mentioned this pull request Aug 1, 2023

Document default_container_logs setting of k8s provider #3158

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix log collection with hints when missing container ports or annotations #2386

Fix log collection with hints when missing container ports or annotations #2386

ChrsMark commented Mar 22, 2023 •

edited

Loading

mergify bot commented Mar 22, 2023

elasticmachine commented Mar 22, 2023 •

edited

Loading

Build stats

Test stats 🧪

elasticmachine commented Mar 22, 2023 •

edited

Loading

gizas commented Mar 22, 2023

gizas commented Mar 23, 2023

Fix log collection with hints when missing container ports or annotations #2386

Fix log collection with hints when missing container ports or annotations #2386

Conversation

ChrsMark commented Mar 22, 2023 • edited Loading

What does this PR do?

How to test this PR manaully

Multiple containers case

Generic logs collection for all non annotated containers

mergify bot commented Mar 22, 2023

elasticmachine commented Mar 22, 2023 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

elasticmachine commented Mar 22, 2023 • edited Loading

🌐 Coverage report

gizas commented Mar 22, 2023

gizas commented Mar 23, 2023

ChrsMark commented Mar 22, 2023 •

edited

Loading

elasticmachine commented Mar 22, 2023 •

edited

Loading

elasticmachine commented Mar 22, 2023 •

edited

Loading