Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Kubernetes] Add top level processor add_fields for the cluster name usecase #10878

Open
tetianakravchenko opened this issue Aug 26, 2024 · 2 comments
Assignees
Labels
enhancement New feature or request Integration:kubernetes Kubernetes Team:Cloudnative-Monitoring Label for the Cloud Native Monitoring team [elastic/obs-cloudnative-monitoring]

Comments

@tetianakravchenko
Copy link
Contributor

One of the common processor that might be used is a add_fields to add a orchestrator.cluster.name field - we mention it in the k8s module doc - https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-kubernetes.html#_dashboard_32:

processors:
  - add_fields:
      target: orchestrator.cluster
      fields:
        name: clusterName
        url: clusterURL

Now we provide an option to set processors on the data_stream level, but it is quite a lot of changes for the such a common scenario.

We should provide a way to set orchestrator.cluster.name on the integration level

@andrewkroh andrewkroh added Integration:kubernetes Kubernetes enhancement New feature or request Team:Cloudnative-Monitoring Label for the Cloud Native Monitoring team [elastic/obs-cloudnative-monitoring] labels Aug 26, 2024
@tetianakravchenko
Copy link
Contributor Author

draft branch https://github.com/elastic/integrations/compare/main...tetianakravchenko:integrations:k8s_add_cluster_name?expand=1
UI:
Image

as a result only this processor will be added to all data_stream:

{{#if cluster_name}}
- add_fields:
    target: orchestrator.cluster
    fields:
      name: {{cluster_name}}
{{/if}}

@tetianakravchenko tetianakravchenko self-assigned this Sep 10, 2024
@tetianakravchenko
Copy link
Contributor Author

tetianakravchenko commented Sep 10, 2024

Few issues:

  1. In case orchestrator.cluster.name is set - add_fields processor add the value to the array:
    Image

This behavior actually contradicts the processor description https://www.elastic.co/guide/en/beats/filebeat/current/add-fields.html:

The add_fields processor will overwrite the target field if it already exists.

Issue - elastic/beats#40780
2. state_namespace metricset does not include orchestrator.cluster.name field in kind env

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request Integration:kubernetes Kubernetes Team:Cloudnative-Monitoring Label for the Cloud Native Monitoring team [elastic/obs-cloudnative-monitoring]
Projects
None yet
Development

No branches or pull requests

2 participants