Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding information for NETINFO environmental variable #463

Merged
merged 8 commits into from
Sep 12, 2023
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,9 @@ include::shared-env.asciidoc[tag=kibana-password]

include::shared-env.asciidoc[tag=kibana-ca]

include::shared-env.asciidoc[tag=elastic-netinfo]


|===

[discrete]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -560,4 +560,20 @@ contains your CA's certificate.

// end::kibana-ca[]

// tag::elastic-netinfo[]
|
[id="env-{type}-elastic-netinfo"]
`ELASTIC_NETINFO`

| (bool) When `false`, disables `netinfo.enabled` parameter of `add_host_metadata` processor.
Setting this to `false` is recommended for large scale setups where the host.ip and host.mac fields index size increases.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lets add some more details here around what the problem is -> tons of ip addresses, tons of storage needed, slower queries

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Let me know if is ok

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can still add more details here but lets do this as a follow up.


By default, {agent} initializes the `add_host_metadata` processor. The `netinfo.enabled` parameter defines ingestion of IP addresses and MAC addresses as fields `host.ip` and `host.mac`.
For more information see <<add_host_metadata-processor>>


*Default:* `"false"`

// end::elastic-netinfo[]

// =============================================================================
Original file line number Diff line number Diff line change
Expand Up @@ -135,7 +135,7 @@ Setting the low priority to {agent} comparing to other pods might also result to
Trying to prioritise the agent installation before rest of application microservices, https://github.com/elastic/elastic-agent/blob/main/docs/manifests/elastic-agent-managed-gke-autopilot.yaml#L8-L16[PriorityClasses suggested]

[discrete]
=== {k8s} Policy Configuration
=== {k8s} Package Configuration

Policy configuration of {k8s} package can heavily affect the amount of metrics collected and finally ingested. Factors that should be considered in order to make your collection and ingestion lighter:

Expand All @@ -145,9 +145,21 @@ Policy configuration of {k8s} package can heavily affect the amount of metrics c
- Disable events dataset
- Disable {k8s} control plane datasets in Cloud managed {k8s} instances (see more info ** <<running-on-gke-managed-by-fleet>>, <<running-on-eks-managed-by-fleet>>, <<running-on-aks-managed-by-fleet>> pages)


[discrete]
=== Dashboards and Visualisations

The https://github.com/elastic/integrations/blob/main/docs/dashboard_guidelines.md[Dashboard Guidelines] document provides guidance on how to implement your dashboards and is constantly updated to track the needs of Observability at scale.

User experience regarding Dashboard responses, is also affected from the size of data being requested. As dashboards can contain multiple visualisations, the general consideration is to split visualisations and group them according to the frequency of access. The less number of visualisations tends to improve user experience.

Additionally, https://github.com/elastic/integrations/blob/main/docs/dashboard_guidelines.md[Dashboard Guidelines] is constantly updated also to track needs of observability at scale.
[discrete]
=== Disabling indexing host.ip and host.mac fields

A new environemntal variable `ELASTIC_NETINFO: false` has been introduced to globally disable the indexing of `host.ip` and `host.mac` fields in your {k8s} integration. For more information see <<agent-environment-variables>>.

Setting this to `false` is recommended for large scale setups where the `host.ip` and `host.mac` fields' index size increases. The number of IPs and MAC addresses reported increases significantly as a Kubenetes cluster grows. This leads to considerably increased indexing time, as well as the need for extra storage and additional overhead for visualization rendering.


[discrete]
=== Elastic Stack Configuration
Expand Down