Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

adding docs for default container in 8.9 #386

Merged
merged 1 commit into from
Aug 4, 2023
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,7 @@ and avoid placing passwords in plain text.

The stream to use for logs collection, for example, stdout/stderr.

If the specified package has no logs support, a generic container's logs input will be used as a fallback.
If the specified package has no logs support, a generic container's logs input will be used as a fallback. See the `Hints autodiscovery for kubernetes log collection` example below.

[discrete]
== Available packages that support hints autodiscovery
Expand All @@ -92,7 +92,7 @@ https://github.com/elastic/elastic-agent/tree/main/deploy/kubernetes/elastic-age
[discrete]
== Configure hints autodiscovery

To enable hints, you must add `hints.enabled: true` to the provider's configuration:
To enable hints autodiscovery, you must add `hints.enabled: true` to the provider's configuration:
Copy link
Member

@ChrsMark ChrsMark Aug 3, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That section does not exist in https://github.com/elastic/ingest-docs/pull/304/files. Is this coming from another PR? If so we should mention it as well in the description.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated the desc, it was the #380


[source,yaml]
----
Expand Down Expand Up @@ -134,8 +134,15 @@ initContainers:
mountPath: /etc/elastic-agent/inputs.d
----


NOTE: The {agent} can load multiple configuration files from `{path.config}/inputs.d` and finally produce a unified one (refer to <<elastic-agent-configuration>>). Users have the ability to manually mount their own templates under `/etc/elastic-agent/inputs.d` *if they want to skip enabling initContainers section*.


[discrete]
== Examples:

[discrete]
== Example: Hints autodiscovery
=== Hints autodiscovery for redis

Enabling hints allows users deploying Pods on the cluster to automatically turn on Elastic
monitoring at Pod deployment time.
Expand Down Expand Up @@ -164,6 +171,115 @@ After deploying this Pod, the data will start flowing in automatically. You can
NOTE: All assets (dashboards, ingest pipelines, and so on) related to the Redis integration are not installed. You need to explicitly <<install-uninstall-integration-assets,install them through {kib}>>.


[discrete]
=== Hints autodiscovery for kubernetes log collection

The log collection for Kubernetes autodiscovered pods can be supported by using https://github.com/elastic/elastic-agent/tree/main/deploy/kubernetes/elastic-agent-standalone/templates.d/container_logs.yml[container_logs.yml template]. Elastic Agent needs to emit a container_logs mapping so as to start collecting logs for all the discovered containers *even if no annotations are present in the containers*.

1. Follow steps described above to enable Hints Autodiscover
2. Make sure that relevant `container_logs.yml` template will be mounted under /etc/elastic-agent/inputs.d/ folder of Elastic Agent
3. Deploy Elastic Agent Manifest
4. Elastic Agent should be able to discover all containers inside kuernetes cluster and to collect available logs.

The previous default behaviour can be disabled with `hints.default_container_logs: false`.
So this will disable the automatic logs collection from all discovered pods. Users need specifically to annotate their pod with following annotations:

[source,yaml]
----
annotations:
co.elastic.hints/package: "container_logs"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a note that container_logs package only made it to main with elastic/elastic-agent#2981 but it doesn't make sense to restrict it since we always pull the templates from latest main: https://github.com/elastic/elastic-agent/blob/083bf7805afc9d57727b099da6fa48cf96719585/deploy/kubernetes/elastic-agent-standalone-kubernetes.yaml#L686

----


[source,yaml]
----
providers.kubernetes:
node: ${NODE_NAME}
scope: node
hints:
enabled: true
default_container_logs: false
...
----

In the following sample nginx manifest, we will additionally provide specific stream annotation, in order to configure the filestream input to read only stderr stream:

[source,yaml]
----
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: nginx
name: nginx
namespace: default
spec:
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
annotations:
co.elastic.hints/package: "container_logs"
co.elastic.hints/stream: "stderr"
spec:
containers:
- image: nginx
name: nginx
...
----

Users can monitor the final rendered Elastic Agent configuration:

[source,bash]
----
kubectl exec -ti -n kube-system elastic-agent-7fkzm -- bash


/usr/share/elastic-agent# /elastic-agent inspect -v --variables --variables-wait 2s

inputs:
- data_stream.namespace: default
id: hints-container-logs-3f69573a1af05c475857c1d0f98fc55aa01b5650f146d61e9653a966cd50bd9c-kubernetes-1780aca0-3741-4c8c-aced-b9776ba3fa81.nginx
name: filestream-generic
original_id: hints-container-logs-3f69573a1af05c475857c1d0f98fc55aa01b5650f146d61e9653a966cd50bd9c
[output truncated ....]
streams:
- data_stream:
dataset: kubernetes.container_logs
type: logs
exclude_files: []
exclude_lines: []
parsers:
- container:
format: auto
stream: stderr
paths:
- /var/log/containers/*3f69573a1af05c475857c1d0f98fc55aa01b5650f146d61e9653a966cd50bd9c.log
prospector:
scanner:
symlinks: true
tags: []
type: filestream
use_output: default
outputs:
default:
hosts:
- https://elasticsearch:9200
password: changeme
type: elasticsearch
username: elastic
providers:
kubernetes:
hints:
default_container_logs: false
enabled: true
node: control-plane
scope: node
----


[discrete]
== Troubleshooting
Expand Down