Kubernetes logs and metrics are not being collected #1894
Comments
MichaelKatsoulis
added
bug
|
Hmm, there is definitely nothing obviously causing in the log snippers here. If you have the complete set of agent logs or even better diagnostics collected with It may be worth retesting this after elastic/beats#33921 is merged, as the bugs fixed there have been causing several problems with the wrong input type being started or the wrong data streams being used. |
|
I packaged elastic agent from the latest 8.6 branch which includes the fix you mention as well and run it inside kubernetes with kubernetes integration. Still agent is unhealthy. One more error I see is that right after it creates new components for logs and metrics it logs: Diagnostics file |
|
Hmm, |
|
Latest try.
|
|
After #1936 I tested latest agent image at https://snapshots.elastic.co/8.6.0-6aa72183/summary-8.6.0-SNAPSHOT.html#elastic-agent. diagnostics: Metrics and Logs are collected but there is another kind of problem. Same for metrics: Troubleshooting in the code with @ChrsMark we found out that the kubernetes provider adds the mappings and processors in the emitted events to the controller. We tried out locally with this config: and the result of inspect is which means that variable resolution is not working. |
|
Hey! Here is another test, more realistic:
providers:
kubernetes:
kube_config: /home/chrismark/.kube/config
node: "kind-control-plane"
inputs:
- name: container-log
id: container-log-${kubernetes.pod.name}-${kubernetes.container.id}
type: filestream
use_output: default
meta:
package:
name: kubernetes
version: 1.9.0
data_stream:
namespace: default
streams:
- data_stream:
dataset: kubernetes.container_logs
type: logs
condition: ${kubernetes.labels.app} == 'redis'
prospector.scanner.symlinks: true
parsers:
- container: ~
# - ndjson:
# target: json
# - multiline:
# type: pattern
# pattern: '^\['
# negate: true
# match: after
paths:
- /var/log/containers/*${kubernetes.container.id}.log
- name: redis
type: redis/metrics
use_output: default
meta:
package:
name: redis
version: 0.3.6
data_stream:
namespace: default
streams:
- data_stream:
dataset: redis.info
type: metrics
metricsets:
- info
hosts:
- '${kubernetes.pod.ip}:6379'
idle_timeout: 20s
maxconn: 10
network: some/${kubernetes.container.id}
period: 10s
condition: ${kubernetes.labels.app} == 'redis'Then deploy the target Pod:
apiVersion: v1
kind: Pod
metadata:
name: redis
labels:
k8s-app: redis
app: redis
spec:
containers:
- image: redis
imagePullPolicy: IfNotPresent
name: redis
ports:
- name: redis
containerPort: 6379
protocol: TCPThen from the latest main ( Running for Is this something that we miss when running the new form of |
|
it may be that inspect command is broken. what do you see when you run diagnostics command unzip archive and check |
|
Good catch. I checked the But the fields added by the processors are not populated into elasticsearch. Maybe the higher lever processors are not respected in the lower level streams? Should I open a new issue for this? |
|
i'd go with a separate issue to track this |
|
Not known, thanks for creating a new issue. |
|
as metrics and logs are present but without meta, can we close this? |
Description
Using latest elastic-agent, Kubernetes Integration is not able to collect metrics and logs from Kubernetes pods.
Versions
stack version: latest 8.6.0-SNAPSHOT
kubernetes package version: 1.29.2
Steps to reproduce
elastic-package stack up --version=8.6.0-SNAPSHOT -v -dkind create clusterhttps://elasticsearch:9200and Advanced YAML configurationssl.verification_mode: "none"Possible Logs showing the problem
If we remove the metrics collection from the policy and leave only the container_logs then most of the error messages disappear and the only constant one is
This is related to #1860. Nothing else suspicious is logged. But no pod logs are collected. So something prevents filebeat from running.
The text was updated successfully, but these errors were encountered: