elastic-agent standalone not collecting logs from multiples containers at the same pod #2375
Comments
cmacknz
added
the
Team:Cloudnative-Monitoring
|
Hey @framsouza , thanks for filing this issue. The function you mention is called once per container, since we loop over the Pod's containers at .We will need to investigate this a bit to check what's going wrong here. |
|
So I tried to reproduce the issue and it seems that it's not an issue on the hints codebase as I mentioned already in the previous comment. Here is what I try: target sample Pod with 2 containers: apiVersion: v1
kind: Pod
metadata:
name: redis
annotations:
co.elastic.hints/package: redis
co.elastic.hints/data_streams: info, log
co.elastic.hints/host: '${kubernetes.pod.ip}:6379'
co.elastic.hints/info.period: 1m
labels:
k8s-app: redis
app: redis
spec:
containers:
- image: redis
imagePullPolicy: IfNotPresent
name: redis
ports:
- name: redis
containerPort: 6379
protocol: TCP
command:
- redis-server
- "--requirepass 'myred1sp@ss'"
- image: redis
imagePullPolicy: IfNotPresent
name: redis2
ports:
- name: redis
containerPort: 6379
protocol: TCP
command:
- redis-server
- "--requirepass 'myred1sp@ss'"agent's config: providers:
kubernetes:
kube_config: /home/chrismark/.kube/config
node: "kind-control-plane"
hints.enabled: true
inputs:
- name: filestream-redis
type: filestream
use_output: default
streams:
- condition: ${kubernetes.hints.redis.log.enabled} == true or ${kubernetes.hints.redis.enabled} == true
data_stream:
dataset: redis.log
type: logs
exclude_files:
- .gz$
exclude_lines:
- ^\s+[\-`('.|_]
parsers:
- container:
format: auto
stream: ${kubernetes.hints.redis.log.stream|'all'}
paths:
- /var/log/containers/*${kubernetes.hints.container_id}.log
prospector:
scanner:
symlinks: true
tags:
- redis-logThen running the inspect command to evaluate the produced variables:
the produced output: agent:
logging:
to_stderr: true
inputs:
- id: kubernetes-dd54c599-cf20-40db-8394-e1f4458c05a7.redis
name: filestream-redis
processors:
- add_fields:
fields:
id: db404a606cf0e00b5fb596b645a87d5197155bb903e3b8f8389dabac5245c262
image:
name: redis
runtime: containerd
target: container
- add_fields:
fields:
container:
name: redis
labels:
app: redis
k8s-app: redis
namespace: default
namespace_labels:
kubernetes_io/metadata_name: default
namespace_uid: a5692398-e32f-4332-979f-97f6f040faa3
node:
hostname: kind-control-plane
labels:
beta_kubernetes_io/arch: amd64
beta_kubernetes_io/os: linux
kubernetes_io/arch: amd64
kubernetes_io/hostname: kind-control-plane
kubernetes_io/os: linux
node-role_kubernetes_io/control-plane: ""
node_kubernetes_io/exclude-from-external-load-balancers: ""
name: kind-control-plane
uid: 1e3c36d2-f3e7-4d1f-ba66-75f72102cf22
pod:
ip: 10.244.0.7
name: redis
uid: dd54c599-cf20-40db-8394-e1f4458c05a7
target: kubernetes
- add_fields:
fields:
cluster:
name: kind-kind
url: https://127.0.0.1:35435
target: orchestrator
streams:
- data_stream:
dataset: redis.log
type: logs
exclude_files:
- .gz$
exclude_lines:
- ^\s+[\-`('.|_]
parsers:
- container:
format: auto
stream: all
paths:
- /var/log/containers/*db404a606cf0e00b5fb596b645a87d5197155bb903e3b8f8389dabac5245c262.log
prospector:
scanner:
symlinks: true
tags:
- redis-log
type: filestream
use_output: default
- id: kubernetes-dd54c599-cf20-40db-8394-e1f4458c05a7.redis2
name: filestream-redis
processors:
- add_fields:
fields:
id: 4a8630bdf661ddb23a697f1acb547a8487803a66a61cdeace1550ea7169a6f8e
image:
name: redis
runtime: containerd
target: container
- add_fields:
fields:
container:
name: redis2
labels:
app: redis
k8s-app: redis
namespace: default
namespace_labels:
kubernetes_io/metadata_name: default
namespace_uid: a5692398-e32f-4332-979f-97f6f040faa3
node:
hostname: kind-control-plane
labels:
beta_kubernetes_io/arch: amd64
beta_kubernetes_io/os: linux
kubernetes_io/arch: amd64
kubernetes_io/hostname: kind-control-plane
kubernetes_io/os: linux
node-role_kubernetes_io/control-plane: ""
node_kubernetes_io/exclude-from-external-load-balancers: ""
name: kind-control-plane
uid: 1e3c36d2-f3e7-4d1f-ba66-75f72102cf22
pod:
ip: 10.244.0.7
name: redis
uid: dd54c599-cf20-40db-8394-e1f4458c05a7
target: kubernetes
- add_fields:
fields:
cluster:
name: kind-kind
url: https://127.0.0.1:35435
target: orchestrator
streams:
- data_stream:
dataset: redis.log
type: logs
exclude_files:
- .gz$
exclude_lines:
- ^\s+[\-`('.|_]
parsers:
- container:
format: auto
stream: all
paths:
- /var/log/containers/*4a8630bdf661ddb23a697f1acb547a8487803a66a61cdeace1550ea7169a6f8e.log
prospector:
scanner:
symlinks: true
tags:
- redis-log
type: filestream
use_output: default
outputs:
default:
api-key: example-key
hosts:
- 127.0.0.1:9200
type: elasticsearch
providers:
kubernetes:
hints:
enabled: true
kube_config: /home/chrismark/.kube/config
node: kind-control-planeAs we can see there are 2 inputs populated with the proper paths accordingly:
I will also try with init containers just in case we miss something here. In the meantime @framsouza it would help if you could provide Agent's diagnostics so as to check the status of your populated inputs. Last but not least it seems that we need to update our templates for hints like the one at https://github.com/elastic/elastic-agent/blob/main/deploy/kubernetes/elastic-agent-standalone/templates.d/redis.yml#L3 to also include the @gizas @mlunadia raising this with you since all these would need some extra capacity from the team. |
|
@ChrsMark I'll provide the elastic-agent diagnostic shortly. Just for clarity, this problem is happening when there's an InitContainer and not a sidecar, if you spin up ECK you'll see an initContainer called elastic-internal-init-filesystem , only the logs from this initContainer are being ingested, we're missing the logs from the main container |
|
Thanks for clarifying @framsouza . Could you also provide the annotations you use and how you configure Elastic Agent (the k8s manifest would be enough) ? |
|
@ChrsMark I'm running this configmap: And, With the following daemonset: Against the vault-ci-dev GKE cluster, there's a vault statefulset running on vault-ci-dev namespace, the logs outputs are being sent to the logging cluster (RLC). If you check access the cluster, you'll see the following output for the vault-ci-dev namespace vault container has 3 pods (filebeat, logrotate and vault), if you check the Saved search, you'll see we only see the logs from the filebeat container , we're missing logs from vault and logrotate. See elastic-agent inject output: I didn't find a container.name called vault-ci-dev on the file mentioned above, only filebeat. The logs file exist inside the elastic-agent pod, see |
|
attaching the elastic-agent diagnostic, |
|
Thanks for the extra info @framsouza! Let me try to put all these into an order. So first of all we have the following input: inputs:
- name: container-log
condition: startsWith(${kubernetes.pod.name}, "elastic-agent") != true && ${kubernetes.hints.logs.enabled} == true
type: filestream
processors:
- add_fields:
target: orchestrator.cluster
fields:
name: {{ .Values.cluster.name }}
use_output: logs
meta:
package:
name: kubernetes
version: 1.9.0
data_stream:
namespace: default
streams:
- data_stream:
dataset: kubernetes.container_logs
type: logs
prospector.scanner.symlinks: true
parsers:
- container: ~
- ndjson:
target: json
add_error_key: true
message_key: message
overwrite_keys: true
ignore_decoding_error: true
# - multiline:
# type: pattern
# pattern: '^\['
# negate: true
# match: after
paths:
- /var/log/containers/*${kubernetes.container.id}.logSo here I see that you mix 2 different things. First you use the Also I see that you have remove the init container that downloads the hints templates but that's ok since you try to define your own templates. However if you are defining your own templates you can skip the whole hints mechanism and just define your own conventions. The hints feature is only useful if you are willing to use the out of the box conventions coming the predefined templates at https://github.com/elastic/elastic-agent/tree/main/deploy/kubernetes/elastic-agent-standalone/templates.d. And keep in mind that Hints feature is still in beta. Having said this a working template should look the one at https://github.com/elastic/elastic-agent/blob/main/deploy/kubernetes/elastic-agent-standalone/templates.d/cassandra.yml#L21, see that the path is like So in that case I wonder how the provided config was working but definitely the templates need to be fixed. Can you please fix this and try again? It would also help if you can provide a minimal example with target Pods with multiple containers so as to have a specific example to reproduce. Apparently debugging against am active vault-ci-dev GKE cluster is not a good idea. For reference I have spotted an issue in hints codebase. At we only emit hints mappings for containers that have defined Ports. This means that containers with no Ports defined will not have an emitted mapping. This is an issue and I will push a fix for it, however I will wait to see if we can spot any additional issues here. |
|
This PR seems to be a potential fix #2386 |
|
I've performed the changes as @ChrsMark suggested above, the daemonset now is using the initContainer to download the templates, However, once I applied the new configuration the logs stopped flowing and as @ChrsMark mentioned, it might be because we missed a generic input for all the container on the templates.d/ |
|
@framsouza #2386 is sth we definitely need. So I will go ahead and try to have it merged soon. Let's see if after this the issue is fixed for you. In general you can add in Agent's config a generic input for catching logs for all containers. Just make sure you follow the proper convention. However this is sth that we need to fix. Do you mind filing a new issue for this so as to prioritize it properly? Last but not least, as I mentioned already I see that hints work properly when you use specific packages in the hints like the one at #2375 (comment). Having said this, if you can provide a minimal example (like the one I shared at #2375 (comment)) of what is not working for you it would help us to understand what is the case that we don't cover. So we would need specific target Pod with annotations attached and proper Agent configuration/template. Thanks. |
|
Update: in #2386 I add functionality to collect logs from all of the non annotated Pods by using a specific fallback. |
|
great stuff, thanks for that. I'll wait for the merge and test that out. I'll keep you in the loop (#2386) |
|
@framsouza were you able to test @ChrsMark PR and if yes can I close this issue? |
We're currently having some issues running elastic-agent standalone with Hints annotations-based autodiscovery, the agent is currently collecting logs only from one container inside the pod.
To test that, you can spin up Elasticseach with ECK and enable the hints annotations, you'll see something similar the following:
I can see below we're getting only the
elastic-internal-init-filesystemwhich is a sidecar container, but we don't get any logs from the elasticsearch container itself.Checking the logs inside the container, you'll see that the logs are being properly generated but not being shipped:
I had a look at the hints.go code and it seems the function responsible for collecting those logs has a
containerIDparameter mapped asstringand I think it should be astructinstead to be able to collect more than one containerID, seeMore insights about the issue here
The text was updated successfully, but these errors were encountered: