Filebeat Kubernetes metadata is not optional

[shane99a]

Report:

• Version: 6.2.3
• Operating System: CentOS
• Discuss Forum URL: https://discuss.elastic.co/t/filebeat-kubernets-api-metadata-isnt-optional/134273

Steps to Reproduce:

• Add kubernetes metadata config to filebeat. Create a service account that CANNOT read the k8s api. Check if the logs get sent without the metadata
• What will happen is the logs do not get sent
• You have to modify the service account to access the api and then it will send those logs

I am testing this in minikube. I believe the logs should be sent, but without the metadata. Let me know if this is expected or not. This is the error Filebeat will throw when it cant access the api:

{"time":"2018-06-01T16:32:04.950783-04:00","log":"2018-06-01T20:32:04.950Z#011INFO#011kubernetes\/watcher.go:140#011kubernetes: Watching API for pod events"}
{"time":"2018-06-01T16:32:04.951133-04:00","log":"2018-06-01T20:32:04.950Z#011ERROR#011kubernetes\/watcher.go:145#011kubernetes: Watching API error kubernetes api: Failure 403 pods is forbidden: Ubeat-user\" cannot watch pods at the cluster scope"}
{"time":"2018-06-01T16:32:04.951374-04:00","log":"2018-06-01T20:32:04.950Z#011INFO#011kubernetes\/watcher.go:140#011kubernetes: Watching API for pod events"}
{"time":"2018-06-01T16:32:04.952057-04:00","log":"2018-06-01T20:32:04.951Z#011ERROR#011kubernetes\/watcher.go:145#011kubernetes: Watching API error kubernetes api: Failure 403 pods is forbidden: Ubeat-user\" cannot watch pods at the cluster scope"}

[shane99a]

All right good news. I updated to version 6.2.4, and its no longer an issue. However I would love it if someone can confirm who worked on Filebeat that this was indeed fixed in that version. I can feel confident in closing this ticket then. Thank you.

[exekias]

ey @shane99a, you were probably suffering from this issue: #6504

Thank you for the report, and confirming the fix, sorry for the inconveniences

[shane99a]

Hi, @exekias thanks for responding. I probably should have updated that I no longer have this issue even on 6.2.3. If the ticket linked above is an issue that only happens sometimes, then it probably makes sense why it wasn't working before and suddenly it worked again. Do you recommend I upgrade to 6.2.4 just to be safe?

[exekias]

Yes, I definitely recommend using 6.2.4 here, as 6.2.3 can cause issues like that (for instance when your API server is restarted)