Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NH-9834: Logs monitoring #49

Merged
merged 4 commits into from
Jul 1, 2022
Merged

NH-9834: Logs monitoring #49

merged 4 commits into from
Jul 1, 2022

Conversation

gantrior
Copy link
Contributor

@gantrior gantrior commented Jul 1, 2022

Collecting container logs using using separate Otel collector.

Based on information provided by @pawel-kedzior-sw in https://swicloud.atlassian.net/wiki/spaces/NIT/pages/2849178774/FAS+K8s+Agent-based+Logs+Collector
with following difference

  • I had to rework filelog configuration, as the one mentioned in https://gist.github.com/sumo-drosiek/c7b9d812cc3279062b0fe74e13aea64e is designed for older Otel collector version and did not work anymore for the Otel Collector v51 that we are using
  • I had to mount also /var/log/containers and /var/lib/docker/containers as /var/log/pods contains only symlink files to those location
  • I also excluded monitoring of logs of monitor itself
  • I also added node name and cluster uid/name

This PR does not include journal logs as it is more complicated due to the fact that journalreceiver required journalctl binary to be present in the image

I am concerned about amount of that we will be sending to our infrastructure, as it could be hunderds of megabytes every minute

@gantrior gantrior requested a review from a team July 1, 2022 07:30
@gantrior gantrior requested a review from etichy as a code owner July 1, 2022 08:57
@gantrior gantrior requested a review from pawel-kedzior-sw July 1, 2022 09:05
gantrior added 3 commits July 1, 2022 11:35
also adding basic test, verifying that some logs were recieved
To reduce amount of logs being sent by default if this feature goes to public without any further adjustments
@gantrior
Copy link
Contributor Author

gantrior commented Jul 1, 2022

After discussion with @pstranak-sw I am whitelisting logs to only logs coming from pods in internal kubernetes namespaces (kube-system, kube-public). To make sure that if this goes public without any filtering options it wouldn't kill our infrastructure. See 590a0c0

@gantrior gantrior merged commit cfa1f57 into master Jul 1, 2022
@gantrior gantrior deleted the feature/NH-9834 branch July 1, 2022 12:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

2 participants