Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sidecar Container not starting #335

Closed
shaunfink opened this issue Jul 1, 2022 · 3 comments
Closed

Sidecar Container not starting #335

shaunfink opened this issue Jul 1, 2022 · 3 comments

Comments

@shaunfink
Copy link

HI all

We're using version 2.7.0 of the official sumologic helm chart for deploying sumo to our EKS clusters. We've found a situation where we've had to make use of the tailing sidecar to ingest logs from a container for a service, as these logs are not written to stdout/stderr.

As the sumo chart has this bundled in as a dependant chart, it seems like the sidecar image is not working as expected, but is also running a MUCH older version of the chart (version 0.3.2, based on what I can see in Chart.yaml)

I get the following in logs when running the sidecar

[2022/07/01 09:42:11] [error] [sqldb] cannot open database /tailing-sidecar/var/fluent-bit-state.db
[2022/07/01 09:42:11] [error] [input:tail:tail.0] could not open/create database
[2022/07/01 09:42:11] [error] Failed initialize input tail.0
[2022/07/01 09:42:11] [error] [lib] backend failed

Overriding the version of both the sidecar to version 0.5.0 seems to work a treat though.

Configs I'm defining in values.yaml for the sumo chart are the following:

tailing-sidecar-operator:
  enabled: true
  
  sidecar:
    image:
      tag: 0.5.0

If I do overrides for later versions of the operator (pretty much anything up from 0.3.2), I see loads of these issues in operator logs (which look to relate to #168:

E0701 10:10:45.493493       1 leaderelection.go:325] error retrieving resource lock monitoring/7b555970.sumologic.com: leases.coordination.k8s.io "7b555970.sumologic.com" is forbidden: User "system:serviceaccount:monitoring:tailing-sidecar-service-account" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "monitoring"

Reverting all these configs to use the 0.3.2 operator, and the 0.5.0 sidecar looks to work.

@kasia-kujawa
Copy link
Contributor

In sumologic-kubernetes-collection the version of tailing sidecar operator is quite old because the newest version contains breaking changes (SumoLogic/sumologic-kubernetes-collection#1739)
I don't recommend of using older operator with newer version of sidecar image but testing the configuration that you mention I haven't notice any issue.

Could you share the configuration that you prepared for tailing sidecar operator and specification of the Pod with changes made by tailing sidecar operator?

If you want to use newer version of the Tailing Sidecar Operator you can install it as separated helm chart without sumologic-kubernetes-collection, details: https://github.com/SumoLogic/tailing-sidecar#tldr

#168 may be observed is some clusters, now tailing-sidecar-leader-election-role role must be manually extended, following permissions must be add:

- apiGroups:
  - coordination.k8s.io
  resources:
  - leases
  verbs:
  - create
  - get
  - list
  - update

@sumo-drosiek
Copy link
Contributor

This would be fixed by #377

@kasia-kujawa
Copy link
Contributor

I'm closing this issue, @shaunfink if you still observe the issue, please reopen it ;)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants