Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

operator: cilium-operator terminates when Kubernetes libraries try to log a non-existent directory #7006

Closed
ianvernon opened this issue Feb 8, 2019 · 3 comments · Fixed by #7050
Closed

operator: cilium-operator terminates when Kubernetes libraries try to log a non-existent directory #7006

ianvernon opened this issue Feb 8, 2019 · 3 comments · Fixed by #7050
Assignees
@ianvernon
Labels
area/operator Impacts the cilium-operator component kind/bug This is a bug in the Cilium logic.

Comments

@ianvernon
Copy link
Member

The following error caused the cilium-operator to exit:

W0208 21:57:08.151222       1 reflector.go:270] github.com/cilium/cilium/pkg/k8s/client/informers/externalversions/factory.go:115: watch of *v2.CiliumNetworkPolicy ended with: too old resource version: 1273 (3745)
log: exiting because of error: log: cannot create log: open /tmp/cilium-operator.cilium-operator-b49fd8bf7-2mgs8.unknownuser.log.WARNING.20190208-215708.1: no such file or director

Related-issues from other consumers of the client-go library have hit this too: traefik/traefik#1817

We should implement a similar fix in the cilium-operator.
@ianvernon ianvernon added kind/bug This is a bug in the Cilium logic. needs-backport/1.4 area/operator Impacts the cilium-operator component labels Feb 8, 2019
@ianvernon ianvernon self-assigned this Feb 8, 2019
@ianvernon ianvernon mentioned this issue Feb 8, 2019
Merged
@ianvernon
Copy link
Member Author

Fixed by #7007

@ianvernon ianvernon reopened this Feb 12, 2019
@ianvernon
Copy link
Member Author

Reopening; fix in #7007 was not enough.

The problem with getting this to work gracefully is that the default golang flags library does not play nicely with the pflags library / cobra. As a stop-gap, we can create the /tmp/ directory within the cilium-operator image to satisfy glog, and come up with a better long-term fix to handle how glog interacts with cilium.

@ianvernon ianvernon mentioned this issue Feb 13, 2019
Merged
ianvernon pushed a commit that referenced this issue Feb 13, 2019
cilium-operator.Dockerfile: set klog logging values from cilium-ope…
feede67 
…rator

`klog`, which is used by the Kubernetes libraries consumed by `cilium-operator`,
always tries to log to a file in `/tmp`. Since the `cilium-operator` container
image is built off of `scratch`, such a directory does not exist. If the
directory does not exist, `klog` exits and causes `cilium-operator` to crash.
To not crash, do not log to any directories, and instead set the `logtostderr`
value within `klog` to `true` so that it does not log to a directory.

Fixes: #7006

Signed-off by: Ian Vernon <ian@cilium.io>
@ianvernon
Copy link
Member Author

Update to anyone that his this issue: the issue was that I was trying to configure glog, and not klog, so the values to configure were not getting configured within the klog dependency via flag.Set. Explanation here:
#7050 (comment)

Credit to @aanm for the correct fix.

borkmann pushed a commit that referenced this issue Feb 13, 2019
@borkmann
cilium-operator.Dockerfile: set klog logging values from cilium-ope…
eee2e8d 
…rator

`klog`, which is used by the Kubernetes libraries consumed by `cilium-operator`,
always tries to log to a file in `/tmp`. Since the `cilium-operator` container
image is built off of `scratch`, such a directory does not exist. If the
directory does not exist, `klog` exits and causes `cilium-operator` to crash.
To not crash, do not log to any directories, and instead set the `logtostderr`
value within `klog` to `true` so that it does not log to a directory.

Fixes: #7006

Signed-off by: Ian Vernon <ian@cilium.io>
eloycoto pushed a commit to eloycoto/cilium that referenced this issue Feb 14, 2019
@eloycoto
cilium-operator.Dockerfile: set klog logging values from cilium-ope…
2ae09ae 
…rator

[ upstream commit eee2e8d ]

`klog`, which is used by the Kubernetes libraries consumed by `cilium-operator`,
always tries to log to a file in `/tmp`. Since the `cilium-operator` container
image is built off of `scratch`, such a directory does not exist. If the
directory does not exist, `klog` exits and causes `cilium-operator` to crash.
To not crash, do not log to any directories, and instead set the `logtostderr`
value within `klog` to `true` so that it does not log to a directory.

Fixes: cilium#7006

Signed-off by: Ian Vernon <ian@cilium.io>

Signed-off-by: Eloy Coto <eloy.coto@gmail.com>
tgraf pushed a commit that referenced this issue Feb 14, 2019
@tgraf
cilium-operator.Dockerfile: set klog logging values from cilium-ope…
532d4ed 
…rator

[ upstream commit eee2e8d ]

`klog`, which is used by the Kubernetes libraries consumed by `cilium-operator`,
always tries to log to a file in `/tmp`. Since the `cilium-operator` container
image is built off of `scratch`, such a directory does not exist. If the
directory does not exist, `klog` exits and causes `cilium-operator` to crash.
To not crash, do not log to any directories, and instead set the `logtostderr`
value within `klog` to `true` so that it does not log to a directory.

Fixes: #7006

Signed-off by: Ian Vernon <ian@cilium.io>

Signed-off-by: Eloy Coto <eloy.coto@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ianvernon ianvernon

Labels
area/operator Impacts the cilium-operator component kind/bug This is a bug in the Cilium logic.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

cilium-operator.Dockerfile: set klog logging values from cilium-operator cilium/cilium
1 participant
@ianvernon