Fix infinite failure on Kubernetes watch #6504
Conversation
|
Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually? |
|
@exekias i stole the binary back off method from your attempt to fix this. Apologies for that :) |
| if !(err == io.EOF || err == io.ErrUnexpectedEOF) { | ||
| // This is an error event which can be recovered by moving to the latest resource verison | ||
| logp.Info("kubernetes: Ignoring event, moving to most recent resource version") | ||
| w.lastResourceVersion = "" |
There was a problem hiding this comment.
We may need to resync the stored objects, e.g. what would happen if we lose a delete event while we reconnect?
There was a problem hiding this comment.
I've been doing some tests. At least when I reproduce this last resource version is increased by several hundred while no real change happened.
I agree with you @jsoriano, a resync may be needed to guarantee no events are missed during the reconnect.
Still, taking into account the severity of this issue, I'm ok with merging + backporting this as it is, as it improves the current situation (infinite loop + crash). I can do some more research and come up with a resync mechanism afterwards.
What do you think?
There was a problem hiding this comment.
i think we are good to merge this one in.
exekias
added
bug
libbeat
needs_backport
|
jenkins, test this please |
|
+1 the reconciliation might be a bigger change as we need to do some state management. we can unblock customers and fix that one in a subsequent PR. |
|
@exekias based on this comment, it seems to be normal: |
|
Failing test is not related to this change |
exekias
removed
the
needs_backport
(cherry picked from commit a44818c)
…6530) Cherry-pick of PR #6504 to 6.2 branch. Original message: This PR fixes #6503 How to reproduce: Run filebeat pointing to minikube. ``` minikube ssh sudo su ps aux | grep localkube kill -9 process_id ``` This will force a failure on the API server, and when the API server comes back up it will not be able to serve up the last resource version that we had requested with the failure: ``` type:"ERROR" object:<raw:"k8s\000\n\014\n\002v1\022\006Status\022C\n\004\n\000\022\000\022\007Failure\032)too old resource version: 310742 (310895)\"\004Gone0\232\003\032\000\"\000" > typeMeta:<apiVersion:"v1" kind:"Status" > raw:"\n\004\n\000\022\000\022\007Failure\032)too old resource version: 310742 (310895)\"\004Gone0\232\003" contentEncoding:"" contentType:"" <nil> ``` In such scenarios the only mitigation would be to move the resource version to the latest. Scenarios like this would be addressed by `client-go`. The reason why the code fails with error is because we pass a `Pod` resource to do the `watcher.Next()` in this scenario the resource that is attempted to be parsed is an `Error` resource and the protobuf unmarshalling fails. This is a limitation in the client that we use as the resource needs to be passed explicitly. This fix is not the best in the world as it might miss few state changes.
…atch (elastic#6530) Cherry-pick of PR elastic#6504 to 6.2 branch. Original message: This PR fixes elastic#6503 How to reproduce: Run filebeat pointing to minikube. ``` minikube ssh sudo su ps aux | grep localkube kill -9 process_id ``` This will force a failure on the API server, and when the API server comes back up it will not be able to serve up the last resource version that we had requested with the failure: ``` type:"ERROR" object:<raw:"k8s\000\n\014\n\002v1\022\006Status\022C\n\004\n\000\022\000\022\007Failure\032)too old resource version: 310742 (310895)\"\004Gone0\232\003\032\000\"\000" > typeMeta:<apiVersion:"v1" kind:"Status" > raw:"\n\004\n\000\022\000\022\007Failure\032)too old resource version: 310742 (310895)\"\004Gone0\232\003" contentEncoding:"" contentType:"" <nil> ``` In such scenarios the only mitigation would be to move the resource version to the latest. Scenarios like this would be addressed by `client-go`. The reason why the code fails with error is because we pass a `Pod` resource to do the `watcher.Next()` in this scenario the resource that is attempted to be parsed is an `Error` resource and the protobuf unmarshalling fails. This is a limitation in the client that we use as the resource needs to be passed explicitly. This fix is not the best in the world as it might miss few state changes.
This PR fixes #6503
How to reproduce: Run filebeat pointing to minikube.
This will force a failure on the API server, and when the API server comes back up it will not be able to serve up the last resource version that we had requested with the failure:
In such scenarios the only mitigation would be to move the resource version to the latest. Scenarios like this would be addressed by
client-go. The reason why the code fails with error is because we pass aPodresource to do thewatcher.Next()in this scenario the resource that is attempted to be parsed is anErrorresource and the protobuf unmarshalling fails. This is a limitation in the client that we use as the resource needs to be passed explicitly.This fix is not the best in the world as it might miss few state changes.