Cherry-pick #6504 to 6.2: Fix infinite failure on Kubernetes watch #6530
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
(cherry picked from commit a44818c)
exekias
force-pushed
the
backport_6504_6.2
branch
from
f779541 to
6d6d7c6
Compare
leweafan
pushed a commit
to leweafan/beats
that referenced
this pull request
Apr 28, 2023
…atch (elastic#6530) Cherry-pick of PR elastic#6504 to 6.2 branch. Original message: This PR fixes elastic#6503 How to reproduce: Run filebeat pointing to minikube. ``` minikube ssh sudo su ps aux | grep localkube kill -9 process_id ``` This will force a failure on the API server, and when the API server comes back up it will not be able to serve up the last resource version that we had requested with the failure: ``` type:"ERROR" object:<raw:"k8s\000\n\014\n\002v1\022\006Status\022C\n\004\n\000\022\000\022\007Failure\032)too old resource version: 310742 (310895)\"\004Gone0\232\003\032\000\"\000" > typeMeta:<apiVersion:"v1" kind:"Status" > raw:"\n\004\n\000\022\000\022\007Failure\032)too old resource version: 310742 (310895)\"\004Gone0\232\003" contentEncoding:"" contentType:"" <nil> ``` In such scenarios the only mitigation would be to move the resource version to the latest. Scenarios like this would be addressed by `client-go`. The reason why the code fails with error is because we pass a `Pod` resource to do the `watcher.Next()` in this scenario the resource that is attempted to be parsed is an `Error` resource and the protobuf unmarshalling fails. This is a limitation in the client that we use as the resource needs to be passed explicitly. This fix is not the best in the world as it might miss few state changes.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Cherry-pick of PR #6504 to 6.2 branch. Original message:
This PR fixes #6503
How to reproduce: Run filebeat pointing to minikube.
This will force a failure on the API server, and when the API server comes back up it will not be able to serve up the last resource version that we had requested with the failure:
In such scenarios the only mitigation would be to move the resource version to the latest. Scenarios like this would be addressed by
client-go. The reason why the code fails with error is because we pass aPodresource to do thewatcher.Next()in this scenario the resource that is attempted to be parsed is anErrorresource and the protobuf unmarshalling fails. This is a limitation in the client that we use as the resource needs to be passed explicitly.This fix is not the best in the world as it might miss few state changes.