Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Invoker doesn't recognize pods stuck in "Pending" state #7

Open
shgriffi opened this issue Mar 14, 2018 · 0 comments
Open

Invoker doesn't recognize pods stuck in "Pending" state #7

shgriffi opened this issue Mar 14, 2018 · 0 comments

Comments

@shgriffi
Copy link

We had an issue in our deployment where the infrastructure ran out of resources. As a result, pods could not be created and all new pods were stuck in the "pending" state. The invoker log showed the following error:

[2018-03-14T09:21:45.694Z] [ERROR] [#sid_121] [KubernetesClient] Failed create pod for 'wskinvoker-00-54-whisksystem-invokerhealthtestaction0': class io.fabric8.kubernetes.client.KubernetesClientTimeoutException - Timed out waiting for [60000] milliseconds for [Pod] with name:[wskinvoker-00-54-whisksystem-invokerhealthtestaction0] in namespace [dh-stage-ow].

There was never an attempt for the invoker to kill the pending pod. This resulted in hundreds of pods left in the "pending" state and required me to manually clean them up.

If a pod times out during creation, the invoker should attempt to kill that pod to avoid leaving orphaned pods.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant