Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
ensures failure to start reflectors is fatal, rather than resulting in permanent hangs waiting for first_load_future
I believe this closes #627 (it at least explains one such event).
Relevant bits:
ensure_future
. Exceptions on this future are ignored.All these combined mean that if the first load fails, the exception is ignored (it will be logged by asyncio with
Task exception was never retrieved
), and all subsequent calls wait forever instead of raising, never attempting to restart the reflector.The events described in #627 are:
iohttp.client_exceptions.ClientConnectorError
)Pods reflector failed, halting Hub
aiohttp.client_exceptions.ClientConnectorError
, but this time it's ignored (ERROR:asyncio:Task exception was never retrieved
)The fix here is to:
Reflector.start()
, andon_reflector_failure
functionfirst_load_future
so that anyone awaiting the first load will get the exception, rather than a forever-incomplete FutureThe result should be that the Hub goes into CrashLoopBackoff while it can't talk to the k8s API, which I think is probably correct.
We could also give the first load more retry chances with
exponential_backoff
before it considers errors fatal, but I'm not sure if that's the right thing to do.