improve resource.WaitForState and add refreshGracePeriod #13778
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Make sure that we can cancel the WaitForState refresh loop when reaching a timeout, otherwise it may run indefinitely. There's no need to try and store and read the Result concurrently, just pass the value over a channel.
This test unfortunately relies on the timing of the loops in WaitForState, and the text of the error message. Adjust the timing so the timeout isn't an even multiple of the poll interval, and make sure we reach a minimum number of retries.
apparentlymart
approved these changes
Apr 19, 2017
There was a problem hiding this comment.
I'm a bit hesitant here just because it seems like this bit of code has lots of complex interactions and my mental model of how things fit together here is still lacking, but it looks like the code does what the comment says it should, and in principle what you described sounds reasonable, so I think that's as good as I'm going to be able to get here... 👍
jbardin
force-pushed
the
jbardin/GH-13617
branch
from
a0fe0cd
to
14bea66
Compare
Refresh calls may have side effects that need to be recorded if it succeeds, especially common when when WaitForState is called from resource.Retry. If the WaitForState timeout is reached and there is a Refresh call in-flight, wait up to refreshGracePeriod (set to 30s) for it to complete.
A couple tests require lowering the grace period to keep the test from taking the full 30s timeout. The Retry_hang test also needed to be removed from the Parallel group, becuase it modifies the global refreshGracePeriod variable.
|
I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further. |
ghost
locked and limited conversation to collaborators
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The Refresh goroutine in WaitForState was never being canceled, and could end up running until the entire process exited. Because of this situation, successful calls to refresh could happen long after the timeout is reached. This can be a serious problem when the Refresh call has side effects that need to be recorded (as is often the case when called through
resource.Retry).Start by making the goroutine properly cancellable, returning immediately during the wait period.
Once the refresh goroutine can be cancelled, the case where there is a Refresh still in-flight needs to be taken care of. Because Refresh can't be cancelled directly, all we can do is wait and hope it returns in a reasonable amount of time.
Add a grace period after the timeout elapses, to wait for the function to return.
Fixes #13617