No retry worker shutdown & fix shutdown hang #1054
Conversation
| // Bump the workflow stream with a pointless input, since if a client initiates shutdown | ||
| // and then immediately blocks waiting on a workflow activation poll, it's possible that | ||
| // there may not be any more inputs ever, and that poll will never resolve. | ||
| self.workflows.send_get_state_info_msg(); |
There was a problem hiding this comment.
This is the real fix for the shutdown hang. There was a race where "ever polled" could be set true, but nothing else ever ended up in the stream, and driving the stream with bonus messages doesn't happen until we get to shutdown itself, but that won't happen (at least in some test setups, but maybe some actual SDK loops too) until pollers have returned.
| let hb_callbacks = { | ||
| heartbeat_map_clone.read().values().cloned().collect::<Vec<_>>() | ||
| }; | ||
| for heartbeat_callback in hb_callbacks { |
There was a problem hiding this comment.
This is just some bonus cleanup to avoid holding this lock while doing the callback gathering
cretz
left a comment
cretz
left a comment
There was a problem hiding this comment.
I don't think we have seen a need for customizable retries from lang yet on any call, though it is nice that this exists now if we do get a request. Today on a per-call basis they can only opt in/out of retries and set timeouts.
What was changed
Avoid retrying worker shutdown RPCs and also fix a shutdown hang
Why?
Avoid hangs on shutdown
Checklist
Closes [Feature Request] Use shorter/lesser retry on shutdown worker call #1045
How was this tested:
Added test, this test easily reprod the shutdown hang that I think Andrew is looking into.
Any docs updates needed?