Bugfix/mw/keepalive timer from interval to send after 1.4 [JIRA: RIAK-1888] #683
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This resolves an issue where the provider process could become blocked or stuck for an extended period of time, only to receive many pointless messages from the timer system. First change is to up the keepalive from 1 second to 60. This should be enough for the remote end to realize if the socket connection has died, but not so much as to be spammy. The second, more invasive change, is to use the erlang:send_after/3 reset pattern rather than timer:send_interval/3. By using the erlang:send_after, messages will not be sent if the process becomes stuck, as no code capable of resetting the timer will run. Secondly, the return values of the erlang:send_after and erlang:cancel_timer don't need to be pattern matched to ensure correctness (since the former returns the reference itself, and the latter returns an integer or false).
Based on feedback, resetting the keep-alive timer on each socket message may not be strictly needed. We just need some data every 60 seconds or so to ensure a node is alive (rather than the tcp timeout which could be much longer). An interval is still not used (reasons in previous commit still stand), so instead it's just re-creating the timer on each keepalive.
We never cancel it, nor refer to the timer in the code, ever. One could flood the process by manually sending 'keepalive' to it, but if someone is doing that, you've got bigger problems. This means that now the timer is like a send interval that only intervals if the process is not stuck, and does not add a timer on proxy get requests.
|
@bors retry Unrelated failure |
jonmeredith
pushed a commit
that referenced
this pull request
May 20, 2015
…erval-to-send-after-1.4 Bugfix/mw/keepalive timer from interval to send after 1.4 Good effort bors, merging manually as eunit not reliable enough back in prehistory.
|
Quick test - mention of #680 (RIAK-1507) again to see if annotated with JIRA number. |
|
Also - does it work the other way - RIAK-1507 (#680) |
jonmeredith
pushed a commit
that referenced
this pull request
Jun 23, 2015
…mer-from-interval-to-send-after-1.4 Bugfix/mw/keepalive timer from interval to send after 1.4 Good effort bors, merging manually as eunit not reliable enough back in prehistory. (cherry picked from commit cf1e203)
Basho-JIRA
changed the title
jonmeredith
pushed a commit
that referenced
this pull request
Jun 24, 2015
…om-interval-to-send-after-2.0 Backport of merge pull request #683 from basho/bugfix/mw/keepalive-ti… bors in poor state due to 2.0.6 reversioning - merging manually.
|
Merged in the fix. _[posted via JIRA by Jon Meredith]_ |
Merged
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
#680 (RIAK-1507) applied to 1.4 series for customer.