Bugfix: Recreate activity heartbeat timeout after first timer fire #658
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
yiminc-zz
reviewed
Apr 5, 2018
| @@ -303,7 +303,7 @@ func (tb *timerBuilder) loadActivityTimers(msBuilder *mutableStateBuilder) { | |||
| td := &timerDetails{ | |||
| TimerSequenceID: TimerSequenceID{VisibilityTimestamp: heartBeatExpiry}, | |||
| ActivityID: v.ScheduleID, | |||
| EventID: v.StartedID, | |||
| EventID: v.ScheduleID, | |||
| TimeoutType: w.TimeoutTypeHeartbeat, | |||
| TimeoutSec: v.HeartbeatTimeout, | |||
| TaskCreated: (v.TimerTaskStatus & TimerTaskStatusCreatedHeartbeat) != 0} | |||
There was a problem hiding this comment.
The problem is after first heartbeat timer is created, the bit for heartbeat timer will be set, and after that this TaskCreated will always be true so firstActivityTimerTask() will not return the timer task and it won't be created.
There was a problem hiding this comment.
@yiminc
if my understanding is correct, https://github.com/uber/cadence/blob/master/service/history/timerQueueActiveProcessor.go#L307 will take care of that?
There was a problem hiding this comment.
i actually have some doubt:
here: https://github.com/uber/cadence/pull/658/files#diff-b3d25d1c01ad6e7393fd3958caea1333R287
the start event ID is actually checked
so when the timer detail is created (by loading activity into memory), we are sure that the start event ID is a valid event ID
There was a problem hiding this comment.
The problem is the startedID is buffered event id which changed to real start event after it is flushed.
yiminc-zz
approved these changes
Apr 5, 2018
| @@ -303,7 +303,7 @@ func (tb *timerBuilder) loadActivityTimers(msBuilder *mutableStateBuilder) { | |||
| td := &timerDetails{ | |||
| TimerSequenceID: TimerSequenceID{VisibilityTimestamp: heartBeatExpiry}, | |||
| ActivityID: v.ScheduleID, | |||
| EventID: v.StartedID, | |||
| EventID: v.ScheduleID, | |||
| TimeoutType: w.TimeoutTypeHeartbeat, | |||
| TimeoutSec: v.HeartbeatTimeout, | |||
| TaskCreated: (v.TimerTaskStatus & TimerTaskStatusCreatedHeartbeat) != 0} | |||
wxing1292
approved these changes
Apr 5, 2018
meiliang86
pushed a commit
that referenced
this pull request
Apr 10, 2018
) When the heartbeat timeout is created for the very first time it uses StartEventID as the dedupe event ID. But it is possible that it is a buffered event and no ID is specified when the timer task is created. This trips the recreation logic from recreating the timer. This change instead relies on the schedule ID for the event for dedupe event ID of the heartbeat timer.
meiliang86
pushed a commit
that referenced
this pull request
Apr 10, 2018
) When the heartbeat timeout is created for the very first time it uses StartEventID as the dedupe event ID. But it is possible that it is a buffered event and no ID is specified when the timer task is created. This trips the recreation logic from recreating the timer. This change instead relies on the schedule ID for the event for dedupe event ID of the heartbeat timer.
nathanboktae
pushed a commit
that referenced
this pull request
Apr 12, 2018
) When the heartbeat timeout is created for the very first time it uses StartEventID as the dedupe event ID. But it is possible that it is a buffered event and no ID is specified when the timer task is created. This trips the recreation logic from recreating the timer. This change instead relies on the schedule ID for the event for dedupe event ID of the heartbeat timer.
nathanboktae
added a commit
that referenced
this pull request
Apr 12, 2018
* bump cadence-web to 1.1.1 * bugfix: when multiple activity got timeouted, there will be at most one being actually deleted in Cassandra (#655) * Bugfix: Recreate activity heartbeat timeout after first timer fire (#658) When the heartbeat timeout is created for the very first time it uses StartEventID as the dedupe event ID. But it is possible that it is a buffered event and no ID is specified when the timer task is created. This trips the recreation logic from recreating the timer. This change instead relies on the schedule ID for the event for dedupe event ID of the heartbeat timer. * Bump cadence web to 1.2.0
samarabbas
added a commit
to samarabbas/cadence
that referenced
this pull request
Apr 16, 2018
PR cadence-workflow#658 fixes an issue with heartbeat timers which require us to use activity scheduleID in the timertask. This change modifies the heartbeat timer creation check to account for incorrect heartbeat timer tasks created before the bugfix.
samarabbas
added a commit
that referenced
this pull request
Apr 16, 2018
) PR #658 fixes an issue with heartbeat timers which require us to use activity scheduleID in the timertask. This change modifies the heartbeat timer creation check to account for incorrect heartbeat timer tasks created before the bugfix.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When the heartbeat timeout is created for the very first time it
uses StartEventID as the dedupe event ID. But it is possible that
it is a buffered event and no ID is specified when the timer task
is created. This trips the recreation logic from recreating the
timer.
This change instead relies on the schedule ID for the event
for dedupe event ID of the heartbeat timer.