history service should do event reordering making sure corresponding …

[wxing1292]

…events for decision will have exactly the same order and no irrelevant event will be inserted in between, so client can predict the event ID of a corresponding decision.

tested using canary

counterpart: #600

[coveralls]

Coverage increased (+0.1%) to 64.161% when pulling a67b4e7 on decision-order-2 into 35bf3de on master.

[samarabbas]

remove this.

[samarabbas]

Is removing this safe? This changes the logic.

[wxing1292]

from what i see in the code logic
prev:
if no decision on the fly, just assign the event ID
if decision on the fly, postpone until decision is done.

now:
if events in the list, assign event ID
if events not in the list, assign event ID when there is no decision on the fly, by FlushBufferedEvents():
https://github.com/uber/cadence/blob/master/service/history/mutableStateBuilder.go#L157

So let us consider several cases:

1. no decision on the fly
   prev: event ID will be assign immediately
   now: events in the list will be assigned an ID immediately, events not in the list will be assigned an ID when FlushBufferedEvents is called, which is before persistence of mutable state.

2. decision on the fly
   prev: only several event will assign ID immediately
   now: in addition, other events will be assign ID immediately:
   category a:
   EventTypeDecisionTaskScheduled
   EventTypeDecisionTaskStarted
   EventTypeWorkflowExecutionStarted
   category b:
   EventTypeActivityTaskScheduled
   EventTypeActivityTaskCancelRequested
   EventTypeTimerStarted
   EventTypeTimerCanceled
   EventTypeCancelTimerFailed
   EventTypeRequestCancelExternalWorkflowExecutionInitiated
   EventTypeMarkerRecorded
   EventTypeStartChildWorkflowExecutionInitiated
   EventTypeSignalExternalWorkflowExecutionInitiated

category a is obvious
category b is the events generated by decision, which can only happen when a decision is complete.

so overall i think it is safe.

[samarabbas]

Let's hold onto this change until we cut a release.