[TiCDC] (unreleased feature) High inter-node latency causes ErrProcessorDuplicateOperations

[liuzix]

• When the new scheduler is enabled, high inter-node latency can cause ErrProcessorDuplicateOperations.
• Scenario: 100K tables with 8 TiCDC nodes. Injects a latency of 2000ms jitter 10ms between TiCDC nodes. Then restarts all TiCDC nodes all at once.

[
  {
    "id": "test-1",
    "summary": {
      "state": "error",
      "tso": 431564666934132811,
      "checkpoint": "2022-03-03 14:22:34.894",
      "error": {
        "addr": "new-sched-test-cdc-3.liuzixiong.tispace:8300",
        "code": "CDC:ErrProcessorDuplicateOperations",
        "message": "[CDC:ErrProcessorDuplicateOperations]table processor duplicate operation, table-id: 26194"
      }
    }
  }
]

• Root cause: Processor may send Sync to Owner to tell the owner about the processor's internal state. However, the internal state can be updated by a in-flight owner's dispatch command, making the owner's knowledge of the processor' state inconsistent.
• Severity: (1) This problem rarely occurs if the latency is low (sub 100ms) (2) This problem pauses the changefeed, but the changefeed can retry and will eventually succeed if the latency recovers to normal. (3) This problem will not cause data corruption.