TiCDC cluster suffers a round robin owner election during rolling update

[amyangfei]

What did you do?

1. Create a TiCDC cluster with multiple nodes, such as 7 nodes.
2. Rolling update the TiCDC cluster

What did you expect to see?

Replication continues normally during TiCDC is rolling update

What did you see instead?

Supposing the owner is restarted at first, then owner will be elected to each following TiCDC node(This is caused by the election way in etcd, it simply selects the election key with the smallest revision as the campaign winner), while the elected owner will be restarted soon by rolling update.

The initialization phase of a TiCDC owner could cost long time, it has many procedures, including initializing each existing changefeeds (when initializing a changefeed it will create a downstream sink, imaging we create a Kafka sink and do some verification jobs, it is heavy work).

Then we will waste a lot of time in each TiCDC owner node to do owner initialization. What's more, maybe no owner finishes initialization before it restarts, the replication checkpoint could pause during rolling update, and the longer rolling update takes, the larger replication lag may happen.

Versions of the cluster

Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

v5.3.0

TiCDC version (execute cdc version):

master@pingcap/ticdc@fe92b89

Brainstorming

• When processing rolling update, update the TiCDC owner at last, then the owner will be transferred only once, and before the last node is restarted, the owner works as normal. (This may be simple, just change the deploy tools, but may be not reasonable, why should the deploy tool care about which node is TiCDC owner?)
• Separate independent owner nodes, as what DM-master does in DM. We can update owner nodes first, then processor nodes. (This changes existing architecture of TiCDC)
• Add an API to notify TiCDC cluster that it will go into a rolling update mode. When TiCDC enters rolling update mode, it will have special logic to avoid too much replication lag. (But this is too tricky)
• Introduce a more intelligent leader election strategy? (Then we have to deal with consensus, it is complex)

[amyangfei]

After discussion, we decide to make this a feature, for two reasons

• Current behavior doesn't break TiCDC boundary (in extreme scenarios, there checkpoint lag could reach 10 minutes)
• In the short term, we tend to update the rolling update strategy in TiUP or TiDB-operator to restart TiCDC owner at last.

[overvenus]

Since this issue doesn't break TiCDC boundary, change to severity/moderate.

[3AceShowHand]

pingcap/tiup#1972

This is solved by supporting upgrade the owner at the last strategy in the PR above.