[server] Coordinator Server Supports High-Available

[zcoo]

Purpose

Coordinator Server Supports High-Available.

Linked issue: close #188

Brief change log

Tests

see : com.alibaba.fluss.server.coordinator.CoordinatorServerElectionTest

API and Format

Documentation

Currently I make it by using an active (or leader) coordinator server and several standby (or alive) coordinator server. When several coordinator servers start up at the same time, only one can successfully preempt the ZK node and win the election and become active coordinator server leader. Once it failovers, the standby servers will take over it with an leader election process.

[zcoo]

not ready for review

[michaelkoepf]

@zcoo if it is not ready for review, you can click on Convert to draft in the right side bar, and, optionally, add [WIP] as prefix in the title of the PR.

[zcoo]

@michaelkoepf i get it, thank you

[zcoo]

Ready for review now! @wuchong @swuferhong

[LiebingYu]

I have a question here. Do we need to properly handle coordinatorEpoch and coordinatorEpochZkVersion in this PR, just like Kafka does? In my opinion:

• coordinatorEpoch prevents TabletServers from processing requests from zombie CoordinatorServers in the event of a split-brain scenario among CoordinatorServers.
• coordinatorEpochZkVersion(KAFKA-6082) prevents zombie CoordinatorServers from modifying the ZK state during a split-brain scenario.

Should we take these two concerns into consideration in the same way as Kafka?

[zcoo]

I have a question here. Do we need to properly handle coordinatorEpoch and coordinatorEpochZkVersion in this PR, just like Kafka does? In my opinion:

• coordinatorEpoch prevents TabletServers from processing requests from zombie CoordinatorServers in the event of a split-brain scenario among CoordinatorServers.
• coordinatorEpochZkVersion(KAFKA-6082) prevents zombie CoordinatorServers from modifying the ZK state during a split-brain scenario.

Should we take these two concerns into consideration in the same way as Kafka?

Thanks for reminder. I considered coordinatorEpoch and coordinatorEpochZkVersion , but ultimately did not implement them in this PR to simplify its complexity. I think we can handle it later.

Do you have any suggestions?
@LB-Yu @wuchong @swuferhong

[LiebingYu]

I have a question here. Do we need to properly handle coordinatorEpoch and coordinatorEpochZkVersion in this PR, just like Kafka does? In my opinion:

• coordinatorEpoch prevents TabletServers from processing requests from zombie CoordinatorServers in the event of a split-brain scenario among CoordinatorServers.
• coordinatorEpochZkVersion(KAFKA-6082) prevents zombie CoordinatorServers from modifying the ZK state during a split-brain scenario.

Should we take these two concerns into consideration in the same way as Kafka?

Thanks for reminder. I considered coordinatorEpoch and coordinatorEpochZkVersion , but ultimately did not implement them in this PR to simplify its complexity. I think we can handle it later.

Do you have any suggestions? @LB-Yu @wuchong @swuferhong

IMO, we can split them into different PRs, but they should be merged together? Otherwise, we might introduce other metadata inconsistency issues when introducing HA. What's your opinion?

[LiebingYu]

Why we don't use LeaderLatch here if we need to hold the leadership?

[zcoo]

Very good suggestion to me! It seems LeaderLatch is more suitable in this scene and I will try to use it instead

[LiebingYu]

This logic looks strange to me. Why do we explicitly create the path for the first election, but use LeaderLatch for the standby Coordinator? Since we’re already using the framework, perhaps we could standardize on LeaderLatch?

Also, this logic seems problematic. If cs-1 becomes the leader on the first startup and later loses leadership due to a network issue, it appears it will never participate in the election again, because no LeaderLatch was registered for the node that became leader during the first startup.

[zcoo]

yes I think we don't need to use special logic in first round election. I have changed it.

[LiebingYu]

Hi @zcoo. Overall, I have some concerns about several issues:

1. The mechanisms for leader election, startup, and resignation are not sufficiently robust. For reference, Kafka Controller provides a comprehensive implementation for controller lifecycle management:

Startup:
  Register ControllerChangeHandler and check whether the /controller path exists
  elect()
    Get activeControllerId from ZK
    If activeControllerId != -1, it means another broker has already become the controller, so just return
    Attempt to register as controller in ZK: registerControllerAndIncrementControllerEpoch
    If successful: call onControllerFailover()
    If failed:
      If it’s ControllerMovedException, call maybeResign (election failed)
      Otherwise, call triggerControllerMove (startup process failed)

When /controller node is created or its data changes:
  Call maybeResign()

When /controller node is deleted:
  processReelect
    maybeResign
    elect

2. There is no management of epochs. Combined with the lack of a leader resignation mechanism, this becomes extremely risky in the case of split-brain scenarios. There is the possibility that two leaders could be working simultaneously, leading to metadata inconsistencies.

3. After introducing multiple coordinators, should we re-examine potential race conditions in metadata management and ZK access to avoid the risks associated with split-brain situations?

[zcoo]

@LiebingYu Sure. I think we need to these things and I will introduce epoc mechanism in this PR soon