Master service fails when orchestrator is down

[juexun]

The rebooted node failed with the following message:
New IP of rebooted node: 10.233.5.244
Old IP before rebooted: 10.233.5.243
Other nodes without rebooted: 10.233.0.160 and 10.233.17.75

• Rebooted node side

2019/01/15 04:52:56 [INFO] raft: Node at 10.233.5.244:10008 [Follower] entering Follower state (Leader: "")
2019/01/15 04:52:58 [WARN] raft: Heartbeat timeout from "" reached, starting election
2019/01/15 04:52:58 [INFO] raft: Node at 10.233.5.244:10008 [Candidate] entering Candidate state
2019/01/15 04:52:58 [WARN] raft: Remote peer 10.233.0.160:10008 does not have local node 10.233.5.244:10008 as a peer
2019/01/15 04:52:58 [WARN] raft: Remote peer 10.233.17.75:10008 does not have local node 10.233.5.244:10008 as a peer

• existed nodes side

019/01/15 04:55:55 [DEBUG] raft: Failed to contact 10.233.5.243:10008 in 3m59.07956023s
2019/01/15 04:55:56 [DEBUG] raft: Failed to contact 10.233.5.243:10008 in 3m59.528575654s
2019/01/15 04:55:56 [WARN] raft: Rejecting vote request from 10.233.5.244:10008 since we have a leader: 10.233.17.75:10008

The rebooted node failed to join the existed cluster because

• the ip of rebooted node had changed
• other nodes keep the old ip of rebooted node

[AMecea]

Hi @juexun, This issue is duplicate of #107. The issue is because orchestrator does not refresh the IPs, as you observed. What I recommend is to use only one node for orchestrator until #107 is fixed.

[juexun]

@AMecea , Thanks.

[juexun]

If this issue can not fixed, the orchestrator will be the SOPF of the cluster. The master service of mysql is unreachable if orchestrator died