[NEW] Support redis cluster mode

[ShooterIT]

1 Background

Currently, we discard the support for codis mode, if we want to deploy kvrocks cluster, we need to pre-shard for cluster and deploy a proxy like twemproxy to route requests. This cluster solution also is adopted when redis doesn't support cluster mode, but it is not convenient for failover, linear scale, management, moreover, we must route requests by a proxy, that must cause performance loss because of extra network communication. So we hope there a way we can implement a feature that is similar with redis cluster mode.

But if we totally support redis cluster mode in kvrocks, it doesn't seem a good idea, because it is hard to implement redis cluster management and gossip protocol without bugs. We must upgrade server if we find bugs, after all, it costs many years to make redis cluster work well. In order to get benefits from redis cluster mode, we can only support partial redis cluster commands that make a chance to let redis cluster sdk(such as jedis) or redis cluster proxy(such as redis-cluster-proxy, predixy) work well.

2 Implementation

2.1 Basic version

Firstly, every node needs to know the entire cluster topology, itself identifier. When clients(redis-cli, SDK or proxy) connect to arbitrary one node of kvrocks cluster, they could know slot distribution of cluster by sending CLUSTER NODES command, then route requests. And the kvrocks should reply MOVED when clients send requests to wrong server. To do that, we should implement CLUSTER SETNODES command to set the entire cluster topology for every nodes, and provide CLUSTER NODES command to make kvrocks node similar with redis.

• CLUSTER NODES
  The return of this command is the same format of redis cluster.
  Example:

  07c37dfeb235213a872192d90877d0cd55635b91 127.0.0.1:30004 slave e7d1eecce10fd6bb5eb35b9f99a514335d9ba9ca 0 1426238317239 4 connected
  67ed2db8d677e59ec4a4cefb06858cf2a1a89fa1 127.0.0.1:30002 master - 0 1426238316232 2 connected 5461-10922
  

• CLUSTER SETNODES $ALL_NODES_INFO $VERSION FORCE
$ALL_NODES_INFO format: $node_id $ip $port $role $master_node_id $slot_range. $role: master or slave, $master_node_id: master_node_id if it is slave, otherwise, this value is -. slave needn't set $slot_range.
  Example:

  07c37dfeb235213a872192d90877d0cd55635b91 127.0.0.1 30004 slave e7d1eecce10fd6bb5eb35b9f99a514335d9ba9ca
  67ed2db8d677e59ec4a4cefb06858cf2a1a89fa1 127.0.0.1 30002 master - 5461-10922
  

  $VERSION means current cluster topology version that is stored or generated by management component such as config server or meta-server.
  Kvrocks can know itself node id by compared with itself ip and port in received all nodes info

From now, it works well, we just tell every node of kvrocks cluster by CLUSTER SETNODES when scale(slot data migration) or failover. Clients(redis-cli, sdk, proxy) will reacquire cluster topology when they finds requests are rejected or MOVED by using old topology to route requests. To avoid missing notify nodes new topology, we can set new cluster topology every longer interval.

*Please notice that: the cluster solution for kvrocks doesn't handle scale or failover.

2.1 Advanced version

We find we always tell every node entire cluster topology even just changing one node info. If the cluster is very large, the message will be big, so we also fall into the damage of network communication, and cluster management component may can't support that. So we may need to support incremental changing notification. When changing topology, we only send one light command to change the topology in kvrocks server. To implement incremental changing notification, we must use version to identify current topology version, otherwise, kvrocks may keep a wrong cluster topology if missing one changing topology message. Kvrocks only supports to apply one command content when its version is valid by compared with the $version in command, if not, kvrocks must return an error, and we will set new entire cluster topology by CLUSTER SETNODES command.

There are some commands we may need to support.

• (optional) CLUSTER VERSION
  Get version of node, so that we can check its' version is right or not.
• (optional) CLUSTER SETNODEID $NODE_ID
  Explicitly tell the node its node id.
• (optional) CLUSTER SETNODE $NODE_ID $NODE_INFO $VERSION
  Only update one node info
• (optional) CLUSTER DELNODE $NODE_ID $VERSION
  Delete one node of cluster
• (optional) CLUSTER REPLICATE $NODE_ID
  Just like slaveof, similar with redis command.
• (optional) CLUSTER MIGRATE $SLOT_ID $NODE_ID
  Migrate all keys belonging the slot to another node
• (optional) CLUSTER SETSLOT $SLOT_ID $NODE_ID $VERSION
  Only update one slot distribution, that is useful to avoid too many big messages when we scale cluster.
• (optional) CLUSTER SLOTS/INFO/KEYSLOT/GETKEYSINSLOT/COUNTKEYSINSLOT
  To make kvrocks cluster similar with redis cluster

3 Advantages and disadvantages

3.1 Advantages

• It is easy to implement and make less bugs.
• Cluster logic are removed from server, so that we needn't change kvrocks code when optimizing fault check or failover for kvrocks instances.

3.2 Disadvantages

• Topology changing notifications are sent by other component, clients may feel latency when cluster topology change, even although that is low-frequency.
• It is not perfect, we still need a cluster management component to manage cluster such as failover and scale.

4 Extra works

• More redis cluster commands to manage kvrocks cluster conveniently just like redis cluster.
• Cluster management component. To make kvrocks cluster easy, we may provide a cluster management component to deal with linear scale, failover, or management.

[ShooterIT]

Reopen for unfinished extra work

[tisonkun]

@ShooterIT can you list out the pending works to complete this issue? Or superseding this one with a new issue describing the status quo.

[ShooterIT]

Hi @tisonkun i think we can close it, kvrocks has supported slot-based data migration, and kvrocks controller already can be used. Thanks for your attention.