Redis Cluster (4.0)

lettuce provides a dedicated client for Redis Cluster (since 3.0)

The clustering support covers:

• Support of all CLUSTER commands
• Command routing based on the hash slot of the commands' key
• High-level abstraction for selected cluster commands
• Execution of commands on multiple cluster nodes
• MOVED and ASK redirection handling
• Obtaining direct connections to cluster nodes by slot and host/port (since 3.3)
• SSL and authentication (since 4.2)
• Periodic and adaptive cluster topology updates
• Publish/Subscribe

Connecting to a Redis Cluster requires one or more initial seed nodes. The full cluster topology view (partitions) is obtained on the first connection so you're not required to specify all cluster nodes. Specifying multiple seed nodes helps to improve resiliency as lettuce is able to connect the cluster even if a seed node is not available. Lettuce holds multiple connections, which are opened on demand. You are free to operate on these connections. Connections can be bound to specific hosts or nodeIds. Connections bound to a nodeId will always stick to the nodeId, even if the nodeId is handled by a different host. Requests to unknown nodeId's or host/ports that are not part of the cluster are rejected. Do not close the connections. Otherwise, unpredictable behavior will occur. Keep also in mind that the node connections are used by the cluster connection itself to perform cluster operations: If you block one connection all other users of the cluster connection might be affected.

Command routing

The concept of Redis Cluster bases on sharding. Every master node within the cluster handles one or more slots. Slots are the unit of sharding and calculated from the commands' key using CRC16 MOD 16384. Hash slots can also be specified using hash tags such as {user:1000}.foo.

Every request, which incorporates at least one key is routed based on its hash slot to the corresponding node. Commands without a key are executed on the default connection that points most likely to the first provided RedisURI. The same rule applies to commands operating on multiple keys but with the limitation that all keys have to be in the same slot. Commands operating on multiple slots will be terminated with a CROSSSLOT error.

Cross-slot command execution/High-level abstraction for some cluster commands

Regular Redis Cluster commands are limited to single-slot keys, basically either single key commands or multi-key commands that share the same hash slot of their keys.

The cross slot limitation can be mitigated by using the advanced cluster API for some multi-key commands. Commands that operate on keys with different slots are decomposed into multiple commands. The single commands are fired in a fork/join fashion. The commands are issued concurrently to avoid synchronous chaining. Results are synchronized before the command is completed (from a user perspective).

Following commands are supported for cross-slot command execution:

• DEL: Delete the KEYs from the affected cluster. Returns the number of keys that were removed
• MGET: Get the values of all given KEYs. Returns the values in the order of the keys.
• MSET: Set multiple key/value pairs for all given KEYs. Returns always OK.
• CLIENT SETNAME: Set the client name on all known cluster node connections. Returns always OK.
• FLUSHALL: Flush all data on the cluster masters. Returns always OK.
• FLUSHDB: Flush all data on the cluster masters. Returns always OK.
• DBSIZE: Return the number of keys that are stored on all masters.
• KEYS: Return/Stream all keys that are stored on all masters.
• RANDOMKEY: Return a random key from a random master.
• SCRIPT FLUSH: Remove all the scripts from the script cache on all cluster nodes.
• SCRIPT KILL: Kill the script currently in execution on all cluster nodes. This call does not fail even if no scripts are running.
• SHUTDOWN: Synchronously save the dataset to disk and then shut down all nodes of the cluster.

Cross-slot command execution is available on the following APIs:

• RedisAdvancedClusterCommands
• RedisAdvancedClusterAsyncCommands
• RedisAdvancedClusterReactiveCommands

Execution of commands on one or multiple cluster nodes

Sometimes commands have to be executed on multiple cluster nodes. The advanced cluster API allows to select a set of nodes (e.g. all masters, all slaves) and trigger a command on this set.

RedisAdvancedClusterAsyncCommands<String, String> async = clusterClient.connect().async();
AsyncNodeSelection<String, String> slaves = connection.slaves();

AsyncExecutions<List<String>> executions = slaves.commands().keys("*");
executions.forEach(result -> result.thenAccept(keys -> System.out.println(keys)));

The commands are triggered concurrently. This API is currently only available for async commands. Commands are dispatched to the nodes within the selection, the result (CompletionStage) is available through AsyncExecutions.

A node selection can be either dynamic or static. A dynamic node selection updates its node set upon a cluster topology view refresh. Node selections can be constructed by the following presets:

• masters
• slaves (operate on connections with activated READONLY mode)
• all nodes

A custom selection of nodes is available by implementing custom predicates or lambdas.

The particular results map to a cluster node (RedisClusterNode) that was involved in the node selection. You can obtain the set of involved RedisClusterNodes and all results as CompletableFuture from AsyncExecutions.

The node selection API is a technical preview and can change at any time. That approach allows powerful operations but it requires further feedback from the users. So feel free to contribute.

Refreshing the cluster topology view

The Redis Cluster configuration may change at runtime. New nodes can be added, the master for a specific slot can change. Lettuce handles MOVED and ASK redirects transparently but in case too many commands run into redirects, you should refresh the cluster topology view. The topology is bound to a RedisClusterClient instance. All cluster connections that are created by one RedisClusterClient instance share the same cluster topology view. The view can be updated in three ways:

1. Either by calling RedisClusterClient.reloadPartitions
2. Periodic updates in the background based on an interval
3. Adaptive updates in the background based on persistent disconnects and MOVED/ASK redirections

By default, commands follow -ASK and -MOVED redirects up to 5 times until the command execution is considered to be failed.

Client-options

See Cluster-specific Client options

Examples

Connecting to a Redis Cluster

RedisURI redisUri = RedisURI.Builder.redis("localhost").withPassword("authentication").build();

RedisClusterClient clusterClient = RedisClusterClient.create(rediUri);
StatefulRedisClusterConnection<String, String> connection = clusterClient.connect();
RedisAdvancedClusterCommands<String, String> syncCommands = connection.sync();

...

connection.close();
clusterClient.shutdown();

Connecting to a Redis Cluster with multiple seed nodes

RedisURI node1 = RedisURI.create("node1", 6379);
RedisURI node2 = RedisURI.create("node2", 6379);

RedisClusterClient clusterClient = RedisClusterClient.create(Arrays.asList(node1, node2));
StatefulRedisClusterConnection<String, String> connection = clusterClient.connect();
RedisAdvancedClusterCommands<String, String> syncCommands = connection.sync();

...

connection.close();
clusterClient.shutdown();

Enabling periodic cluster topology view updates

RedisClusterClient clusterClient = RedisClusterClient.create(RedisURI.create("localhost", 6379));

ClusterTopologyRefreshOptions topologyRefreshOptions = ClusterTopologyRefreshOptions.builder()
                                .enablePeriodicRefresh(10, TimeUnit.MINUTES)
                                .build();

clusterClient.setOptions(ClusterClientOptions.builder()
                                .topologyRefreshOptions(topologyRefreshOptions)
                                .build());
...

clusterClient.shutdown();

Enabling adaptive cluster topology view updates

RedisURI node1 = RedisURI.create("node1", 6379);
RedisURI node2 = RedisURI.create("node2", 6379);

RedisClusterClient clusterClient = RedisClusterClient.create(Arrays.asList(node1, node2));

ClusterTopologyRefreshOptions topologyRefreshOptions = ClusterTopologyRefreshOptions.builder()
                                .enableAdaptiveRefreshTrigger(RefreshTrigger.MOVED_REDIRECT, RefreshTrigger.PERSISTENT_RECONNECTS)
                                .adaptiveRefreshTriggersTimeout(30, TimeUnit.SECONDS)
                                .build();

clusterClient.setOptions(ClusterClientOptions.builder()
                                .topologyRefreshOptions(topologyRefreshOptions)
                                .build());
...

clusterClient.shutdown();

Obtaining a node connection

RedisURI node1 = RedisURI.create("node1", 6379);
RedisURI node2 = RedisURI.create("node2", 6379);

RedisClusterClient clusterClient = RedisClusterClient.create(Arrays.asList(node1, node2));
StatefulRedisClusterConnection<String, String> connection = clusterClient.connect();

RedisClusterCommands<String, String> node1 = connection.getConnection("host", 7379).sync();

...

// do not close node1
connection.close();
clusterClient.shutdown();

Internals

See Redis Cluster