Skip to content

Commit 6a4a94b

Browse files
ggivoCopilotatakavci
authored
[automatic failover] Refactor MultiDb API: Rename endpoint methods to database and encapsulate retry/circuit breaker configuration (#4310)
* Rename MultiDbClient endpoint related methods - MultiDbClient::addEndpoint -> MultiDbClient::addDatabase - MultiDbClient::forceActiveEndpoint -> MultiDbClient::forceActiveDatabase - MultiDbClient::removeEndpoint -> MultiDbClient::removeDatabase - MultiDbClient::getActiveEndpoint → MultiDbClient::getActiveDatabaseEndpoint - MultiDbClient::getEndpoints → MultiDbClient::getDatabaseEndpoints * Methods Renamed in MultiDbConfig.Builder - endpoint(DatabaseConfig databaseConfig) → database(DatabaseConfig databaseConfig) - endpoint(Endpoint endpoint, float weight, JedisClientConfig clientConfig) → database(Endpoint endpoint, float weight, JedisClientConfig clientConfig) * Refactor command retry configuration - Related retry settings grouped together - RetryConfig class created with builder pattern Example usage MultiDbConfig.RetryConfig.builder() .maxAttempts(5) .waitDuration(1000) .exponentialBackoffMultiplier(2) .includedExceptionList(Arrays.asList(JedisConnectionException.class)) .build()) * Refactor circuit breaker configuration - Related cb settings grouped together Example usage .failureDetector(MultiDbConfig.CircuitBreakerConfig.builder() .slidingWindowSize(3) .minNumOfFailures(2) .failureRateThreshold(50f) .build()) * fix typo in javadocs * fix examples in failover.md * fix failover.md example identation * fix leftover references to activeCuster * Apply suggestion from @Copilot Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Apply suggestion from @Copilot Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * fix leftover cluster references in local/private scope * clean up leftover 'cluster' references in local/private scope in mcf package * fix failing test delayBetweenFailoverAttempts_gatesCounterIncrementsWithinWindow * Apply suggestion from @atakavci Co-authored-by: atakavci <a_takavci@yahoo.com> * Apply suggestion from @atakavci Co-authored-by: atakavci <a_takavci@yahoo.com> * more left overs of 'cluster' in javadocs outside mcf package --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: atakavci <a_takavci@yahoo.com>
1 parent 14356c4 commit 6a4a94b

29 files changed

+1103
-1110
lines changed

docs/failover.md

Lines changed: 77 additions & 61 deletions
Original file line numberDiff line numberDiff line change
@@ -41,8 +41,8 @@ Let's look at one way of configuring Jedis for this scenario.
4141
First, start by defining the initial configuration for each Redis database available and prioritize them using weights.
4242

4343
```java
44-
JedisClientConfig config = DefaultJedisClientConfig.builder().user("cache").password("secret")
45-
.socketTimeoutMillis(5000).connectionTimeoutMillis(5000).build();
44+
JedisClientConfig config = DefaultJedisClientConfig.builder().user("cache").password("secret")
45+
.socketTimeoutMillis(5000).connectionTimeoutMillis(5000).build();
4646

4747
// Custom pool config per database can be provided
4848
ConnectionPoolConfig poolConfig = new ConnectionPoolConfig();
@@ -58,8 +58,8 @@ HostAndPort east = new HostAndPort("redis-east.example.com", 14000);
5858
HostAndPort west = new HostAndPort("redis-west.example.com", 14000);
5959

6060
MultiDbConfig.Builder multiConfig = MultiDbConfig.builder()
61-
.endpoint(DatabaseConfig.builder(east, config).connectionPoolConfig(poolConfig).weight(1.0f).build())
62-
.endpoint(DatabaseConfig.builder(west, config).connectionPoolConfig(poolConfig).weight(0.5f).build());
61+
.database(DatabaseConfig.builder(east, config).connectionPoolConfig(poolConfig).weight(1.0f).build())
62+
.database(DatabaseConfig.builder(west, config).connectionPoolConfig(poolConfig).weight(0.5f).build());
6363
```
6464

6565
The configuration above represents your two Redis deployments: `redis-east` and `redis-west`.
@@ -68,28 +68,33 @@ Continue using the `MultiDbConfig.Builder` builder to set your preferred retry a
6868
Then build a `MultiDbClient`.
6969

7070
```java
71-
multiDbBuilder.circuitBreakerSlidingWindowSize(2) // Sliding window size in number of calls
72-
.circuitBreakerFailureRateThreshold(10.0f) // percentage of failures to trigger circuit breaker
73-
.circuitBreakerMinNumOfFailures(1000) // Minimum number of failures before circuit breaker is tripped
74-
75-
.failbackSupported(true) // Enable failback
76-
.failbackCheckInterval(1000) // Check every second the unhealthy database to see if it has recovered
77-
.gracePeriod(10000) // Keep database disabled for 10 seconds after it becomes unhealthy
78-
79-
// Optional: configure retry settings
80-
.retryMaxAttempts(3) // Maximum number of retry attempts (including the initial call)
81-
.retryWaitDuration(500) // Number of milliseconds to wait between retry attempts
82-
.retryWaitDurationExponentialBackoffMultiplier(2) // Exponential backoff factor multiplied against wait duration between retries
83-
84-
// Optional: configure fast failover
85-
.fastFailover(true) // Force closing connections to unhealthy database on failover
86-
.retryOnFailover(false); // Do not retry failed commands during failover
87-
88-
MultiDbClient multiDbClient = multiDbBuilder.build();
71+
// Configure circuit breaker for failure detection
72+
multiConfig
73+
.failureDetector(MultiDbConfig.CircuitBreakerConfig.builder()
74+
.slidingWindowSize(1000) // Sliding window size in number of calls
75+
.failureRateThreshold(50.0f) // percentage of failures to trigger circuit breaker
76+
.minNumOfFailures(500) // Minimum number of failures before circuit breaker is tripped
77+
.build())
78+
.failbackSupported(true) // Enable failback
79+
.failbackCheckInterval(1000) // Check every second the unhealthy database to see if it has recovered
80+
.gracePeriod(10000) // Keep database disabled for 10 seconds after it becomes unhealthy
81+
// Optional: configure retry settings
82+
.commandRetry(MultiDbConfig.RetryConfig.builder()
83+
.maxAttempts(3) // Maximum number of retry attempts (including the initial call)
84+
.waitDuration(500) // Number of milliseconds to wait between retry attempts
85+
.exponentialBackoffMultiplier(2) // Exponential backoff factor multiplied against wait duration between retries
86+
.build())
87+
// Optional: configure fast failover
88+
.fastFailover(true) // Force closing connections to unhealthy database on failover
89+
.retryOnFailover(false); // Do not retry failed commands during failover
90+
91+
MultiDbClient multiDbClient = MultiDbClient.builder()
92+
.multiDbConfig(multiConfig.build())
93+
.build();
8994
```
9095

91-
In the configuration here, we've set a sliding window size of 10 and a failure rate threshold of 50%.
92-
This means that a failover will be triggered if 5 out of any 10 calls to Redis fail.
96+
In the configuration here, we've set a sliding window size of 1000 and a failure rate threshold of 50%.
97+
This means that a failover will be triggered only if both 500 out of any 1000 calls to Redis fail (i.e., the failure rate threshold is reached) and the minimum number of failures is also met.
9398

9499
You can now use this `MultiDbClient` instance, and the connection management and failover will be handled transparently.
95100

@@ -98,22 +103,22 @@ You can now use this `MultiDbClient` instance, and the connection management and
98103
Under the hood, Jedis' failover support relies on [resilience4j](https://resilience4j.readme.io/docs/getting-started),
99104
a fault-tolerance library that implements [retry](https://resilience4j.readme.io/docs/retry) and [circuit breakers](https://resilience4j.readme.io/docs/circuitbreaker).
100105

101-
Once you configure Jedis for failover using the `MultiClusterPooledConnectionProvider`, each call to Redis is decorated with a resilience4j retry and circuit breaker.
106+
Once you configure Jedis for failover using the `MultiDbConnectionProvider`, each call to Redis is decorated with a resilience4j retry and circuit breaker.
102107

103108
By default, any call that throws a `JedisConnectionException` will be retried up to 3 times.
104109
If the call fail then the circuit breaker will record a failure.
105110

106111
The circuit breaker maintains a record of failures in a sliding window data structure.
107-
If the failure rate reaches a configured threshold (e.g., when 50% of the last 10 calls have failed),
112+
If the failure rate reaches a configured threshold (e.g., when 50% of the last 1000 calls have failed),
108113
then the circuit breaker's state transitions from `CLOSED` to `OPEN`.
109114
When this occurs, Jedis will attempt to connect to the next Redis database with the highest weight in its client configuration list.
110115

111116
The supported retry and circuit breaker settings, and their default values, are described below.
112-
You can configure any of these settings using the `MultiClusterClientConfig.Builder` builder.
117+
You can configure any of these settings using the `MultiDbConfig.Builder` builder.
113118
Refer the basic usage above for an example of this.
114119

115120
### Retry configuration
116-
121+
Configuration for command retry behavior is encapsulated in `MultiDbConfig.RetryConfig`.
117122
Jedis uses the following retry settings:
118123

119124
| Setting | Default value | Description |
@@ -124,10 +129,11 @@ Jedis uses the following retry settings:
124129
| Retry included exception list | [JedisConnectionException] | A list of Throwable classes that count as failures and should be retried. |
125130
| Retry ignored exception list | null | A list of Throwable classes to explicitly ignore for the purposes of retry. |
126131

127-
To disable retry, set `maxRetryAttempts` to 1.
132+
To disable retry, set `maxAttempts` to 1.
128133

129134
### Circuit breaker configuration
130-
135+
For failover, Jedis uses a circuit breaker to detect when a Redis database has failed.
136+
Failover configuration is encapsulated in `MultiDbConfig.CircuitBreakerConfig` and can be provided using the `MultiDbConfig.Builder.failureDetector()`.
131137
Jedis uses the following circuit breaker settings:
132138

133139
| Setting | Default value | Description |
@@ -221,23 +227,23 @@ Use the `healthCheckStrategySupplier()` method to provide a custom health check
221227

222228
```java
223229
// Custom strategy supplier
224-
MultiClusterClientConfig.StrategySupplier customStrategy =
225-
(hostAndPort, jedisClientConfig) -> {
226-
// Return your custom HealthCheckStrategy implementation
227-
return new MyCustomHealthCheckStrategy(hostAndPort, jedisClientConfig);
228-
};
230+
MultiDbConfig.StrategySupplier customStrategy =
231+
(hostAndPort, jedisClientConfig) -> {
232+
// Return your custom HealthCheckStrategy implementation
233+
return new MyCustomHealthCheckStrategy(hostAndPort, jedisClientConfig);
234+
};
229235

230-
MultiClusterClientConfig.ClusterConfig dbConfig =
231-
MultiClusterClientConfig.ClusterConfig.builder(hostAndPort, clientConfig)
232-
.healthCheckStrategySupplier(customStrategy)
233-
.weight(1.0f)
234-
.build();
236+
MultiDbConfig.DatabaseConfig dbConfig =
237+
MultiDbConfig.DatabaseConfig.builder(hostAndPort, clientConfig)
238+
.healthCheckStrategySupplier(customStrategy)
239+
.weight(1.0f)
240+
.build();
235241
```
236242

237243
You can implement custom health check strategies by implementing the `HealthCheckStrategy` interface:
238244

239245
```java
240-
MultiClusterClientConfig.StrategySupplier pingStrategy = (hostAndPort, jedisClientConfig) -> {
246+
MultiDbConfig.StrategySupplier pingStrategy = (hostAndPort, jedisClientConfig) -> {
241247
return new HealthCheckStrategy() {
242248
@Override
243249
public int getInterval() {
@@ -249,11 +255,21 @@ MultiClusterClientConfig.StrategySupplier pingStrategy = (hostAndPort, jedisClie
249255
return 500; // 500ms timeout
250256
}
251257

258+
252259
@Override
253-
public int minConsecutiveSuccessCount() {
254-
return 1; // Single success required
260+
public int getNumProbes() {
261+
return 1;
255262
}
256263

264+
@Override
265+
public ProbingPolicy getPolicy() {
266+
return ProbingPolicy.BuiltIn.ANY_SUCCESS;
267+
}
268+
269+
@Override
270+
public int getDelayInBetweenProbes() {
271+
return 100;
272+
}
257273
@Override
258274
public HealthStatus doHealthCheck(Endpoint endpoint) {
259275
try (UnifiedJedis jedis = new UnifiedJedis(hostAndPort, jedisClientConfig)) {
@@ -271,20 +287,20 @@ MultiClusterClientConfig.StrategySupplier pingStrategy = (hostAndPort, jedisClie
271287
};
272288
};
273289

274-
MultiClusterClientConfig.ClusterConfig dbConfig =
275-
MultiClusterClientConfig.ClusterConfig.builder(hostAndPort, clientConfig)
276-
.healthCheckStrategySupplier(pingStrategy)
277-
.build();
290+
MultiDbConfig.DatabaseConfig dbConfig =
291+
MultiDbConfig.DatabaseConfig.builder(hostAndPort, clientConfig)
292+
.healthCheckStrategySupplier(pingStrategy)
293+
.build();
278294
```
279295

280296
#### Disabling Health Checks
281297

282298
Use the `healthCheckEnabled(false)` method to completely disable health checks:
283299

284300
```java
285-
DatabaseConfig dbConfig = DatabaseConfig.builder(east, config)
286-
.healthCheckEnabled(false) // Disable health checks entirely
287-
.build();
301+
MultiDbConfig.DatabaseConfig dbConfig = MultiDbConfig.DatabaseConfig.builder(east, config)
302+
.healthCheckEnabled(false) // Disable health checks entirely
303+
.build();
288304
```
289305

290306
### Fallback configuration
@@ -306,7 +322,7 @@ To use this feature, you'll need to design a class that implements `java.util.fu
306322
This class must implement the `accept` method, as you can see below.
307323

308324
```java
309-
public class FailoverReporter implements Consumer<DatabaseSwitchEvent> {
325+
public class FailoverReporter implements Consumer<DatabaseSwitchEvent> {
310326

311327
@Override
312328
public void accept(DatabaseSwitchEvent e) {
@@ -317,18 +333,18 @@ This class must implement the `accept` method, as you can see below.
317333

318334
DatabaseSwitchEvent consumer can be registered as follows:
319335

320-
```
321-
FailoverReporter reporter = new FailoverReporter();
322-
MultiDbClient client = MultiDbClient.builder()
323-
.databaseSwitchListener(reporter)
324-
.build();
336+
```java
337+
FailoverReporter reporter = new FailoverReporter();
338+
MultiDbClient client = MultiDbClient.builder()
339+
.databaseSwitchListener(reporter)
340+
.build();
325341
```
326342
The provider will call your `accept` whenever a failover occurs.
327343
or directly using lambda expression:
328-
```
329-
MultiDbClient client = MultiDbClient.builder()
330-
.databaseSwitchListener(event -> System.out.println("Switched to: " + event.getEndpoint()))
331-
.build();
344+
```java
345+
MultiDbClient client = MultiDbClient.builder()
346+
.databaseSwitchListener(event -> System.out.println("Switched to: " + event.getEndpoint()))
347+
.build();
332348
```
333349

334350

@@ -414,7 +430,7 @@ HealthCheckStrategy.Config config = HealthCheckStrategy.Config.builder()
414430
.build();
415431

416432
// Adjust failback timing
417-
MultiDbConfig multiConfig = new MultiDbConfig.Builder()
433+
MultiDbConfig multiConfig = MultiDbConfig.builder()
418434
.gracePeriod(5000) // Shorter grace period
419435
.build();
420436
```

0 commit comments

Comments
 (0)