From fea4498d945db99f7a0de187984fb2bdf765756d Mon Sep 17 00:00:00 2001 From: Andy Stark Date: Thu, 1 May 2025 13:16:04 +0100 Subject: [PATCH 1/5] DOC-5065 started Python production usage page --- content/develop/clients/redis-py/produsage.md | 214 ++++++++++++++++++ 1 file changed, 214 insertions(+) create mode 100644 content/develop/clients/redis-py/produsage.md diff --git a/content/develop/clients/redis-py/produsage.md b/content/develop/clients/redis-py/produsage.md new file mode 100644 index 000000000..00ddae12f --- /dev/null +++ b/content/develop/clients/redis-py/produsage.md @@ -0,0 +1,214 @@ +--- +categories: +- docs +- develop +- stack +- oss +- rs +- rc +- oss +- kubernetes +- clients +description: Get your `redis-py` app ready for production +linkTitle: Production usage +title: Production usage +weight: 70 +--- + +This guide offers recommendations to get the best reliability and +performance in your production environment. + +## Checklist + +Each item in the checklist below links to the section +for a recommendation. Use the checklist icons to record your +progress in implementing the recommendations. + +{{< checklist "prodlist" >}} + {{< checklist-item "#connection-pooling" >}}Connection pooling{{< /checklist-item >}} + {{< checklist-item "#client-side-caching" >}}Client-side caching{{< /checklist-item >}} + {{< checklist-item "#retries" >}}Retries{{< /checklist-item >}} + {{< checklist-item "#timeouts" >}}Timeouts{{< /checklist-item >}} + {{< checklist-item "#health-checks" >}}Health checks{{< /checklist-item >}} + {{< checklist-item "#exception-handling" >}}Exception handling{{< /checklist-item >}} +{{< /checklist >}} + +## Recommendations + +The sections below offer recommendations for your production environment. Some +of them may not apply to your particular use case. + +### Connection pooling + +Example code often opens a connection at the start, demonstrates a feature, +and then closes the connection at the end. However, production code +typically uses connections many times intermittently. Repeatedly opening +and closing connections has a performance overhead. + +Use [connection pooling]({{< relref "/develop/clients/pools-and-muxing" >}}) +to avoid the overhead of opening and closing connections without having to +write your own code to cache and reuse open connections. See +[Connect with a connection pool]({{< relref "/develop/clients/redis-py/connect#connect-with-a-connection-pool" >}}) +to learn how to use this technique with `redis-py`. + +### Client-side caching + +[Client-side caching]({{< relref "/develop/clients/client-side-caching" >}}) +involves storing the results from read-only commands in a local cache. If the +same command is executed again later, the results can be obtained from the cache, +without contacting the server. This improves command execution time on the client, +while also reducing network traffic and server load. See +[Connect using client-side caching]({{< relref "/develop/clients/redis-py/connect#connect-using-client-side-caching" >}}) +for more information and example code. + +### Retries + +Redis connections and commands can often fail due to transient problems, +such as temporary network outages or timeouts. When this happens, +the operation will generally succeed after a few attempts, despite +failing the first time. + +You can configure `redis-py` to retry connections automatically after +specified errors occur. Use an instance of the `Retry` class to +specify the number of times to retry after a failure and a backoff +strategy to determine the time between attempts. For example, the +following code creates a `Retry` with +[exponential backoff](https://en.wikipedia.org/wiki/Exponential_backoff) +that will make three repeated attempts after a failure: + +```py +from redis.backoff import ExponentialBackoff +from redis.retry import Retry + +# Run 3 retries with exponential backoff strategy. +retry = Retry(ExponentialBackoff(), 3) +``` + +Pass the `Retry` instance in the `retry` parameter when you connect +to Redis. When you are connecting to a standalone Redis server, +you can also set the `retry_on_timeout` parameter to `True` +to retry only after timeout errors, or pass a list of exception +types in the `retry_on_error` parameter. + +```py +# Retry only on timeout +r = Redis( + retry=retry, + retry_on_timeout=True + . + . +) + +# Retry on any of a specified list of command exceptions. +from redis.exceptions import ( + BusyLoadingError, + ConnectionError, + TimeoutError +) + . + . + +r = Redis( + retry=retry, + retry_on_error=[BusyLoadingError, ConnectionError, TimeoutError], + . + . +) +``` + +If you specify either `retry_on_timeout` or `retry_on_error` without +a `retry` parameter, the default is to retry once immediately with no +backoff. + +For a connection to a Redis cluster, you can specify a `retry` instance. +However, the list of exceptions is not configurable and is always set +to `TimeoutError`, `ConnectionError`, and `ClusterDownError`. You can use +the `cluster_error_retry_attempts` parameter to specify the number of +attempts to make after encountering one of these errors: + +```py + +``` + +### Timeouts + +If a network or server error occurs while your code is opening a +connection or issuing a command, it can end up hanging indefinitely. +You can prevent this from happening by setting timeouts for socket +reads and writes and for opening connections. + +To set a timeout for a connection, use the `JedisPooled` or `JedisPool` constructor with the `timeout` parameter, or use `JedisClientConfig` with the `socketTimeout` and `connectionTimeout` parameters. +(The socket timeout is the maximum time allowed for reading or writing data while executing a +command. The connection timeout is the maximum time allowed for establishing a new connection.) + +```java +HostAndPort hostAndPort = new HostAndPort("localhost", 6379); + +JedisPooled jedisWithTimeout = new JedisPooled(hostAndPort, + DefaultJedisClientConfig.builder() + .socketTimeoutMillis(5000) // set timeout to 5 seconds + .connectionTimeoutMillis(5000) // set connection timeout to 5 seconds + .build(), + poolConfig +); +``` + +### Health checks + +If your code doesn't access the Redis server continuously then it +might be useful to make a "health check" periodically (perhaps once +every few seconds) to verify that the connection is working. +Set the `health_check_interval` parameter during +a connection (with either `Redis` or `ConnectionPool`) to specify +an integer number of seconds. If the connection remains idle for +longer than this interval, it will automatically issue a +[`PING`]({{< relref "/commands/ping" >}}) command and check the +response before continuing with any client commands. + +```py +# Issue a health check if the connection is idle for longer +# than three seconds. +r = Redis( + health_check_interval = 3, + . + . +) +``` + +Health checks help to detect problems as soon as possible without +waiting for a user to report them. If a `ConnectionError` or `TimeoutError` +occurs for the health check itself, a second attempt will be made before +reporting failure. + +### Exception handling + +Redis handles many errors using return values from commands, but there +are also situations where exceptions can be thrown. In production code, +you should handle exceptions wherever they can occur. In particular, +exceptions can occur when you +[connect to the server]({{< relref "/develop/clients/redis-py/connect" >}}), +create a [query index]({{< relref "/develop/interact/search-and-query/indexing" >}}), +or execute a +[watched transaction]({{< relref "/develop/clients/redis-py/transpipe#watch-keys-for-changes" >}}). + +#### General exceptions + +In general, Jedis can throw the following exceptions while executing commands: + +- `JedisConnectionException` - when the connection to Redis is lost or closed unexpectedly. Configure failover to handle this exception automatically with Resilience4J and the built-in Jedis failover mechanism. +- `JedisAccessControlException` - when the user does not have the permission to execute the command or the user ID and/or password are incorrect. +- `JedisDataException` - when there is a problem with the data being sent to or received from the Redis server. Usually, the error message will contain more information about the failed command. +- `JedisException` - this exception is a catch-all exception that can be thrown for any other unexpected errors. + +Conditions when `JedisException` can be thrown: +- Bad return from a health check with the [`PING`]({{< relref "/commands/ping" >}}) command +- Failure during SHUTDOWN +- Pub/Sub failure when issuing commands (disconnect) +- Any unknown server messages +- Sentinel: can connect to sentinel but master is not monitored or all Sentinels are down. +- MULTI or DISCARD command failed +- Shard commands key hash check failed or no Reachable Shards +- Retry deadline exceeded/number of attempts (Retry Command Executor) +- POOL - pool exhausted, error adding idle objects, returning broken resources to the pool + +All the Jedis exceptions are runtime exceptions and in most cases irrecoverable, so in general bubble up to the API capturing the error message. From a8b8eeea5f86bf851cbc65f9d56516c552c8e81d Mon Sep 17 00:00:00 2001 From: Andy Stark Date: Fri, 2 May 2025 11:35:21 +0100 Subject: [PATCH 2/5] DOC-5065 added redis-py production usage page --- content/develop/clients/redis-py/produsage.md | 99 ++++++------------- 1 file changed, 32 insertions(+), 67 deletions(-) diff --git a/content/develop/clients/redis-py/produsage.md b/content/develop/clients/redis-py/produsage.md index 00ddae12f..e0a524703 100644 --- a/content/develop/clients/redis-py/produsage.md +++ b/content/develop/clients/redis-py/produsage.md @@ -28,7 +28,6 @@ progress in implementing the recommendations. {{< checklist-item "#connection-pooling" >}}Connection pooling{{< /checklist-item >}} {{< checklist-item "#client-side-caching" >}}Client-side caching{{< /checklist-item >}} {{< checklist-item "#retries" >}}Retries{{< /checklist-item >}} - {{< checklist-item "#timeouts" >}}Timeouts{{< /checklist-item >}} {{< checklist-item "#health-checks" >}}Health checks{{< /checklist-item >}} {{< checklist-item "#exception-handling" >}}Exception handling{{< /checklist-item >}} {{< /checklist >}} @@ -68,11 +67,11 @@ such as temporary network outages or timeouts. When this happens, the operation will generally succeed after a few attempts, despite failing the first time. -You can configure `redis-py` to retry connections automatically after -specified errors occur. Use an instance of the `Retry` class to -specify the number of times to retry after a failure and a backoff -strategy to determine the time between attempts. For example, the -following code creates a `Retry` with +You can configure `redis-py` to retry commands automatically when +errors occur. Use an instance of the `Retry` class to +specify the number of times to retry after a failure. You can also +specify a backoff strategy to control the time gap between attempts. +For example, the following code creates a `Retry` with [exponential backoff](https://en.wikipedia.org/wiki/Exponential_backoff) that will make three repeated attempts after a failure: @@ -87,7 +86,7 @@ retry = Retry(ExponentialBackoff(), 3) Pass the `Retry` instance in the `retry` parameter when you connect to Redis. When you are connecting to a standalone Redis server, you can also set the `retry_on_timeout` parameter to `True` -to retry only after timeout errors, or pass a list of exception +(to retry only after timeout errors), or pass a list of exception types in the `retry_on_error` parameter. ```py @@ -120,38 +119,9 @@ If you specify either `retry_on_timeout` or `retry_on_error` without a `retry` parameter, the default is to retry once immediately with no backoff. -For a connection to a Redis cluster, you can specify a `retry` instance. -However, the list of exceptions is not configurable and is always set -to `TimeoutError`, `ConnectionError`, and `ClusterDownError`. You can use -the `cluster_error_retry_attempts` parameter to specify the number of -attempts to make after encountering one of these errors: - -```py - -``` - -### Timeouts - -If a network or server error occurs while your code is opening a -connection or issuing a command, it can end up hanging indefinitely. -You can prevent this from happening by setting timeouts for socket -reads and writes and for opening connections. - -To set a timeout for a connection, use the `JedisPooled` or `JedisPool` constructor with the `timeout` parameter, or use `JedisClientConfig` with the `socketTimeout` and `connectionTimeout` parameters. -(The socket timeout is the maximum time allowed for reading or writing data while executing a -command. The connection timeout is the maximum time allowed for establishing a new connection.) - -```java -HostAndPort hostAndPort = new HostAndPort("localhost", 6379); - -JedisPooled jedisWithTimeout = new JedisPooled(hostAndPort, - DefaultJedisClientConfig.builder() - .socketTimeoutMillis(5000) // set timeout to 5 seconds - .connectionTimeoutMillis(5000) // set connection timeout to 5 seconds - .build(), - poolConfig -); -``` +For a connection to a Redis cluster, you can specify a `retry` instance, +but the list of exceptions is not configurable and is always set +to `TimeoutError`, `ConnectionError`, and `ClusterDownError`. ### Health checks @@ -184,31 +154,26 @@ reporting failure. Redis handles many errors using return values from commands, but there are also situations where exceptions can be thrown. In production code, -you should handle exceptions wherever they can occur. In particular, -exceptions can occur when you -[connect to the server]({{< relref "/develop/clients/redis-py/connect" >}}), -create a [query index]({{< relref "/develop/interact/search-and-query/indexing" >}}), -or execute a -[watched transaction]({{< relref "/develop/clients/redis-py/transpipe#watch-keys-for-changes" >}}). - -#### General exceptions - -In general, Jedis can throw the following exceptions while executing commands: - -- `JedisConnectionException` - when the connection to Redis is lost or closed unexpectedly. Configure failover to handle this exception automatically with Resilience4J and the built-in Jedis failover mechanism. -- `JedisAccessControlException` - when the user does not have the permission to execute the command or the user ID and/or password are incorrect. -- `JedisDataException` - when there is a problem with the data being sent to or received from the Redis server. Usually, the error message will contain more information about the failed command. -- `JedisException` - this exception is a catch-all exception that can be thrown for any other unexpected errors. - -Conditions when `JedisException` can be thrown: -- Bad return from a health check with the [`PING`]({{< relref "/commands/ping" >}}) command -- Failure during SHUTDOWN -- Pub/Sub failure when issuing commands (disconnect) -- Any unknown server messages -- Sentinel: can connect to sentinel but master is not monitored or all Sentinels are down. -- MULTI or DISCARD command failed -- Shard commands key hash check failed or no Reachable Shards -- Retry deadline exceeded/number of attempts (Retry Command Executor) -- POOL - pool exhausted, error adding idle objects, returning broken resources to the pool - -All the Jedis exceptions are runtime exceptions and in most cases irrecoverable, so in general bubble up to the API capturing the error message. +you should handle exceptions wherever they can occur. + +Import the exceptions you need to check from the `redis.exceptions` +module. The list below describes some of the most common exceptions. + +- `ConnectionError`: Thrown when a connection attempt fails + (for example, when connection parameters are invalid or the server + is unavailable) and sometimes when a [health check](#health-checks) + fails. There is also a subclass, `AuthenticationError`, specifically + for authentication failures. +- `ResponseError`: Thrown when you attempt an operation that has no valid + response. Examples include executing a command on the wrong type of key + (as when you try an + ['LPUSH']({{< relref "/develop/data-types/lists#automatic-creation-and-removal-of-keys" >}}) + command on a string key), creating an + [index]({{< relref "/develop/interact/search-and-query/indexing" >}}) + with a name that already exists, and using an invalid ID for a + [stream entry]({{< relref "/develop/data-types/streams/#entry-ids" >}}). +- `TimeoutError`: Thrown when a timeout persistently happens for a command, + despite any [retries](#retries). +- `WatchError`: Thrown when a + [watched key]({{< relref "/develop/clients/redis-py/transpipe#watch-keys-for-changes" >}}) is + modified during a transaction. From 7349e950f2452a95ef1fa101806d8c5bf300fdef Mon Sep 17 00:00:00 2001 From: Andy Stark Date: Fri, 2 May 2025 16:38:43 +0100 Subject: [PATCH 3/5] DOC-5065 added unique name for checklist --- content/develop/clients/redis-py/produsage.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/develop/clients/redis-py/produsage.md b/content/develop/clients/redis-py/produsage.md index e0a524703..8ae5db47e 100644 --- a/content/develop/clients/redis-py/produsage.md +++ b/content/develop/clients/redis-py/produsage.md @@ -24,7 +24,7 @@ Each item in the checklist below links to the section for a recommendation. Use the checklist icons to record your progress in implementing the recommendations. -{{< checklist "prodlist" >}} +{{< checklist "pyprodlist" >}} {{< checklist-item "#connection-pooling" >}}Connection pooling{{< /checklist-item >}} {{< checklist-item "#client-side-caching" >}}Client-side caching{{< /checklist-item >}} {{< checklist-item "#retries" >}}Retries{{< /checklist-item >}} From 24c9f3981365b94cb527653b8bebf42baec6e180 Mon Sep 17 00:00:00 2001 From: Andy Stark Date: Wed, 7 May 2025 11:39:57 +0100 Subject: [PATCH 4/5] DOC-5065 implemented feedback --- content/develop/clients/redis-py/produsage.md | 79 +++++++++---------- 1 file changed, 36 insertions(+), 43 deletions(-) diff --git a/content/develop/clients/redis-py/produsage.md b/content/develop/clients/redis-py/produsage.md index 8ae5db47e..614460977 100644 --- a/content/develop/clients/redis-py/produsage.md +++ b/content/develop/clients/redis-py/produsage.md @@ -25,7 +25,6 @@ for a recommendation. Use the checklist icons to record your progress in implementing the recommendations. {{< checklist "pyprodlist" >}} - {{< checklist-item "#connection-pooling" >}}Connection pooling{{< /checklist-item >}} {{< checklist-item "#client-side-caching" >}}Client-side caching{{< /checklist-item >}} {{< checklist-item "#retries" >}}Retries{{< /checklist-item >}} {{< checklist-item "#health-checks" >}}Health checks{{< /checklist-item >}} @@ -37,19 +36,6 @@ progress in implementing the recommendations. The sections below offer recommendations for your production environment. Some of them may not apply to your particular use case. -### Connection pooling - -Example code often opens a connection at the start, demonstrates a feature, -and then closes the connection at the end. However, production code -typically uses connections many times intermittently. Repeatedly opening -and closing connections has a performance overhead. - -Use [connection pooling]({{< relref "/develop/clients/pools-and-muxing" >}}) -to avoid the overhead of opening and closing connections without having to -write your own code to cache and reuse open connections. See -[Connect with a connection pool]({{< relref "/develop/clients/redis-py/connect#connect-with-a-connection-pool" >}}) -to learn how to use this technique with `redis-py`. - ### Client-side caching [Client-side caching]({{< relref "/develop/clients/client-side-caching" >}}) @@ -67,58 +53,65 @@ such as temporary network outages or timeouts. When this happens, the operation will generally succeed after a few attempts, despite failing the first time. -You can configure `redis-py` to retry commands automatically when -errors occur. Use an instance of the `Retry` class to -specify the number of times to retry after a failure. You can also -specify a backoff strategy to control the time gap between attempts. -For example, the following code creates a `Retry` with +`redis-py` can retry commands automatically when +errors occur. From version 6.0.0 onwards, the default behavior is to +attempt a failed command three times. +The timing between successive attempts is calculated using [exponential backoff](https://en.wikipedia.org/wiki/Exponential_backoff) -that will make three repeated attempts after a failure: +with some random "jitter" added to avoid two or more connections retrying +commands in sync with each other. + +You can override the default behavior using an instance of the `Retry` class to +specify the number of times to retry after a failure along with your +own choice of backoff strategy. +Pass the `Retry` object in the `retry` parameter when you connect. +For example, the connection in the code below uses an exponential backoff strategy +(without jitter) that will make eight repeated attempts after a failure: ```py from redis.backoff import ExponentialBackoff from redis.retry import Retry -# Run 3 retries with exponential backoff strategy. -retry = Retry(ExponentialBackoff(), 3) -``` - -Pass the `Retry` instance in the `retry` parameter when you connect -to Redis. When you are connecting to a standalone Redis server, -you can also set the `retry_on_timeout` parameter to `True` -(to retry only after timeout errors), or pass a list of exception -types in the `retry_on_error` parameter. +# Run 8 retries with exponential backoff strategy. +retry = Retry(ExponentialBackoff(), 8) -```py -# Retry only on timeout r = Redis( retry=retry, - retry_on_timeout=True . . ) +``` -# Retry on any of a specified list of command exceptions. +A retry is triggered when a command throws any exception +listed in the `supported_errors` attribute of the `Retry` class. +By default, the list only includes `ConnectionError` and `TimeoutError`, +but you can set your own choice of exceptions when you create the +instance: + +```py +# Only retry after a `TimeoutError`. +retry = Retry(ExponentialBackoff(), 3, supported_errors=(TimeoutError,)) +``` + +You can also add extra exceptions to the default list using the `retry_on_error` +parameter when you connect: + +```py +# Add `BusyLoadingError` to the default list of exceptions. from redis.exceptions import ( BusyLoadingError, - ConnectionError, - TimeoutError ) . . r = Redis( retry=retry, - retry_on_error=[BusyLoadingError, ConnectionError, TimeoutError], + retry_on_error=[BusyLoadingError], . . ) ``` -If you specify either `retry_on_timeout` or `retry_on_error` without -a `retry` parameter, the default is to retry once immediately with no -backoff. - For a connection to a Redis cluster, you can specify a `retry` instance, but the list of exceptions is not configurable and is always set to `TimeoutError`, `ConnectionError`, and `ClusterDownError`. @@ -146,9 +139,9 @@ r = Redis( ``` Health checks help to detect problems as soon as possible without -waiting for a user to report them. If a `ConnectionError` or `TimeoutError` -occurs for the health check itself, a second attempt will be made before -reporting failure. +waiting for a user to report them. Note that health checks, like +other commands, will be [retried](#retries) using the strategy +that you specified for the connection. ### Exception handling From 3220fc070b328629caafba6b82ae0aaaf1014dc0 Mon Sep 17 00:00:00 2001 From: Andy Stark Date: Fri, 9 May 2025 10:26:07 +0100 Subject: [PATCH 5/5] DOC-5065 fixes to checklist shortcodes --- layouts/shortcodes/checklist-item.html | 6 ++++-- layouts/shortcodes/checklist.html | 4 ++-- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/layouts/shortcodes/checklist-item.html b/layouts/shortcodes/checklist-item.html index 653688172..cc4607e91 100644 --- a/layouts/shortcodes/checklist-item.html +++ b/layouts/shortcodes/checklist-item.html @@ -1,11 +1,13 @@
  • - +{{- end }} -{{- if index .Params 0 -}} +{{- if .Get 0 -}} {{ .Inner }} {{- else -}} {{ .Inner }} diff --git a/layouts/shortcodes/checklist.html b/layouts/shortcodes/checklist.html index 685c3422d..2c67c05f9 100644 --- a/layouts/shortcodes/checklist.html +++ b/layouts/shortcodes/checklist.html @@ -1,4 +1,4 @@ -{{ $formId := index .Params 0 }} +{{ $formId := .Get 0 }}
      {{ .Inner }} @@ -17,7 +17,7 @@ document.addEventListener('DOMContentLoaded', () => { let itemString = localStorage.getItem("{{ $formId }}"); - if (itemString !== "") { + if (itemString) { setCLItemsFromString("{{ $formId }}", itemString); } else { clChange("{{ $formId }}");