Infinity retry on the exception of max number of clients reached

[zhuang42]

Version: What redis-py and what redis version is the issue happening on?
redis-py: 3.5.3
redis: 6.0.10

Platform: What platform / version? (For example Python 3.5.1 on Windows 7 / Ubuntu 15.10 / Azure)
Python 3.8.7
OS: Ubuntu 20.04 x86/64

Description: Description of your issue, stack traces from errors and code that reproduces the issue

The redis-py keeps retrying in the redis.exceptions.ConnectionError: max number of clients reached, the worse thing is it keeps retrying in its exception handler which causes an infinity loop. The following logs are repeated hundreds of times.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/.venv/lib/python3.8/site-packages/redis/connection.py", line 685, in check_health
    if nativestr(self.read_response()) != 'PONG':
  File "/.venv/lib/python3.8/site-packages/redis/connection.py", line 739, in read_response
    response = self._parser.read_response()
  File "/.venv/lib/python3.8/site-packages/redis/connection.py", line 340, in read_response
    raise error
redis.exceptions.ConnectionError: max number of clients reached

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/.venv/lib/python3.8/site-packages/redis/connection.py", line 685, in check_health
    if nativestr(self.read_response()) != 'PONG':
  File "/.venv/lib/python3.8/site-packages/redis/connection.py", line 739, in read_response
    response = self._parser.read_response()
  File "/.venv/lib/python3.8/site-packages/redis/connection.py", line 340, in read_response
    raise error
redis.exceptions.ConnectionError: max number of clients reached

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/.venv/lib/python3.8/site-packages/redis/connection.py", line 685, in check_health
    if nativestr(self.read_response()) != 'PONG':
  File "/.venv/lib/python3.8/site-packages/redis/connection.py", line 739, in read_response
    response = self._parser.read_response()
  File "/.venv/lib/python3.8/site-packages/redis/connection.py", line 340, in read_response
    raise error
redis.exceptions.ConnectionError: max number of clients reached

Then this infinity loop terminates with RecursionError

  File "/.venv/lib/python3.8/site-packages/redis/client.py", line 1606, in get
    return self.execute_command('GET', name)
  File "/.venv/lib/python3.8/site-packages/redis/client.py", line 898, in execute_command
    conn = self.connection or pool.get_connection(command_name, **options)
  File "/.venv/lib/python3.8/site-packages/redis/connection.py", line 1192, in get_connection
    connection.connect()
  File "/.venv/lib/python3.8/site-packages/redis/connection.py", line 567, in connect
    self.on_connect()
  File "/.venv/lib/python3.8/site-packages/redis/connection.py", line 663, in on_connect
    self.send_command('SELECT', self.db)
  File "/.venv/lib/python3.8/site-packages/redis/connection.py", line 725, in send_command
    self.send_packed_command(self.pack_command(*args),
  File "/.venv/lib/python3.8/site-packages/redis/connection.py", line 701, in send_packed_command
    self.check_health()
  File "/.venv/lib/python3.8/site-packages/redis/connection.py", line 690, in check_health
    self.send_command('PING', check_health=False)
  File "/.venv/lib/python3.8/site-packages/redis/connection.py", line 725, in send_command
    self.send_packed_command(self.pack_command(*args),
  # --------------------------------- repeated hundreds of times ---------------------------------
  File "/.venv/lib/python3.8/site-packages/redis/connection.py", line 698, in send_packed_command
    self.connect()
  File "/.venv/lib/python3.8/site-packages/redis/connection.py", line 567, in connect
    self.on_connect()
  File "/.venv/lib/python3.8/site-packages/redis/connection.py", line 663, in on_connect
    self.send_command('SELECT', self.db)
  File "/.venv/lib/python3.8/site-packages/redis/connection.py", line 725, in send_command
    self.send_packed_command(self.pack_command(*args),
  File "/.venv/lib/python3.8/site-packages/redis/connection.py", line 701, in send_packed_command
    self.check_health()
  File "/.venv/lib/python3.8/site-packages/redis/connection.py", line 684, in check_health
    self.send_command('PING', check_health=False)
  File "/.venv/lib/python3.8/site-packages/redis/connection.py", line 725, in send_command
    self.send_packed_command(self.pack_command(*args),
  File "/.venv/lib/python3.8/site-packages/redis/connection.py", line 706, in send_packed_command
    sendall(self._sock, item)
  File "/.venv/lib/python3.8/site-packages/redis/_compat.py", line 9, in sendall
    return sock.sendall(*args, **kwargs)
  File "/.venv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 699, in sendall
    return _sendall(self, data_memory, flags)
  File "/.venv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 389, in _sendall
    len_data_memory = len(data_memory)
RecursionError: maximum recursion depth exceeded while calling a Python object

Is there a limit on the number of retries, or it's apparently a bug in exception handling?

Thanks in advance

[dvora-h]

@zhuang42 Thanks for reporting this! Can you please check if the error occurred in newer versions of redis-py? we are only supports 4.x versions

[zhuang42]

@dvora-h Thanks! This doesn't occur in 4.x versions

[rad-pat]

I just came across this issue with version 4.6.0.
I will see if I can generate a replica.

[rad-pat]

Mine is a very similar issue, but seems to be for Sentinel only. Created a new issue here #2866