Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ConnectionLoss when reading large data over SSL #618

Closed
jeblair opened this issue Jun 22, 2020 · 0 comments · Fixed by #619
Closed

ConnectionLoss when reading large data over SSL #618

jeblair opened this issue Jun 22, 2020 · 0 comments · Fixed by #619

Comments

@jeblair
Copy link
Contributor

jeblair commented Jun 22, 2020

Using kazoo 2.7.0 or git master, I observe the following error
when calling get_children on a node with 8280 children when using
TLS:

Traceback (most recent call last):
  File "../zktest.py", line 17, in <module>
    zk.get_children('/nodepool/requests-lock')
  File "/home/corvus/kazoo/kazoo/client.py", line 1219, in get_children
    include_data=include_data).get()
  File "/home/corvus/kazoo/kazoo/handlers/utils.py", line 75, in get
    raise self._exception
kazoo.exceptions.ConnectionLoss

By removing the exception masking in _socket_error_handling I'm
able to see the underlying exception:

Traceback (most recent call last):
  File "/home/corvus/kazoo/kazoo/protocol/connection.py", line 605, in _connect_attempt
    response = self._read_socket(read_timeout)
  File "/home/corvus/kazoo/kazoo/protocol/connection.py", line 438, in _read_socket
    header, buffer, offset = self._read_header(read_timeout)
  File "/home/corvus/kazoo/kazoo/protocol/connection.py", line 226, in _read_header
    b = self._read(4, timeout)
  File "/home/corvus/kazoo/kazoo/protocol/connection.py", line 254, in _read
    chunk = self._socket.recv(remaining)
  File "/usr/lib/python3.5/ssl.py", line 914, in recv
    return self.read(buflen)
  File "/usr/lib/python3.5/ssl.py", line 791, in read
    return self._sslobj.read(len, buffer)
  File "/usr/lib/python3.5/ssl.py", line 577, in read
    v = self._sslobj.read(len)
ssl.SSLWantReadError: The operation did not complete (read) (_ssl.c:1981)

The OpenSSL documentation indicates that when we receive this, we should call the function
again when the underlying socket is readable. The same thing is
true for writing as well.

It is possible this bug is related to issues #587 and #580 as well.

jeblair pushed a commit to jeblair/kazoo that referenced this issue Jun 22, 2020
This adds a simple recovery path in case an SSL connection receives
an SSL_WANT_READ or WRITE error.  Either error can occur while
reading or writing.  The error indicates that the underlying
operation should be retried after the socket is once again readable
or writable (per the error code).

Fixes python-zk#618
jeblair pushed a commit to jeblair/kazoo that referenced this issue Jun 22, 2020
This adds a simple recovery path in case an SSL connection receives
an SSL_WANT_READ or WRITE error.  Either error can occur while
reading or writing.  The error indicates that the underlying
operation should be retried after the socket is once again readable
or writable (per the error code).

Closes python-zk#618
jeblair pushed a commit to jeblair/kazoo that referenced this issue Jun 26, 2020
This adds a simple recovery path in case an SSL connection receives
an SSL_WANT_READ or WRITE error.  Either error can occur while
reading or writing.  The error indicates that the underlying
operation should be retried after the socket is once again readable
or writable (per the error code).

Closes python-zk#618
jeblair pushed a commit to jeblair/kazoo that referenced this issue Jun 26, 2020
This adds a simple recovery path in case an SSL connection receives
an SSL_WANT_READ or WRITE error.  Either error can occur while
reading or writing.  The error indicates that the underlying
operation should be retried after the socket is once again readable
or writable (per the error code).

Closes python-zk#618
StephenSorriaux pushed a commit that referenced this issue Jun 27, 2020
This adds a simple recovery path in case an SSL connection receives
an SSL_WANT_READ or WRITE error.  Either error can occur while
reading or writing.  The error indicates that the underlying
operation should be retried after the socket is once again readable
or writable (per the error code).

Closes #618

Co-authored-by: James E. Blair <jeblair@redhat.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant