-
-
Notifications
You must be signed in to change notification settings - Fork 30.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
test_ssl: test_get_server_certificate() and test_msg_callback_deadlock_bpo43577() fail randomly on the macOS CI #88395
Comments
See https://github.com/python/cpython/pull/26104/checks?check_run_id=2662511684 ====================================================================== Traceback (most recent call last):
File "/Users/runner/work/cpython/cpython/Lib/test/test_ssl.py", line 2132, in test_get_server_certificate
_test_get_server_certificate(self, *self.server_addr, cert=SIGNING_CA)
File "/Users/runner/work/cpython/cpython/Lib/test/test_ssl.py", line 2329, in _test_get_server_certificate
pem = ssl.get_server_certificate((host, port), ca_certs=cert)
File "/Users/runner/work/cpython/cpython/Lib/ssl.py", line 1520, in get_server_certificate
with create_connection(addr, timeout=timeout) as sock:
File "/Users/runner/work/cpython/cpython/Lib/socket.py", line 844, in create_connection
raise err
File "/Users/runner/work/cpython/cpython/Lib/socket.py", line 832, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused |
The problem could be related to bpo-43921. I neither have a macOS nor a Windows machine to reproduce and debug the issue. Since I'm cannot reproduce the problem on Linux, I'm unable to debug and fix it. |
Can you maybe use a VM for Windows? |
I’ll see if I can provoke it on my Mac. What’s the preferred solution?
Maybe 2. is acceptable :) |
Is possible that these are flaky tests but I am afraid this may be something more important so o would prefer to go with 3 ;) |
I feared that ;) |
Getting there. The error message is similar: $ ./python.exe -m test test_ssl -m test_get_server_certificate -u all -F
...
0:02:11 load avg: 2.10 [328] test_ssl
test test_ssl failed -- Traceback (most recent call last):
File "/Users/erlendaasland/src/cpython-ssl/Lib/test/test_ssl.py", line 2132, in test_get_server_certificate
_test_get_server_certificate(self, *self.server_addr, cert=SIGNING_CA)
File "/Users/erlendaasland/src/cpython-ssl/Lib/test/test_ssl.py", line 2329, in _test_get_server_certificate
pem = ssl.get_server_certificate((host, port), ca_certs=cert)
File "/Users/erlendaasland/src/cpython-ssl/Lib/ssl.py", line 1521, in get_server_certificate
with context.wrap_socket(sock, server_hostname=host) as sslsock:
File "/Users/erlendaasland/src/cpython-ssl/Lib/ssl.py", line 518, in wrap_socket
return self.sslsocket_class._create(
File "/Users/erlendaasland/src/cpython-ssl/Lib/ssl.py", line 1070, in _create
self.do_handshake()
File "/Users/erlendaasland/src/cpython-ssl/Lib/ssl.py", line 1339, in do_handshake
self._sslobj.do_handshake()
ConnectionResetError: [Errno 54] Connection reset by peer test_ssl failed == Tests result: FAILURE == |
Got it: 0:02:29 load avg: 2.81 [389] test_ssl
test test_ssl failed -- Traceback (most recent call last):
File "/Users/erlendaasland/src/cpython-ssl/Lib/test/test_ssl.py", line 2132, in test_get_server_certificate
_test_get_server_certificate(self, *self.server_addr, cert=SIGNING_CA)
File "/Users/erlendaasland/src/cpython-ssl/Lib/test/test_ssl.py", line 2329, in _test_get_server_certificate
pem = ssl.get_server_certificate((host, port), ca_certs=cert)
File "/Users/erlendaasland/src/cpython-ssl/Lib/ssl.py", line 1520, in get_server_certificate
with create_connection(addr, timeout=timeout) as sock:
File "/Users/erlendaasland/src/cpython-ssl/Lib/socket.py", line 844, in create_connection
raise err
File "/Users/erlendaasland/src/cpython-ssl/Lib/socket.py", line 832, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused test_ssl failed == Tests result: FAILURE == |
$ git diff
diff --git a/Lib/ssl.py b/Lib/ssl.py
index 2b131de043..9c281d8028 100644
--- a/Lib/ssl.py
+++ b/Lib/ssl.py
@@ -257,8 +257,9 @@ class _TLSMessageType:
if sys.platform == "win32":
from _ssl import enum_certificates, enum_crls
+from test.support import SHORT_TIMEOUT as _GLOBAL_DEFAULT_TIMEOUT
from socket import socket, SOCK_STREAM, create_connection
-from socket import SOL_SOCKET, SO_TYPE, _GLOBAL_DEFAULT_TIMEOUT
+from socket import SOL_SOCKET, SO_TYPE
import socket as _socket
import base64 # for DER-to-PEM translation
import errno $ % ./python.exe -m test test_ssl -m test_get_server_certificate -u all -F
...
0:03:27 load avg: 2.24 [535] test_ssl
test test_ssl failed -- Traceback (most recent call last):
File "/Users/erlendaasland/src/cpython-ssl/Lib/test/test_ssl.py", line 2132, in test_get_server_certificate
_test_get_server_certificate(self, *self.server_addr, cert=SIGNING_CA)
File "/Users/erlendaasland/src/cpython-ssl/Lib/test/test_ssl.py", line 2329, in _test_get_server_certificate
pem = ssl.get_server_certificate((host, port), ca_certs=cert)
File "/Users/erlendaasland/src/cpython-ssl/Lib/ssl.py", line 1521, in get_server_certificate
with create_connection(addr, timeout=timeout) as sock:
File "/Users/erlendaasland/src/cpython-ssl/Lib/socket.py", line 844, in create_connection
raise err
File "/Users/erlendaasland/src/cpython-ssl/Lib/socket.py", line 832, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused test_ssl failed == Tests result: FAILURE == |
Erlend, could you maybe add some context to the messages, unfortunately, I am failing to understand what they mean :( |
me, too. Did the change of import for timeout setting cause the issue? |
Yes, sorry ‘bout the cryptic messages :) This is against current main branch (I’m not at my computer right now, so I don’t have the git ref). The first two failures came after 328 and 389 runs. Increasing the timeout (using test.support.SHORT_TIMEOUT) made it harder trigging the failure (~500 repetitions). I’ll try to declutter when I’m back on my computer. |
New failure in CI: https://github.com/python/cpython/runs/2669949777?check_suite_focus=true ====================================================================== Traceback (most recent call last):
File "/Users/runner/work/cpython/cpython/Lib/test/test_ssl.py", line 4813, in test_msg_callback_deadlock_bpo43577
s.connect((HOST, server.port))
File "/Users/runner/work/cpython/cpython/Lib/ssl.py", line 1372, in connect
self._real_connect(addr, False)
File "/Users/runner/work/cpython/cpython/Lib/ssl.py", line 1359, in _real_connect
super().connect(addr)
ConnectionRefusedError: [Errno 61] Connection refused |
Build info for macOS CI attached. |
Here's the output when with test.support.verbose = True: test_ssl: testing with 'OpenSSL 1.1.1k 25 Mar 2021' (1, 1, 1, 11, 15) server: bad connection attempt from ('127.0.0.1', 51163):
Traceback (most recent call last):
File "/Users/erlendaasland/src/cpython-ssl/Lib/test/test_ssl.py", line 2366, in wrap_conn
self.sslconn = self.server.context.wrap_socket(
File "/Users/erlendaasland/src/cpython-ssl/Lib/ssl.py", line 518, in wrap_socket
return self.sslsocket_class._create(
File "/Users/erlendaasland/src/cpython-ssl/Lib/ssl.py", line 1070, in _create
self.do_handshake()
File "/Users/erlendaasland/src/cpython-ssl/Lib/ssl.py", line 1339, in do_handshake
self._sslobj.do_handshake()
OSError: [Errno 41] Protocol wrong type for socket
ERROR ====================================================================== Traceback (most recent call last):
File "/Users/erlendaasland/src/cpython-ssl/Lib/test/test_ssl.py", line 2130, in test_get_server_certificate
_test_get_server_certificate(self, *self.server_addr, cert=SIGNING_CA)
File "/Users/erlendaasland/src/cpython-ssl/Lib/test/test_ssl.py", line 2327, in _test_get_server_certificate
pem = ssl.get_server_certificate((host, port), ca_certs=cert)
File "/Users/erlendaasland/src/cpython-ssl/Lib/ssl.py", line 1520, in get_server_certificate
with create_connection(addr, timeout=timeout) as sock:
File "/Users/erlendaasland/src/cpython-ssl/Lib/socket.py", line 844, in create_connection
raise err
File "/Users/erlendaasland/src/cpython-ssl/Lib/socket.py", line 832, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused ---------------------------------------------------------------------- The mock server fails with EPROTOTYPE, thus the client fails with connection refused. I found similar issues after doing some googling:
This might be the same macOS bug. |
It could be, but the same thing happens on Windows |
Are you sure? All the reports are from macOS CI. Christian mentioned that it could be similar to another ssl test failure that also happen on Windows, but those tracebacks do not match the ones in this issue. |
FWIW: 1st shell: 2nd shell: The syscall trace shows that errno 41 is returned by write(): |
I am certainly not sure, but I think they are related. |
Has these issues (ConnectionRefusedError/ConnectionResetError during handshake) been reported on other systems than macOS? AFAICS, they've only been reported for the macOS CI. |
Only on MacOS afaik |
Using this patch, I'm (so far) no longer able to reproduce these test failures: diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py
index cee97a8302..c9c1546467 100644
--- a/Lib/test/test_ssl.py
+++ b/Lib/test/test_ssl.py
@@ -2443,9 +2443,13 @@ def wrap_conn(self):
self.server.conn_errors.append(str(e))
if self.server.chatty:
handle_error("\n server: bad connection attempt from " + repr(self.addr) + ":\n")
- self.running = False
- self.server.stop()
- self.close()
+
+ # bpo-44229, bpo-43855, bpo-44237, and bpo-33450:
+ # Ignore spurious EPROTOTYPE returned by write() on macOS.
+ if e.errno != errno.EPROTOTYPE and sys.platform != "darwin":
+ self.running = False
+ self.server.stop()
+ self.close()
return False
else:
self.server.shared_ciphers.append(self.sslconn.shared_ciphers()) Running for 40 minutes now, right now passing 6k iterations for test_get_server_certificate, test_msg_callback_deadlock_bpo43577, and test_get_server_certificate_sni. |
... should perhaps also return True in that case, but it does not seem to have any thing to say. |
I can reproduce these issues easily on 3.7, 3.8, and 3.9: $ python3.7 -m test test_ssl -u all -F -m test_get_server_certificate -v
[...]
test_get_server_certificate (test.test_ssl.SimpleBackgroundTests) ... server: new connection from ('127.0.0.1', 57294) server: bad connection attempt from ('127.0.0.1', 57294):
Traceback (most recent call last):
File "/Users/erlendaasland/install/lib/python3.7/test/test_ssl.py", line 2313, in wrap_conn
self.sock, server_side=True)
File "/Users/erlendaasland/install/lib/python3.7/ssl.py", line 423, in wrap_socket
session=session
File "/Users/erlendaasland/install/lib/python3.7/ssl.py", line 870, in _create
self.do_handshake()
File "/Users/erlendaasland/install/lib/python3.7/ssl.py", line 1139, in do_handshake
self._sslobj.do_handshake()
OSError: [Errno 41] Protocol wrong type for socket
ERROR ====================================================================== Traceback (most recent call last):
File "/Users/erlendaasland/install/lib/python3.7/test/test_ssl.py", line 2108, in test_get_server_certificate
_test_get_server_certificate(self, *self.server_addr, cert=SIGNING_CA)
File "/Users/erlendaasland/install/lib/python3.7/test/test_ssl.py", line 2273, in _test_get_server_certificate
pem = ssl.get_server_certificate((host, port), ca_certs=cert)
File "/Users/erlendaasland/install/lib/python3.7/ssl.py", line 1313, in get_server_certificate
with create_connection(addr) as sock:
File "/Users/erlendaasland/install/lib/python3.7/socket.py", line 728, in create_connection
raise err
File "/Users/erlendaasland/install/lib/python3.7/socket.py", line 716, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused $ python3.8 -m test test_ssl -u all -F -m test_get_server_certificate -v
test_ssl: testing with 'OpenSSL 1.1.1k 25 Mar 2021' (1, 1, 1, 11, 15)
under Mac ('10.16', ('', '', ''), 'x86_64')
HAS_SNI = True
OP_ALL = 0x80000054
OP_NO_TLSv1_1 = 0x10000000
test_get_server_certificate (test.test_ssl.SimpleBackgroundTests) ... server: new connection from ('127.0.0.1', 59370) server: bad connection attempt from ('127.0.0.1', 59370):
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/test/test_ssl.py", line 2348, in wrap_conn
self.sslconn = self.server.context.wrap_socket(
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/ssl.py", line 500, in wrap_socket
return self.sslsocket_class._create(
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/ssl.py", line 1040, in _create
self.do_handshake()
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/ssl.py", line 1309, in do_handshake
self._sslobj.do_handshake()
OSError: [Errno 41] Protocol wrong type for socket
ERROR ====================================================================== Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/test/test_ssl.py", line 2142, in test_get_server_certificate
_test_get_server_certificate(self, *self.server_addr, cert=SIGNING_CA)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/test/test_ssl.py", line 2309, in _test_get_server_certificate
pem = ssl.get_server_certificate((host, port), ca_certs=cert)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/ssl.py", line 1484, in get_server_certificate
with context.wrap_socket(sock) as sslsock:
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/ssl.py", line 500, in wrap_socket
return self.sslsocket_class._create(
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/ssl.py", line 1040, in _create
self.do_handshake()
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/ssl.py", line 1309, in do_handshake
self._sslobj.do_handshake()
ConnectionResetError: [Errno 54] Connection reset by peer $ python3.9 -m test test_ssl -u all -F -m test_get_server_certificate -v
[...]
test_ssl: testing with 'OpenSSL 1.1.1k 25 Mar 2021' (1, 1, 1, 11, 15)
under Mac ('10.16', ('', '', ''), 'x86_64')
HAS_SNI = True
OP_ALL = 0x80000054
OP_NO_TLSv1_1 = 0x10000000
test_get_server_certificate (test.test_ssl.SimpleBackgroundTests) ... server: new connection from ('127.0.0.1', 60096) server: bad connection attempt from ('127.0.0.1', 60096):
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/test/test_ssl.py", line 2353, in wrap_conn
self.sslconn = self.server.context.wrap_socket(
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 500, in wrap_socket
return self.sslsocket_class._create(
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 1040, in _create
self.do_handshake()
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 1309, in do_handshake
self._sslobj.do_handshake()
OSError: [Errno 41] Protocol wrong type for socket
ERROR ====================================================================== Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/test/test_ssl.py", line 2147, in test_get_server_certificate
_test_get_server_certificate(self, *self.server_addr, cert=SIGNING_CA)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/test/test_ssl.py", line 2314, in _test_get_server_certificate
pem = ssl.get_server_certificate((host, port), ca_certs=cert)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 1484, in get_server_certificate
with context.wrap_socket(sock) as sslsock:
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 500, in wrap_socket
return self.sslsocket_class._create(
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 1040, in _create
self.do_handshake()
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 1309, in do_handshake
self._sslobj.do_handshake()
ConnectionResetError: [Errno 54] Connection reset by peer It does not seem to be specific to 3.10/3.11. |
FYI, here's a CI failure against 3.9: |
http://erickt.github.io/blog/2014/11/19/adventures-in-debugging-a-potential-osx-kernel-bug/ says: "If we trigger a send while the kernel is in the middle of tearing down the socket, it returns EPROTOTYPE." (instead of ECONNRESET) Maybe send() could raise a ConnectionResetError exception on EPROTOTYPE? |
But here, it is write() that returns EPROTOTYPE. See msg394798. |
I'd be interested in hearing Ronald's opinion. (Added to nosy.) |
test_ssl
, ignore spurious EPROTOTYPE on macOS #26893Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: