-
Notifications
You must be signed in to change notification settings - Fork 239
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Debug Circle CI failures #562
Conversation
Something mega weird is going on at CircleCI. Check this out. static:~ distiller$ curl -s 'https://www.python.org/ftp/python/3.9.1/python-3.9.1-macos11.0.pkg' | md5
8b19748473609241e60aa3618bbaf3ed
static:~ distiller$ curl -s 'https://www.python.org/ftp/python/3.9.1/python-3.9.1-macos11.0.pkg' | md5
6787224326c386c83b8577caa2c6a208
static:~ distiller$ curl -s 'https://www.python.org/ftp/python/3.9.1/python-3.9.1-macos11.0.pkg' | md5
0b54c89e6862d3b93bb471b847006bb7
static:~ distiller$ curl -s 'https://www.python.org/ftp/python/3.9.1/python-3.9.1-macos11.0.pkg' | md5
ad477dc2d090bc137f5a8d1741f1d127
static:~ distiller$ curl -s 'https://www.python.org/ftp/python/3.9.1/python-3.9.1-macos11.0.pkg' | md5
05335b79f29ee06eaf4c1f766b7e42c9
static:~ distiller$ curl -s 'https://www.python.org/ftp/python/3.9.1/python-3.9.1-macos11.0.pkg' | md5
8b19748473609241e60aa3618bbaf3ed
static:~ distiller$ curl -s 'https://www.python.org/ftp/python/3.9.1/python-3.9.1-macos11.0.pkg' | md5
8b19748473609241e60aa3618bbaf3ed
static:~ distiller$ curl -s 'https://www.python.org/ftp/python/3.9.1/python-3.9.1-macos11.0.pkg' | md5
8b19748473609241e60aa3618bbaf3ed ^ yes, same URL every time, but if I keep doing it, sometimes I get an error, other times, a corrupt file. The correct hash is |
Curl happens to be using HTTP/2, which I'm guessing static:~ distiller$ curl -s --http1.1 'https://www.python.org/ftp/python/3.9.1/python-3.9.1-macos11.0.pkg' | md5
8b19748473609241e60aa3618bbaf3ed
static:~ distiller$ curl -s --http1.1 'https://www.python.org/ftp/python/3.9.1/python-3.9.1-macos11.0.pkg' | md5
8b19748473609241e60aa3618bbaf3ed
static:~ distiller$ curl -s --http1.1 'https://www.python.org/ftp/python/3.9.1/python-3.9.1-macos11.0.pkg' | md5
8b19748473609241e60aa3618bbaf3ed
static:~ distiller$ curl -s --http1.1 'https://www.python.org/ftp/python/3.9.1/python-3.9.1-macos11.0.pkg' | md5
8b19748473609241e60aa3618bbaf3ed
static:~ distiller$ curl -s --http1.1 'https://www.python.org/ftp/python/3.9.1/python-3.9.1-macos11.0.pkg' | md5
0894f56c96b518cb00cbb1251000df08
static:~ distiller$ curl -s --http1.1 'https://www.python.org/ftp/python/3.9.1/python-3.9.1-macos11.0.pkg' | md5
8b19748473609241e60aa3618bbaf3ed |
Can we add and check the hash? I was going to suggest that anyway, as it would catch this sooner instead of trying to install a corrupted file, which is a little worrisome. |
Did this start happening on Saturday? Seems like maybe it was around a bit more than that? They had a database maintenance job then. https://status.circleci.com |
Good request:
Corrupt request:
|
Thing is, these are HTTPS connections. So I don't see how Circle could be MITM'ing it. So I'm thinking it's a Fastly issue. Dodgy CDN node perhaps... the problems only seem to be happening when curl connects to |
I suppose we could, yes. But there is a built-in assumption of cibuildwheel that we trust HTTPS... after all, it's downloaded from pip over HTTPS in the first place. |
I've filed a support request with Fastly... we'll see... |
You can always add hashes to your requirements, like manylinux does: https://github.com/pypa/manylinux/blob/master/docker/build_scripts/requirements.txt |
I know, I wanted to add them to our constraints in #256, but it wasn't possible at the time. See #256 (comment) for the gory details. |
The "everything pinned" would be an issue that might eventually be solvable, but the nice thing with hashes for the Python versions would be that it would have shown the "correct" error (invalid Python.pkg) immediately for something like this, reducing debugging. It should be pretty easy to add to our collection update script, I'd think, now that we have that. |
Fastly think this might be fixed... let's see. |
Nope. |
I think the close/reopen trick doesn't work on circle CI, that run is from 4 days ago |
Wow, that usually works; I think you are right. How about a |
I've added a line to print the md5, which might be useful if it doesn't work! |
😂 I can look at adding a md5 check to the Python download, that would help the error be better in the future. Though I think we should get #484 in first, I've been holding off waiting for that to land. ;) |
Passed, let's try again... |
That's 3 passes in a row, I think it's good :) |
It might be related to the use of Python 3.6on Circle... or something else. I'm adding more logging.