Retry downloads interrupted by connnection resets #131
Assignees
Comments
|
@yarikoptic #91 only affected downloads that used the |
|
thanks! added Closes statement within that PR to close this issue |
|
nope - #132 didn't affect this anyhow -- I don't see any retries full protocol{
"resources": {
"core": {
"limit": 5000,
"used": 50,
"remaining": 4950,
"reset": 1628702406
},
"search": {
"limit": 30,
"used": 0,
"remaining": 30,
"reset": 1628701261
},
"graphql": {
"limit": 5000,
"used": 2,
"remaining": 4998,
"reset": 1628702968
},
"integration_manifest": {
"limit": 5000,
"used": 0,
"remaining": 5000,
"reset": 1628704801
},
"source_import": {
"limit": 100,
"used": 0,
"remaining": 100,
"reset": 1628701261
},
"code_scanning_upload": {
"limit": 500,
"used": 0,
"remaining": 500,
"reset": 1628704801
}
},
"rate": {
"limit": 5000,
"used": 50,
"remaining": 4950,
"reset": 1628702406
}
}
2021-08-11T13:00:01-0400 [INFO ] tinuous tinuous 0.5.0+6.gba81776
2021-08-11T13:00:02-0400 [INFO ] tinuous Fetching resources from github
2021-08-11T13:00:02-0400 [INFO ] tinuous Fetching runs newer than 2021-08-11 14:22:32+00:00
2021-08-11T13:00:02-0400 [INFO ] tinuous Fetching runs for workflow .github/workflows/benchmarks.yml (Benchmarks)
2021-08-11T13:00:03-0400 [INFO ] tinuous Found run 3484
2021-08-11T13:00:04-0400 [INFO ] tinuous Downloading logs for benchmarks.yml (Benchmarks) #3484 to 2021/08/11/pr/5883/6ba978e/github-Benchmarks-3484-success
2021-08-11T13:00:04-0400 [INFO ] tinuous Fetching runs for workflow .github/workflows/docbuild.yml (Docs)
2021-08-11T13:00:05-0400 [INFO ] tinuous Found run 3484
2021-08-11T13:00:05-0400 [INFO ] tinuous Logs for docbuild.yml (Docs) #3484 already downloaded to 2021/08/11/pr/5883/6ba978e/github-Docs-3484-success; skipping
2021-08-11T13:00:05-0400 [INFO ] tinuous Fetching runs for workflow .github/workflows/test_extensions.yml (Extensions)
2021-08-11T13:00:05-0400 [INFO ] tinuous Found run 3484
2021-08-11T13:00:05-0400 [INFO ] tinuous Downloading logs for test_extensions.yml (Extensions) #3484 to 2021/08/11/pr/5883/6ba978e/github-Extensions-3484-success
2021-08-11T13:00:07-0400 [INFO ] tinuous Fetching runs for workflow (Docs)
2021-08-11T13:00:08-0400 [INFO ] tinuous Fetching runs for workflow .github/workflows/test_crippled.yml (CrippledFS)
2021-08-11T13:00:09-0400 [INFO ] tinuous Found run 1950
2021-08-11T13:00:09-0400 [INFO ] tinuous Downloading logs for test_crippled.yml (CrippledFS) #1950 to 2021/08/11/pr/5883/6ba978e/github-CrippledFS-1950-success
Traceback (most recent call last):
File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
httplib_response = self._make_request(
File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/urllib3/connectionpool.py", line 445, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/urllib3/connectionpool.py", line 440, in _make_request
httplib_response = conn.getresponse()
File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/http/client.py", line 1345, in getresponse
response.begin()
File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/http/client.py", line 307, in begin
version, status, reason = self._read_status()
File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/http/client.py", line 268, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/socket.py", line 704, in readinto
return self._sock.recv_into(b)
File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/ssl.py", line 1241, in recv_into
return self.read(nbytes, buffer)
File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/ssl.py", line 1099, in read
return self._sslobj.read(len, buffer)
ConnectionResetError: [Errno 104] Connection reset by peer
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/requests/adapters.py", line 439, in send
resp = conn.urlopen(
File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/urllib3/connectionpool.py", line 755, in urlopen
retries = retries.increment(
File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/urllib3/util/retry.py", line 532, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/urllib3/packages/six.py", line 734, in reraise
raise value.with_traceback(tb)
File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
httplib_response = self._make_request(
File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/urllib3/connectionpool.py", line 445, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/urllib3/connectionpool.py", line 440, in _make_request
httplib_response = conn.getresponse()
File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/http/client.py", line 1345, in getresponse
response.begin()
File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/http/client.py", line 307, in begin
version, status, reason = self._read_status()
File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/http/client.py", line 268, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/socket.py", line 704, in readinto
return self._sock.recv_into(b)
File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/ssl.py", line 1241, in recv_into
return self.read(nbytes, buffer)
File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/ssl.py", line 1099, in read
return self._sslobj.read(len, buffer)
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/datalad/miniconda3/envs/tinuous-dev/bin/tinuous", line 33, in <module>
sys.exit(load_entry_point('tinuous', 'console_scripts', 'tinuous')())
File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/click/decorators.py", line 33, in new_func
return f(get_current_context().obj, *args, **kwargs)
File "/mnt/datasets/datalad/ci/tinuous/src/tinuous/__main__.py", line 127, in fetch
paths = obj.download(Path(path))
File "/mnt/datasets/datalad/ci/tinuous/src/tinuous/github.py", line 313, in download
self.client.download_zipfile(self.logs_url, path)
File "/mnt/datasets/datalad/ci/tinuous/src/tinuous/base.py", line 141, in download_zipfile
self.download(path, zippath)
File "/mnt/datasets/datalad/ci/tinuous/src/tinuous/base.py", line 111, in download
r = self.get(path, stream=True)
File "/mnt/datasets/datalad/ci/tinuous/src/tinuous/base.py", line 85, in get
r = self.session.get(url, **kwargs)
File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/requests/sessions.py", line 555, in get
return self.request('GET', url, **kwargs)
File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/requests/sessions.py", line 542, in request
resp = self.send(prep, **send_kwargs)
File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/requests/sessions.py", line 677, in send
history = [resp for resp in gen]
File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/requests/sessions.py", line 677, in <listcomp>
history = [resp for resp in gen]
File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/requests/sessions.py", line 237, in resolve_redirects
resp = self.send(
File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/requests/sessions.py", line 655, in send
r = adapter.send(request, **kwargs)
File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/requests/adapters.py", line 498, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
whenever fixed, please retry on a clone resetting a few commits back to see if somehow run into this with correct retries |
|
@yarikoptic I tried to test #134 on a fresh tinuous configuration for datalad with "since" set to two days ago, but no connection error occurred. |
yarikoptic
added a commit
that referenced
this issue
Jan 11, 2022
Retry downloads that fail with ConnectionError
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I would have thought that it is elderly #84 and #91 addressed it in 0f13cf6 which is
0.3.0~10but I presumably (see #130 ) run 0.5.0+2.g37513a4 on smaug and got following crashgory details
{ "resources": { "core": { "limit": 5000, "used": 88, "remaining": 4912, "reset": 1628691603 }, "search": { "limit": 30, "used": 0, "remaining": 30, "reset": 1628690461 }, "graphql": { "limit": 5000, "used": 6, "remaining": 4994, "reset": 1628691450 }, "integration_manifest": { "limit": 5000, "used": 0, "remaining": 5000, "reset": 1628694001 }, "source_import": { "limit": 100, "used": 0, "remaining": 100, "reset": 1628690461 }, "code_scanning_upload": { "limit": 500, "used": 0, "remaining": 500, "reset": 1628694001 } }, "rate": { "limit": 5000, "used": 88, "remaining": 4912, "reset": 1628691603 } } 2021-08-11T10:00:02-0400 [INFO ] tinuous Fetching resources from github 2021-08-11T10:00:02-0400 [INFO ] tinuous Fetching runs newer than 2021-08-11 09:46:06+00:00 2021-08-11T10:00:02-0400 [INFO ] tinuous Fetching runs for workflow .github/workflows/benchmarks.yml (Benchmarks) 2021-08-11T10:00:04-0400 [INFO ] tinuous Run 3482 not completed; skipping 2021-08-11T10:00:04-0400 [INFO ] tinuous Run 3481 not completed; skipping 2021-08-11T10:00:04-0400 [INFO ] tinuous Fetching runs for workflow .github/workflows/docbuild.yml (Docs) 2021-08-11T10:00:04-0400 [INFO ] tinuous Found run 3482 2021-08-11T10:00:05-0400 [INFO ] tinuous Downloading logs for docbuild.yml (Docs) #3482 to 2021/08/11/pr/5883/4bf98ae/github-Docs-3482-success 2021-08-11T10:00:06-0400 [INFO ] tinuous Found run 3481 2021-08-11T10:00:06-0400 [INFO ] tinuous Downloading logs for docbuild.yml (Docs) #3481 to 2021/08/11/pr/UNK/5cab1c1/github-Docs-3481-success 2021-08-11T10:00:07-0400 [INFO ] tinuous Fetching runs for workflow .github/workflows/test_extensions.yml (Extensions) 2021-08-11T10:00:07-0400 [INFO ] tinuous Run 3482 not completed; skipping 2021-08-11T10:00:07-0400 [INFO ] tinuous Run 3481 not completed; skipping 2021-08-11T10:00:07-0400 [INFO ] tinuous Fetching runs for workflow (Docs) 2021-08-11T10:00:08-0400 [INFO ] tinuous Fetching runs for workflow .github/workflows/test_crippled.yml (CrippledFS) 2021-08-11T10:00:08-0400 [INFO ] tinuous Run 1948 not completed; skipping 2021-08-11T10:00:08-0400 [INFO ] tinuous Run 1947 not completed; skipping 2021-08-11T10:00:08-0400 [INFO ] tinuous Fetching runs for workflow .github/workflows/update-contributors.yml (allcontributors-auto-detect) 2021-08-11T10:00:09-0400 [INFO ] tinuous Fetching runs for workflow .github/workflows/test_macos.yml (Test on macOS) 2021-08-11T10:00:10-0400 [INFO ] tinuous Run 1719 not completed; skipping 2021-08-11T10:00:10-0400 [INFO ] tinuous Run 1718 not completed; skipping 2021-08-11T10:00:10-0400 [INFO ] tinuous Fetching runs for workflow .github/workflows/shellcheck.yml (Shellcheck on scripts) 2021-08-11T10:00:10-0400 [INFO ] tinuous Found run 1669 2021-08-11T10:00:10-0400 [INFO ] tinuous Downloading logs for shellcheck.yml (Shellcheck on scripts) #1669 to 2021/08/11/pr/5883/4bf98ae/github-Shellcheck on scripts-1669-success Traceback (most recent call last): File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen httplib_response = self._make_request( File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/urllib3/connectionpool.py", line 445, in _make_request six.raise_from(e, None) File "<string>", line 3, in raise_from File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/urllib3/connectionpool.py", line 440, in _make_request httplib_response = conn.getresponse() File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/http/client.py", line 1345, in getresponse response.begin() File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/http/client.py", line 307, in begin version, status, reason = self._read_status() File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/http/client.py", line 268, in _read_status line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1") File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/socket.py", line 704, in readinto return self._sock.recv_into(b) File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/ssl.py", line 1241, in recv_into return self.read(nbytes, buffer) File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/ssl.py", line 1099, in read return self._sslobj.read(len, buffer) ConnectionResetError: [Errno 104] Connection reset by peer During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/requests/adapters.py", line 439, in send resp = conn.urlopen( File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/urllib3/connectionpool.py", line 755, in urlopen retries = retries.increment( File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/urllib3/util/retry.py", line 532, in increment raise six.reraise(type(error), error, _stacktrace) File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/urllib3/packages/six.py", line 734, in reraise raise value.with_traceback(tb) File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen httplib_response = self._make_request( File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/urllib3/connectionpool.py", line 445, in _make_request six.raise_from(e, None) File "<string>", line 3, in raise_from File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/urllib3/connectionpool.py", line 440, in _make_request httplib_response = conn.getresponse() File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/http/client.py", line 1345, in getresponse response.begin() File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/http/client.py", line 307, in begin version, status, reason = self._read_status() File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/http/client.py", line 268, in _read_status line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1") File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/socket.py", line 704, in readinto return self._sock.recv_into(b) File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/ssl.py", line 1241, in recv_into return self.read(nbytes, buffer) File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/ssl.py", line 1099, in read return self._sslobj.read(len, buffer) urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/datalad/miniconda3/envs/tinuous-dev/bin/tinuous", line 33, in <module> sys.exit(load_entry_point('tinuous', 'console_scripts', 'tinuous')()) File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/click/core.py", line 829, in __call__ return self.main(*args, **kwargs) File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/click/core.py", line 782, in main rv = self.invoke(ctx) File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/click/core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, **ctx.params) File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/click/core.py", line 610, in invoke return callback(*args, **kwargs) File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/click/decorators.py", line 33, in new_func return f(get_current_context().obj, *args, **kwargs) File "/mnt/datasets/datalad/ci/tinuous/src/tinuous/__main__.py", line 126, in fetch paths = obj.download(Path(path)) File "/mnt/datasets/datalad/ci/tinuous/src/tinuous/github.py", line 318, in download r = self.client.get(self.logs_url) File "/mnt/datasets/datalad/ci/tinuous/src/tinuous/base.py", line 80, in get r = self.session.get(url, **kwargs) File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/requests/sessions.py", line 555, in get return self.request('GET', url, **kwargs) File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/requests/sessions.py", line 542, in request resp = self.send(prep, **send_kwargs) File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/requests/sessions.py", line 677, in send history = [resp for resp in gen] File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/requests/sessions.py", line 677, in <listcomp> history = [resp for resp in gen] File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/requests/sessions.py", line 237, in resolve_redirects resp = self.send( File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/requests/sessions.py", line 655, in send r = adapter.send(request, **kwargs) File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/requests/adapters.py", line 498, in send raise ConnectionError(err, request=request) requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))The text was updated successfully, but these errors were encountered: