retry download if connection breaks

[yarikoptic]

while trying to fetch artifacts for datalad/git-annex (in my next greedy attempt to go even further back in history)

fetch crashed with the following

2021-06-09T13:45:02-0400 [INFO    ] tinuous Downloading asset git-annex-macos-dmg_8.20210310+git76-ga76f16cd8_x64 for build-macos.yaml (Build git-annex on macOS) #213 to builds/2021/03/pr-UNK/build-macos.yaml-213-72e6c71d-failed
Traceback (most recent call last):
  File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/urllib3/response.py", line 438, in _error_catcher
    yield
  File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/urllib3/response.py", line 764, in read_chunked
    self._update_chunk_length()
  File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/urllib3/response.py", line 694, in _update_chunk_length
    line = self._fp.fp.readline()
  File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/socket.py", line 704, in readinto
    return self._sock.recv_into(b)
  File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/ssl.py", line 1241, in recv_into
    return self.read(nbytes, buffer)
  File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/ssl.py", line 1099, in read
    return self._sslobj.read(len, buffer)
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/requests/models.py", line 753, in generate
    for chunk in self.raw.stream(chunk_size, decode_content=True):
  File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/urllib3/response.py", line 572, in stream
    for line in self.read_chunked(amt, decode_content=decode_content):
  File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/urllib3/response.py", line 793, in read_chunked
    self._original_response.close()
  File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/contextlib.py", line 135, in __exit__
    self.gen.throw(type, value, traceback)
  File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/urllib3/response.py", line 455, in _error_catcher
    raise ProtocolError("Connection broken: %r" % e, e)
urllib3.exceptions.ProtocolError: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/datalad/miniconda3/envs/tinuous-dev/bin/tinuous", line 33, in <module>
    sys.exit(load_entry_point('tinuous', 'console_scripts', 'tinuous')())
  File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context().obj, *args, **kwargs)
  File "/mnt/datasets/datalad/ci/tinuous/src/tinuous/__main__.py", line 126, in fetch
    paths = obj.download(Path(path))
  File "/mnt/datasets/datalad/ci/tinuous/src/tinuous/github.py", line 337, in download
    stream_to_file(r, zippath)
  File "/mnt/datasets/datalad/ci/tinuous/src/tinuous/util.py", line 32, in stream_to_file
    for chunk in r.iter_content(chunk_size=8192):
  File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/requests/models.py", line 756, in generate
    raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))

tinuous 0.2.0+43.gdd48e39 . And I think it should have just retried

my github limits look... oddly unused

{
  "resources": {
    "core": {
      "limit": 5000,
      "used": 7,
      "remaining": 4993,
      "reset": 1623264286
    },
    "search": {
      "limit": 30,
      "used": 0,
      "remaining": 30,
      "reset": 1623260969
    },
    "graphql": {
      "limit": 5000,
      "used": 1,
      "remaining": 4999,
      "reset": 1623262218
    },
    "integration_manifest": {
      "limit": 5000,
      "used": 0,
      "remaining": 5000,
      "reset": 1623264509
    },
    "source_import": {
      "limit": 100,
      "used": 0,
      "remaining": 100,
      "reset": 1623260969
    },
    "code_scanning_upload": {
      "limit": 500,
      "used": 0,
      "remaining": 500,
      "reset": 1623264509
    }
  },
  "rate": {
    "limit": 5000,
    "used": 7,
    "remaining": 4993,
    "reset": 1623264286
  }
}

rerunning fetch seems to work

2021-06-09T13:50:05-0400 [INFO    ] tinuous Asset git-annex-macos-dmg_8.20210310+git76-ga76f16cd8_x64 from build-macos.yaml (Build git-annex on macOS) #214 already downloaded to builds/2021/03/cron-20210319/build-macos.yaml-214-b4756b5d-success; skipping
2021-06-09T13:50:05-0400 [INFO    ] tinuous Found run 213
2021-06-09T13:50:05-0400 [INFO    ] tinuous Logs for build-macos.yaml (Build git-annex on macOS) #213 already downloaded to builds/2021/03/pr-UNK/build-macos.yaml-213-72e6c71d-failed; skipping
2021-06-09T13:50:05-0400 [INFO    ] tinuous Downloading asset git-annex-macos-dmg_8.20210310+git76-ga76f16cd8_x64 for build-macos.yaml (Build git-annex on macOS) #213 to builds/2021/03/pr-UNK/build-macos.yaml-213-72e6c71d-failed
2021-06-09T13:50:08-0400 [INFO    ] tinuous Found run 212
2021-06-09T13:50:08-0400 [INFO    ] tinuous Downloading logs for build-macos.yaml (Build git-annex on macOS) #212 to builds/2021/03/pr-UNK/build-macos.yaml-212-a3c35cb2-failed
2021-06-09T13:50:12-0400 [INFO    ] tinuous Downloading asset git-annex-macos-dmg_8.20210310+git76-ga76f16cd8_x64 for build-macos.yaml (Build git-annex on macOS) #212 to builds/2021/03/pr-UNK/build-macos.yaml-212-a3c35cb2-failed
2021-06-09T13:50:16-0400 [INFO    ] tinuous Found run 211
2021-06-09T13:50:16-0400 [INFO    ] tinuous Downloading logs for build-macos.yaml (Build git-annex on macOS) #211 to builds/2021/03/cron-20210318/build-macos.yaml-211-b4756b5d-success

so it is indeed just a matter of being resilient to unstable connections etc.

[yarikoptic]

gkhm... as soon as I submitted, it crashed again similarly on some later file

2021-06-09T13:50:43-0400 [INFO    ] tinuous Found run 208
2021-06-09T13:50:43-0400 [INFO    ] tinuous Downloading logs for build-macos.yaml (Build git-annex on macOS) #208 to builds/2021/03/pr-UNK/build-macos.yaml-208-2f96d991-failed
2021-06-09T13:50:47-0400 [INFO    ] tinuous Downloading asset git-annex-macos-dmg_8.20210310+git65-gf1bff235e_x64 for build-macos.yaml (Build git-annex on macOS) #208 to builds/2021/03/pr-UNK/build-macos.yaml-208-2f96d991-failed
Traceback (most recent call last):
...

after a few successful ones. So it seems situation is chronic and should be great to get it auto-handled ;)

[jwodder]

@yarikoptic Are you only seeing this while downloading assets? Retrying on this type of connection error is easy except for requests that go through Pygithub.

[yarikoptic]

insofar it seems only github actions, e.g. happened again while downloading an asset . Didn't spot it happening for other CIs yet.