Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

get: throwing 2 network errors #6709

Closed
jbencook opened this issue Sep 29, 2021 · 6 comments
Closed

get: throwing 2 network errors #6709

jbencook opened this issue Sep 29, 2021 · 6 comments
Assignees
@isidentical
Labels
bug Did we break something? fs: http Related to the HTTP filesystem

Comments

@jbencook
Copy link

Bug Report

Description

I'm randomly getting two different errors when I try to use dvc get to download the cats-dogs example dataset. They seem to be network issues.

Sometimes the terminal gets spammed with the following:

AttributeError: 'RetryClient' object has no attribute '_connector'

And sometimes I get the following:

ERROR: unexpected error - Cannot connect to host http://s3-us-east-2.amazonaws.com:443/ ssl:default [nodename nor servname provided, or not known]: [Errno 8] nodename nor servname provided, or not known

This is happening in the latest release and the development version.

Reproduce

dvc get https://github.com/iterative/dataset-registry use-cases/cats-dogs

To re-run, I just rm -rf cats-dogs and then do it again.

Expected

I expect the command to finish with no errors and data to be in the cats-dogs folder.

Environment information

This was happening for me on macOS BigSur 11.6 with the brew installed version and a version installed in an Anaconda environment.

Output of dvc doctor:

DVC version: 2.7.4 (osxpkg)
---------------------------------
Platform: Python 3.8.12 on macOS-10.16-x86_64-i386-64bit
Supports:
	azure (adlfs = 2021.9.1, knack = 0.8.2, azure-identity = 1.6.1),
	gdrive (pydrive2 = 1.9.4),
	gs (gcsfs = 2021.8.1),
	hdfs (pyarrow = 5.0.0),
	webhdfs (hdfs = 2.6.0),
	http (aiohttp = 3.7.4.post0, aiohttp-retry = 2.4.5),
	https (aiohttp = 3.7.4.post0, aiohttp-retry = 2.4.5),
	s3 (s3fs = 2021.8.1, boto3 = 1.17.106),
	ssh (sshfs = 2021.8.1),
	oss (ossfs = 2021.8.0),
	webdav (webdav4 = 0.9.2),
	webdavs (webdav4 = 0.9.2)
@dberenbaum
Copy link
Collaborator

I'm able to reproduce with the latest dev version. It appears to have been introduced in #6565. Here's the full output:

$ dvc -v get https://github.com/iterative/dataset-registry use-cases/cats-dogs
  1% Downloading directory|▎                         |31/2800 [00:02<02:12, 20.92Files/s]Exception ignored in: <finalize object at 0x11021e360; dead>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/weakref.py", line 580, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/opt/homebrew/lib/python3.9/site-packages/fsspec/implementations/http.py", line 126, in close_session
    if session._connector is not None:
AttributeError: 'RetryClient' object has no attribute '_connector'
Exception ignored in: <finalize object at 0x11021e320; dead>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/weakref.py", line 580, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/opt/homebrew/lib/python3.9/site-packages/fsspec/implementations/http.py", line 126, in close_session
    if session._connector is not None:
AttributeError: 'RetryClient' object has no attribute '_connector'
Exception ignored in: <finalize object at 0x11021e300; dead>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/weakref.py", line 580, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/opt/homebrew/lib/python3.9/site-packages/fsspec/implementations/http.py", line 126, in close_session
    if session._connector is not None:
AttributeError: 'RetryClient' object has no attribute '_connector'
Exception ignored in: <finalize object at 0x11021e2e0; dead>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/weakref.py", line 580, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/opt/homebrew/lib/python3.9/site-packages/fsspec/implementations/http.py", line 126, in close_session
    if session._connector is not None:
AttributeError: 'RetryClient' object has no attribute '_connector'
Exception ignored in: <finalize object at 0x11021e2c0; dead>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/weakref.py", line 580, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/opt/homebrew/lib/python3.9/site-packages/fsspec/implementations/http.py", line 126, in close_session
    if session._connector is not None:
AttributeError: 'RetryClient' object has no attribute '_connector'
Exception ignored in: <finalize object at 0x11021e0c0; dead>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/weakref.py", line 580, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/opt/homebrew/lib/python3.9/site-packages/fsspec/implementations/http.py", line 126, in close_session
    if session._connector is not None:
AttributeError: 'RetryClient' object has no attribute '_connector'
Exception ignored in: <finalize object at 0x11021e080; dead>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/weakref.py", line 580, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/opt/homebrew/lib/python3.9/site-packages/fsspec/implementations/http.py", line 126, in close_session
    if session._connector is not None:
AttributeError: 'RetryClient' object has no attribute '_connector'
Exception ignored in: <finalize object at 0x110296ba0; dead>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/weakref.py", line 580, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/opt/homebrew/lib/python3.9/site-packages/fsspec/implementations/http.py", line 126, in close_session
    if session._connector is not None:
AttributeError: 'RetryClient' object has no attribute '_connector'
Exception ignored in: <finalize object at 0x110296c40; dead>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/weakref.py", line 580, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/opt/homebrew/lib/python3.9/site-packages/fsspec/implementations/http.py", line 126, in close_session
    if session._connector is not None:
AttributeError: 'RetryClient' object has no attribute '_connector'
Exception ignored in: <finalize object at 0x110296ca0; dead>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/weakref.py", line 580, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/opt/homebrew/lib/python3.9/site-packages/fsspec/implementations/http.py", line 126, in close_session
    if session._connector is not None:
AttributeError: 'RetryClient' object has no attribute '_connector'
Exception ignored in: <finalize object at 0x110296cc0; dead>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/weakref.py", line 580, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/opt/homebrew/lib/python3.9/site-packages/fsspec/implementations/http.py", line 126, in close_session
    if session._connector is not None:
AttributeError: 'RetryClient' object has no attribute '_connector'
Exception ignored in: <finalize object at 0x110296ce0; dead>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/weakref.py", line 580, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/opt/homebrew/lib/python3.9/site-packages/fsspec/implementations/http.py", line 126, in close_session
    if session._connector is not None:
AttributeError: 'RetryClient' object has no attribute '_connector'
Exception ignored in: <finalize object at 0x110296d00; dead>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/weakref.py", line 580, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/opt/homebrew/lib/python3.9/site-packages/fsspec/implementations/http.py", line 126, in close_session
    if session._connector is not None:
AttributeError: 'RetryClient' object has no attribute '_connector'
Exception ignored in: <finalize object at 0x110296d20; dead>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/weakref.py", line 580, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/opt/homebrew/lib/python3.9/site-packages/fsspec/implementations/http.py", line 126, in close_session
    if session._connector is not None:
AttributeError: 'RetryClient' object has no attribute '_connector'
Exception ignored in: <finalize object at 0x110296d60; dead>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/weakref.py", line 580, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/opt/homebrew/lib/python3.9/site-packages/fsspec/implementations/http.py", line 126, in close_session
    if session._connector is not None:
AttributeError: 'RetryClient' object has no attribute '_connector'
Exception ignored in: <finalize object at 0x110296da0; dead>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/weakref.py", line 580, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/opt/homebrew/lib/python3.9/site-packages/fsspec/implementations/http.py", line 126, in close_session
    if session._connector is not None:
AttributeError: 'RetryClient' object has no attribute '_connector'
  1% Downloading directory|▎                         |33/2800 [00:19<00:59, 46.32Files/s]Exception ignored in: <finalize object at 0x110296dc0; dead>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/weakref.py", line 580, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/opt/homebrew/lib/python3.9/site-packages/fsspec/implementations/http.py", line 126, in close_session
    if session._connector is not None:
AttributeError: 'RetryClient' object has no attribute '_connector'
Exception ignored in: <finalize object at 0x110296e00; dead>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/weakref.py", line 580, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/opt/homebrew/lib/python3.9/site-packages/fsspec/implementations/http.py", line 126, in close_session
    if session._connector is not None:
AttributeError: 'RetryClient' object has no attribute '_connector'
Exception ignored in: <finalize object at 0x110296e40; dead>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/weakref.py", line 580, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/opt/homebrew/lib/python3.9/site-packages/fsspec/implementations/http.py", line 126, in close_session
    if session._connector is not None:
AttributeError: 'RetryClient' object has no attribute '_connector'
Exception ignored in: <finalize object at 0x110296e60; dead>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/weakref.py", line 580, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/opt/homebrew/lib/python3.9/site-packages/fsspec/implementations/http.py", line 126, in close_session
    if session._connector is not None:
AttributeError: 'RetryClient' object has no attribute '_connector'
Exception ignored in: <finalize object at 0x110296e80; dead>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/weakref.py", line 580, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/opt/homebrew/lib/python3.9/site-packages/fsspec/implementations/http.py", line 126, in close_session
    if session._connector is not None:
AttributeError: 'RetryClient' object has no attribute '_connector'
Exception ignored in: <finalize object at 0x110296ea0; dead>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/weakref.py", line 580, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/opt/homebrew/lib/python3.9/site-packages/fsspec/implementations/http.py", line 126, in close_session
    if session._connector is not None:
AttributeError: 'RetryClient' object has no attribute '_connector'
Exception ignored in: <finalize object at 0x110296ec0; dead>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/weakref.py", line 580, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/opt/homebrew/lib/python3.9/site-packages/fsspec/implementations/http.py", line 126, in close_session
    if session._connector is not None:
AttributeError: 'RetryClient' object has no attribute '_connector'
Exception ignored in: <finalize object at 0x110296ee0; dead>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/weakref.py", line 580, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/opt/homebrew/lib/python3.9/site-packages/fsspec/implementations/http.py", line 126, in close_session
    if session._connector is not None:
AttributeError: 'RetryClient' object has no attribute '_connector'
Exception ignored in: <finalize object at 0x110296f00; dead>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/weakref.py", line 580, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/opt/homebrew/lib/python3.9/site-packages/fsspec/implementations/http.py", line 126, in close_session
    if session._connector is not None:
AttributeError: 'RetryClient' object has no attribute '_connector'
Exception ignored in: <finalize object at 0x110296f20; dead>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/weakref.py", line 580, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/opt/homebrew/lib/python3.9/site-packages/fsspec/implementations/http.py", line 126, in close_session
    if session._connector is not None:
AttributeError: 'RetryClient' object has no attribute '_connector'
Exception ignored in: <finalize object at 0x110296f40; dead>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/weakref.py", line 580, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/opt/homebrew/lib/python3.9/site-packages/fsspec/implementations/http.py", line 126, in close_session
    if session._connector is not None:
AttributeError: 'RetryClient' object has no attribute '_connector'
Exception ignored in: <finalize object at 0x110296f60; dead>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/weakref.py", line 580, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/opt/homebrew/lib/python3.9/site-packages/fsspec/implementations/http.py", line 126, in close_session
    if session._connector is not None:
AttributeError: 'RetryClient' object has no attribute '_connector'
Exception ignored in: <finalize object at 0x110296320; dead>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/weakref.py", line 580, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/opt/homebrew/lib/python3.9/site-packages/fsspec/implementations/http.py", line 126, in close_session
    if session._connector is not None:
AttributeError: 'RetryClient' object has no attribute '_connector'
Exception ignored in: <finalize object at 0x1102962e0; dead>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/weakref.py", line 580, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/opt/homebrew/lib/python3.9/site-packages/fsspec/implementations/http.py", line 126, in close_session
    if session._connector is not None:
AttributeError: 'RetryClient' object has no attribute '_connector'
Exception ignored in: <finalize object at 0x1102965c0; dead>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/weakref.py", line 580, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/opt/homebrew/lib/python3.9/site-packages/fsspec/implementations/http.py", line 126, in close_session
    if session._connector is not None:
AttributeError: 'RetryClient' object has no attribute '_connector'
Exception ignored in: <finalize object at 0x110296600; dead>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/weakref.py", line 580, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/opt/homebrew/lib/python3.9/site-packages/fsspec/implementations/http.py", line 126, in close_session
    if session._connector is not None:
AttributeError: 'RetryClient' object has no attribute '_connector'
Exception ignored in: <finalize object at 0x11091b7e0; dead>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/weakref.py", line 580, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/opt/homebrew/lib/python3.9/site-packages/fsspec/implementations/http.py", line 126, in close_session
    if session._connector is not None:
AttributeError: 'RetryClient' object has no attribute '_connector'
ERROR: unexpected error - Server disconnected

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!

@dberenbaum dberenbaum added bug Did we break something? p0-critical labels Sep 29, 2021
@efiop efiop added the fs: s3 Related to the S3 filesystem label Sep 29, 2021
@skshetry skshetry added fs: http Related to the HTTP filesystem and removed fs: s3 Related to the S3 filesystem labels Sep 30, 2021
@efiop
Copy link
Contributor

efiop commented Oct 1, 2021

Hm, the original report is coming from osxpkg 2.7.4, but I can't reproduce this on master. @dberenbaum Could you show your dvc doctor please, just curious?

@isidentical
Copy link
Contributor

It seems like this was due to asyncio-retry not providing a private method that fsspec sometimes uses (this is why sometimes it is not reproducable) (when there is a timeout). I already sent a PR against fsspec, and will be resolved with the new release (hopefully today) fsspec/filesystem_spec#779

@efiop
Copy link
Contributor

efiop commented Oct 1, 2021

@isidentical Thank you! 🙏

@shcheklein
Copy link
Member

Thanks @isidentical for taking of this promptly 🙏 Good stuff.

@efiop
Copy link
Contributor

efiop commented Oct 2, 2021

Fixed by #6727

@efiop efiop closed this as completed Oct 2, 2021
@daavoo daavoo added the A: get label Feb 22, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@isidentical isidentical

Labels
bug Did we break something? fs: http Related to the HTTP filesystem
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@dberenbaum @shcheklein @jbencook @efiop @daavoo @skshetry @isidentical