Breaks with requests 2.32.0: Not supported URL scheme http+docker

[rra]

With requests 2.32.0 (released about an hour ago as I write this), the docker library as called by tox-docker fails with the following exception:

Traceback (most recent call last):
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/requests/adapters.py", line 532, in send
    conn = self._get_connection(request, verify, proxies=proxies, cert=cert)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/requests/adapters.py", line 400, in _get_connection
    conn = self.poolmanager.connection_from_host(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/urllib3/poolmanager.py", line 304, in connection_from_host
    return self.connection_from_context(request_context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/urllib3/poolmanager.py", line 326, in connection_from_context
    raise URLSchemeUnknown(scheme)
urllib3.exceptions.URLSchemeUnknown: Not supported URL scheme http+docker

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/docker/api/client.py", line 214, in _retrieve_server_version
    return self.version(api_version=False)["ApiVersion"]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/docker/api/daemon.py", line 181, in version
    return self._result(self._get(url), json=True)
                        ^^^^^^^^^^^^^^
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/docker/utils/decorators.py", line 46, in inner
    return f(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/docker/api/client.py", line 237, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/requests/sessions.py", line 602, in get
    return self.request("GET", url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/requests/adapters.py", line 534, in send
    raise InvalidURL(e, request=request)
requests.exceptions.InvalidURL: Not supported URL scheme http+docker

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/tox/session/cmd/run/single.py", line 48, in _evaluate
    code, outcomes = run_commands(tox_env, no_test)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/tox/session/cmd/run/single.py", line 79, in run_commands
    MANAGER.tox_before_run_commands(tox_env)
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/tox/plugin/manager.py", line 88, in tox_before_run_commands
    self.manager.hook.tox_before_run_commands(tox_env=tox_env)
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/pluggy/_hooks.py", line 513, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/pluggy/_callers.py", line 139, in _multicall
    raise exception.with_traceback(exception.__traceback__)
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/pluggy/_callers.py", line 103, in _multicall
    res = hook_impl.function(*args)
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/tox_docker/tox4/plugin.py", line 73, in tox_before_run_commands
    docker_build_or_pull(container_config, log)
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/tox_docker/plugin.py", line 57, in docker_build_or_pull
    docker_pull(container_config, log)
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/tox_docker/plugin.py", line 65, in docker_pull
    docker = docker_module.from_env(version="auto")
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/docker/client.py", line 96, in from_env
    return cls(
           ^^^^
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/docker/client.py", line 45, in __init__
    self.api = APIClient(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/docker/api/client.py", line 197, in __init__
    self._version = self._retrieve_server_version()
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/eagle/dvl/venvs/gafaelfawr/lib/python3.12/site-packages/docker/api/client.py", line 221, in _retrieve_server_version
    raise DockerException(
docker.errors.DockerException: Error while fetching server API version: Not supported URL scheme http+docker

Reverting to requests 2.31.0 without any other changes fixes the problem.

[zhiyue]

same problem

In [1]: import docker
   ...: client = docker.from_env()
---------------------------------------------------------------------------
URLSchemeUnknown                          Traceback (most recent call last)
File /usr/local/lib/python3.8/dist-packages/requests/adapters.py:532, in HTTPAdapter.send(self, request, stream, timeout, verify, cert, proxies)
    531 try:
--> 532     conn = self._get_connection(request, verify, proxies=proxies, cert=cert)
    533 except LocationValueError as e:

File /usr/local/lib/python3.8/dist-packages/requests/adapters.py:400, in HTTPAdapter._get_connection(self, request, verify, proxies, cert)
    398 else:
    399     # Only scheme should be lower case
--> 400     conn = self.poolmanager.connection_from_host(
    401         **host_params, pool_kwargs=pool_kwargs
    402     )
    404 return conn

File /usr/local/lib/python3.8/dist-packages/urllib3/poolmanager.py:304, in PoolManager.connection_from_host(self, host, port, scheme, pool_kwargs)
    302 request_context["host"] = host
--> 304 return self.connection_from_context(request_context)

File /usr/local/lib/python3.8/dist-packages/urllib3/poolmanager.py:326, in PoolManager.connection_from_context(self, request_context)
    325 if not pool_key_constructor:
--> 326     raise URLSchemeUnknown(scheme)
    327 pool_key = pool_key_constructor(request_context)

URLSchemeUnknown: Not supported URL scheme http+docker

During handling of the above exception, another exception occurred:

InvalidURL                                Traceback (most recent call last)
File /usr/local/lib/python3.8/dist-packages/docker/api/client.py:213, in APIClient._retrieve_server_version(self)
    212 try:
--> 213     return self.version(api_version=False)["ApiVersion"]
    214 except KeyError as ke:

File /usr/local/lib/python3.8/dist-packages/docker/api/daemon.py:181, in DaemonApiMixin.version(self, api_version)
    180 url = self._url("/version", versioned_api=api_version)
--> 181 return self._result(self._get(url), json=True)

File /usr/local/lib/python3.8/dist-packages/docker/utils/decorators.py:44, in update_headers.<locals>.inner(self, *args, **kwargs)
     43         kwargs['headers'].update(self._general_configs['HttpHeaders'])
---> 44 return f(self, *args, **kwargs)

File /usr/local/lib/python3.8/dist-packages/docker/api/client.py:236, in APIClient._get(self, url, **kwargs)
    234 @update_headers
    235 def _get(self, url, **kwargs):
--> 236     return self.get(url, **self._set_request_timeout(kwargs))

File /usr/local/lib/python3.8/dist-packages/requests/sessions.py:602, in Session.get(self, url, **kwargs)
    601 kwargs.setdefault("allow_redirects", True)
--> 602 return self.request("GET", url, **kwargs)

File /usr/local/lib/python3.8/dist-packages/requests/sessions.py:589, in Session.request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)
    588 send_kwargs.update(settings)
--> 589 resp = self.send(prep, **send_kwargs)
    591 return resp

File /usr/local/lib/python3.8/dist-packages/requests/sessions.py:703, in Session.send(self, request, **kwargs)
    702 # Send the request
--> 703 r = adapter.send(request, **kwargs)
    705 # Total elapsed time of the request (approximately)

File /usr/local/lib/python3.8/dist-packages/requests/adapters.py:534, in HTTPAdapter.send(self, request, stream, timeout, verify, cert, proxies)
    533 except LocationValueError as e:
--> 534     raise InvalidURL(e, request=request)
    536 self.cert_verify(conn, request.url, verify, cert)

InvalidURL: Not supported URL scheme http+docker

The above exception was the direct cause of the following exception:

DockerException                           Traceback (most recent call last)
Cell In[1], line 2
      1 import docker
----> 2 client = docker.from_env()

File /usr/local/lib/python3.8/dist-packages/docker/client.py:94, in DockerClient.from_env(cls, **kwargs)
     92 version = kwargs.pop('version', None)
     93 use_ssh_client = kwargs.pop('use_ssh_client', False)
---> 94 return cls(
     95     timeout=timeout,
     96     max_pool_size=max_pool_size,
     97     version=version,
     98     use_ssh_client=use_ssh_client,
     99     **kwargs_from_env(**kwargs)
    100 )

File /usr/local/lib/python3.8/dist-packages/docker/client.py:45, in DockerClient.__init__(self, *args, **kwargs)
     44 def __init__(self, *args, **kwargs):
---> 45     self.api = APIClient(*args, **kwargs)

File /usr/local/lib/python3.8/dist-packages/docker/api/client.py:197, in APIClient.__init__(self, base_url, version, timeout, tls, user_agent, num_pools, credstore_env, use_ssh_client, max_pool_size)
    192 # version detection needs to be after unix adapter mounting
    193 if version is None or (isinstance(
    194                         version,
    195                         str
    196                         ) and version.lower() == 'auto'):
--> 197     self._version = self._retrieve_server_version()
    198 else:
    199     self._version = version

File /usr/local/lib/python3.8/dist-packages/docker/api/client.py:220, in APIClient._retrieve_server_version(self)
    215     raise DockerException(
    216         'Invalid response from docker daemon: key "ApiVersion"'
    217         ' is missing.'
    218     ) from ke
    219 except Exception as e:
--> 220     raise DockerException(
    221         f'Error while fetching server API version: {e}'
    222     ) from e

DockerException: Error while fetching server API version: Not supported URL scheme http+docker

pip list
Package                Version             
---------------------- --------------------
asttokens              2.4.1               
attrs                  19.3.0              
Automat                0.8.0               
backcall               0.2.0               
blinker                1.4                 
certifi                2019.11.28          
chardet                3.0.4               
charset-normalizer     3.3.2               
Click                  7.0                 
cloud-init             22.2                
colorama               0.4.3               
command-not-found      0.3                 
configobj              5.0.6               
constantly             15.1.0              
cryptography           2.8                 
dbus-python            1.2.16              
decorator              5.1.1               
distro                 1.4.0               
distro-info            0.23ubuntu1         
docker                 7.0.0               
entrypoints            0.3                 
executing              2.0.1               
httplib2               0.14.0              
hyperlink              19.0.0              
idna                   2.8                 
importlib-metadata     1.5.0               
incremental            16.10.1             
ipython                8.12.3              
jedi                   0.19.1              
Jinja2                 2.10.1              
jsonpatch              1.22                
jsonpointer            2.0                 
jsonschema             3.2.0               
keyring                18.0.1              
language-selector      0.1                 
launchpadlib           1.10.13             
lazr.restfulclient     0.14.2              
lazr.uri               1.0.3               
MarkupSafe             1.1.0               
matplotlib-inline      0.1.7               
more-itertools         4.2.0               
netifaces              0.10.4              
oauthlib               3.1.0               
packaging              24.0                
parso                  0.8.4               
pexpect                4.6.0               
pickleshare            0.7.5               
pip                    20.0.2              
prompt-toolkit         3.0.43              
pure-eval              0.2.2               
pyasn1                 0.4.2               
pyasn1-modules         0.2.1               
pygments               2.18.0              
PyGObject              3.36.0              
PyHamcrest             1.9.0               
PyJWT                  1.7.1               
pymacaroons            0.13.0              
PyNaCl                 1.3.0               
pyOpenSSL              19.0.0              
pyrsistent             0.15.5              
pyserial               3.4                 
python-apt             2.0.0+ubuntu0.20.4.7
python-debian          0.1.36ubuntu1       
PyYAML                 5.3.1               
requests               2.32.0              
requests-unixsocket    0.2.0               
SecretStorage          2.3.1               
service-identity       18.1.0              
setuptools             45.2.0              
simplejson             3.16.0              
six                    1.14.0              
sos                    4.3                 
ssh-import-id          5.10                
stack-data             0.6.3               
systemd-python         234                 
traitlets              5.14.3              
Twisted                18.9.0              
typing-extensions      4.11.0              
ubuntu-advantage-tools 27.9                
ufw                    0.36                
unattended-upgrades    0.1                 
urllib3                2.2.1               
wadllib                1.3.3               
wcwidth                0.2.13              
wheel                  0.34.2              
zipp                   1.0.0               
zope.interface         4.7.1 

[danb27]

same!

[felixfontein]

I'm currently debugging this a bit (ansible-collections/community.docker#860), the issue is commit psf/requests@c0813a2 in requests.

Ok, this isn't very surprising, since that commit makes send() call _get_connection() instead of get_connection(). Now the Docker SDK for Python code overwrites get_connection(), but of course doesn't magically overwrite _get_connection() as well...

See for example https://github.com/docker/docker-py/blob/main/docker/transport/unixconn.py#L66.

[felixfontein]

A simple fix is adding

    def _get_connection(self, request, *args, proxies=None, **kwargs):
        return self.get_connection(request.url, proxies)

at the bottom of https://github.com/docker/docker-py/blob/main/docker/transport/basehttpadapter.py.

Not sure how sustainable that is, but it works for now :)

[felixfontein]

I created #3257 with that fix.

[cardosofede]

same issue here, the fix from @felixfontein works for me!
for now I just replaced the dependency docker for git+https://github.com/felixfontein/docker-py@requests-hotfix

[geerlingguy]

Ditto here :)

Just saw lots of builds start failing, glad there's already a fix... and it seems the CI build is failing already, but unrelated to the fix above.

[AlexiaChen]

haha Seems a lot of CI fails cause this issue. me tooooooo.

[bparthas21]

Our team wasted all day today trying to triage and put a fix to this problem. After this incident, is there any plan to revisit this unpinned dependency specification:

docker-py/pyproject.toml

Line 33 in b6464db

"requests >= 2.26.0",

[felixfontein]

Pinning dependencies, in particular very general dependencies such as requests, generally causes a lot more problems than it solves. The best place where to pin dependencies is at the end of the chain, i.e. where the venv is created that contains all the Python libraries and programs that you need.

[bparthas21]

Pinning dependencies, in particular very general dependencies such as requests, generally causes a lot more problems than it solves. The best place where to pin dependencies is at the end of the chain, i.e. where the venv is created that contains all the Python libraries and programs that you need.

In our process, we had pinned to docker==6.0.0 but did not realize we had to pin requests also.

[felixfontein]

If you only pin direct dependencies, you're usually missing A LOT of indirect dependencies that can cause all kind of problems - such as in this case.

You might want to use pip freeze (https://pip.pypa.io/en/stable/cli/pip_freeze/) or pip-tools (https://pypi.org/project/pip-tools/) to pin all dependencies. Especially with pip-tools you can create a workflow that pins everything, and you have a CI workflow that regularly tries to upgrades dependencies and checks whether the result passes CI. Also check out Renovate (https://github.com/renovatebot/renovate) and Dependabot, which help a lot with that.

[meijiesky]

I am using requests 2.32.3 and have the same issue! Is it fixed by now and is it released?

[slonopotamus]

The fix is included in docker-py 7.1.0.

[someqst]

I faced the same problem.
requests version is 2.28.2. Ubuntu 22.04. Docker 27.1.2
Any solutions?
Thank you

[zhangyichen534]

我把docker版本升级到最新解决了这个问题

translated:

I upgraded the docker version to the latest version to solve this problem.

Version 7.1.0

[thaJeztah]

I'm locking the conversation on this ticket, as this issue should be resolved by #3257, which is part of the 7.1.0 release; https://github.com/docker/docker-py/releases/tag/7.1.0