SSLCertVerificationError with pipenv install, but not with pip in same docker image #5665
Comments
|
Reviewing your description, I believe I know why it's failing. We (@matteius and myself) added a few additional features related to pip indexes, but you seemed to have found the one hole in the logic. Heh. Easy to fix though.
python -m pip install pypkg --trusted-host=pypi.org --trusted-host=pypi.python.orgthat will work also.
I plan to consolidate the |
|
Thanks for the reply. I tried your suggestions, but unfortunately, I still am getting the same errors. Output of Based on your suggestion, tried to install directly a package instead of using requirements.txt with the following command (where I also split the trusted host information and even provided an explicit proxy): This results in the following output: (base) root@d941cd147a69:/foo# pipenv -v install --extra-pip-args "--trusted-host=pypi.org --trusted-host=files.pythonhosted.org --trusted-host=pypi.python.org --proxy=$https_proxy" six
Installing six...
Resolving six...
Installing...
Installing package: six
⠋ Installing six...Writing supplied requirement line to temporary file: 'six'
Installing 'six'
$ /root/.local/share/virtualenvs/foo-YJbvrhmB/bin/python /opt/conda/lib/python3.10/site-packages/pipenv/patched/pip/__pip-runner__.py install --no-input --verbose --upgrade --exists-action=i --trusted-host=pypi.org --trusted-host=files.pythonhosted.org --trusted-host=pypi.python.org --proxy=http://gateway.docker.internal:7070 -r /tmp/pipenv-uqxjtngs-requirements/pipenv-3dmw3xf7-requirement.txt -i https://pypi.org/simple --trusted-host pypi.org
Using source directory: '/root/.local/share/virtualenvs/foo-YJbvrhmB/src'
Adding six to Pipfile's [packages] ...
✔ Installation Succeeded
Pipfile.lock not found, creating...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
Reporter.starting()
INFO:pipenv.patched.pip._internal.resolution.resolvelib.reporter:Reporter.starting()
Reporter.adding_requirement(SpecifierRequirement('six'), None)
INFO:pipenv.patched.pip._internal.resolution.resolvelib.reporter:Reporter.adding_requirement(SpecifierRequirement('six'), None)
Reporter.starting_round(0)
INFO:pipenv.patched.pip._internal.resolution.resolvelib.reporter:Reporter.starting_round(0)
Reporter.pinning(LinkCandidate('https://files.pythonhosted.org/packages/d9/5a/e7c31adbe875f2abbb91bd84cf2dc52d792b5a01506781dbcf25c91daf11/six-1.16.0-py2.py3-none-any.whl (from
https://pypi.org/simple/six/) (requires-python:>=2.7, !=3.0.*, !=3.1.*, !=3.2.*)'))
INFO:pipenv.patched.pip._internal.resolution.resolvelib.reporter:Reporter.pinning(LinkCandidate('https://files.pythonhosted.org/packages/d9/5a/e7c31adbe875f2abbb91bd84cf2dc52d792b5a01506781dbcf25c91da
f11/six-1.16.0-py2.py3-none-any.whl (from https://pypi.org/simple/six/) (requires-python:>=2.7, !=3.0.*, !=3.1.*, !=3.2.*)'))
Reporter.ending_round(0, state)
INFO:pipenv.patched.pip._internal.resolution.resolvelib.reporter:Reporter.ending_round(0, state)
Reporter.starting_round(1)
INFO:pipenv.patched.pip._internal.resolution.resolvelib.reporter:Reporter.starting_round(1)
Reporter.ending(State(mapping=OrderedDict([('six',
LinkCandidate('https://files.pythonhosted.org/packages/d9/5a/e7c31adbe875f2abbb91bd84cf2dc52d792b5a01506781dbcf25c91daf11/six-1.16.0-py2.py3-none-any.whl (from https://pypi.org/simple/six/)
(requires-python:>=2.7, !=3.0.*, !=3.1.*, !=3.2.*)'))]), criteria={'six': Criterion((SpecifierRequirement('six'), via=None))}, backtrack_causes=[]))
INFO:pipenv.patched.pip._internal.resolution.resolvelib.reporter:Reporter.ending(State(mapping=OrderedDict([('six',
LinkCandidate('https://files.pythonhosted.org/packages/d9/5a/e7c31adbe875f2abbb91bd84cf2dc52d792b5a01506781dbcf25c91daf11/six-1.16.0-py2.py3-none-any.whl (from https://pypi.org/simple/six/)
(requires-python:>=2.7, !=3.0.*, !=3.1.*, !=3.2.*)'))]), criteria={'six': Criterion((SpecifierRequirement('six'), via=None))}, backtrack_causes=[]))
Traceback (most recent call last):
File "/opt/conda/lib/python3.10/site-packages/pipenv/patched/pip/_vendor/urllib3/connectionpool.py", line 700, in urlopen
self._prepare_proxy(conn)
File "/opt/conda/lib/python3.10/site-packages/pipenv/patched/pip/_vendor/urllib3/connectionpool.py", line 996, in _prepare_proxy
conn.connect()
File "/opt/conda/lib/python3.10/site-packages/pipenv/patched/pip/_vendor/urllib3/connection.py", line 419, in connect
self.sock = ssl_wrap_socket(
File "/opt/conda/lib/python3.10/site-packages/pipenv/patched/pip/_vendor/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
ssl_sock = _ssl_wrap_socket_impl(
File "/opt/conda/lib/python3.10/site-packages/pipenv/patched/pip/_vendor/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
File "/opt/conda/lib/python3.10/ssl.py", line 513, in wrap_socket
return self.sslsocket_class._create(
File "/opt/conda/lib/python3.10/ssl.py", line 1071, in _create
self.do_handshake()
File "/opt/conda/lib/python3.10/ssl.py", line 1342, in do_handshake
self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/conda/lib/python3.10/site-packages/pipenv/patched/pip/_vendor/requests/adapters.py", line 489, in send
resp = conn.urlopen(
File "/opt/conda/lib/python3.10/site-packages/pipenv/patched/pip/_vendor/urllib3/connectionpool.py", line 787, in urlopen
retries = retries.increment(
File "/opt/conda/lib/python3.10/site-packages/pipenv/patched/pip/_vendor/urllib3/util/retry.py", line 592, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
pipenv.patched.pip._vendor.urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /pypi/six/json (Caused by
SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)')))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/conda/lib/python3.10/site-packages/pipenv/resolver.py", line 845, in <module>
main()
File "/opt/conda/lib/python3.10/site-packages/pipenv/resolver.py", line 831, in main
_main(
File "/opt/conda/lib/python3.10/site-packages/pipenv/resolver.py", line 811, in _main
resolve_packages(
File "/opt/conda/lib/python3.10/site-packages/pipenv/resolver.py", line 759, in resolve_packages
results, resolver = resolve(
File "/opt/conda/lib/python3.10/site-packages/pipenv/resolver.py", line 738, in resolve
return resolve_deps(
File "/opt/conda/lib/python3.10/site-packages/pipenv/utils/resolver.py", line 1097, in resolve_deps
results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
File "/opt/conda/lib/python3.10/site-packages/pipenv/utils/resolver.py", line 897, in actually_resolve_deps
hashes = resolver.resolve_hashes()
File "/opt/conda/lib/python3.10/site-packages/pipenv/utils/resolver.py", line 798, in resolve_hashes
self.hashes = self.collect_hashes(ireq)
File "/opt/conda/lib/python3.10/site-packages/pipenv/utils/resolver.py", line 778, in collect_hashes
hashes = self._get_hashes_from_pypi(ireq)
File "/opt/conda/lib/python3.10/site-packages/pipenv/utils/resolver.py", line 738, in _get_hashes_from_pypi
r = session.get(pkg_url, timeout=10)
File "/opt/conda/lib/python3.10/site-packages/pipenv/patched/pip/_vendor/requests/sessions.py", line 600, in get
return self.request("GET", url, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/pipenv/patched/pip/_vendor/requests/sessions.py", line 587, in request
resp = self.send(prep, **send_kwargs)
File "/opt/conda/lib/python3.10/site-packages/pipenv/patched/pip/_vendor/requests/sessions.py", line 701, in send
r = adapter.send(request, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/pipenv/patched/pip/_vendor/requests/adapters.py", line 563, in send
raise SSLError(e, request=request)
pipenv.patched.pip._vendor.requests.exceptions.SSLError: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /pypi/six/json (Caused by SSLError(SSLCertVerificationError(1,
'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)')))
✘ Locking Failed!
⠹ Locking...Also wanted to check if there is a difference between using an = sign in the trusted-host arguments (e.g. provide it like If I run the command without the = signs, so the following: (base) root@56ff58188d78:/# pipenv install --extra-pip-args="--trusted-host pypi.org --trusted-host files.pythonhosted.org --trusted-host pypi.python.org" six
Installing six...
Resolving six...
Installing...
⠙ Installing six...[31m[1mError: [0m An error occurred while installing [32msix[0m!
Error text:
[36mERROR: Could not find a version that satisfies the requirement files.pythonhosted.org (from versions: none)
ERROR: No matching distribution found for files.pythonhosted.org
[0m
✘ Installation FailedSeems here it tries to install the 'package' files.pythonhosted.org. Look at the documentation of pip, it is actually shown that you should provide the argument as In all of the runs btw, I start out with the The weird thing to me is that when using pip directly, all the above things work, only using pipenv it is not working. Are there any other things I can do to bypass this ssl_verify check? |
|
To answer the latter inquiry first, I have an experiment to try. I noticed the version of pipenv run pip install sixP.S. I noticed this difference in the output above: |
|
Not 100% sure what you mean with the part What I did is the following (in an empty directory): Execute This creates the following output: (base) root@0040736c8498:/foo# pipenv --python=3.10.8
Creating a virtualenv for this project...
Pipfile: /foo/Pipfile
Using /opt/conda/bin/python3.1 (3.10.8) to create virtualenv...
⠴ Creating virtual environment...created virtual environment CPython3.10.8.final.0-64 in 4080ms
creator CPython3Posix(dest=/root/.local/share/virtualenvs/foo-YJbvrhmB, clear=False, no_vcs_ignore=False, global=False)
seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/root/.local/share/virtualenv)
added seed packages: pip==23.1, setuptools==67.6.1, wheel==0.40.0
activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
✔ Successfully created virtual environment!
Virtualenv location: /root/.local/share/virtualenvs/foo-YJbvrhmB
requirements.txt found in /foo instead of Pipfile! Converting...
✔ Success!
Warning: Your Pipfile now contains pinned versions, if your requirements.txt did.
We recommend updating your Pipfile to specify the "*" version, instead.After that, with the pipfile created, I ran the command you provided: (base) root@0040736c8498:/foo# pipenv run pip install six
Collecting six
Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Installing collected packages: six
Successfully installed six-1.16.0Regarding your PS question, yes, I am behind a proxy. Due to some specifics, not always possible to copy/include the PEM file for pip / pipenv to use, that is why I want to disable ssl_verify for this (it is a conscious decision) |
Exactly as you did. 🙂 Can you now rerun |
|
Also, after that, can you run I just want to verify if this SSL fails on just locking or both locking and installing. If it does what I think it will do, I have a vague recollection of seeing something like this myself and have to find my notes to see how I addressed it. |
|
Digging through the code, it appears that during locking, a |
|
Your last message makes sense, because I can see in the proxy log that there are successful calls to the pypi servers being done and things appear to be downloaded, but after that I get the error about the SSL. I completely understand that when we do not have a valid SSL context, we are vulnerable to a MitM attack. However, this is only for building things locally (especially inside docker containers locally) and not for building things inside our CI/CD pipeline. Also without pipenv, but just using pip I would be vulnerable to this if I make use of trusted-host. For sure the default behavior should always be to require a valid SSL context. However, would like to at least have the option to overrule it (potentially with all kinds of big warning messages being shown). Thinking of the particular use-case, it might also be an option to use the |
|
It’s not really my call, but also (and I need to verify this later when I have more time), I don’t believe you need a client cert when connecting to the PyPI public API. I believe the |
|
The problem is the bit 'You only need to have bundle installed'. When building Docker images locally, there is no generic way for me to ensure this bundle is available. The nice advantage of the disabling of SSL is that I don't have to provide any other information. However, as mentioned, it might be that I can just skip the whole lock-file generation anyways when building Docker images locally, since these images are not going to be published and used anywhere else, except on the local computer. However, in general I think it still would be nice if for whatever reason it is needed, you are at least able to overrule this SSL certificate checking for retrieving the hashes (especially because there is the possibility already to set config option ssl_verify = false for specific sources in the Pipfile). |
|
It seems strange to me that the base docker image doesn’t contain the CA cents bundle or there wouldn’t be one available. shrugs Nevertheless, if this bug is solved; I suggest submitting a feature request for this specific option so it doesn’t get lost in the shuffle, and those who have decision-making abilities can review it. |
|
Side note: after some quick searches, I discovered that conda can install the ca certificates: conda install -c anaconda ca-certificates |
matteius
added
the
Type: Regression
|
@gdiepen could this be rechecked with |
|
@matteius Just gave it a try and with this version, if I ensure the Pipfile contains AFAIK there is no way to instruct pipenv to generate this line automatically when no Pipfile exists yet and you use |
Not currently, but always open to improvement PRs! |
|
Is this still an issue on |
|
still happened in 2023.11.15 |
|
same problem pipenv, version 2023.12.1 |
|
Im still setting this issue on The pipenv install --extra-pip-args="--trusted-host foo.example.com --cert /etc/ca-certs/root_certs.pem --client-cert /etc/ca-certs/client.pem"Also setting |
|
This looks to be because we are not using the The The |
|
@dennisme Feel free to open a PR to get this addressed sooner, and if you do, thank you! |
matteius
added
the
Contributor Candidate
|
Yes! Working on it now. Just dropping as much info as I found. |
|
Ok I think I see whats up @matteius. Took a second look at my local setup with With One improvement would be to support the Another option might be to support pip truststore. I hesitate to suggest any pipenv specific fixes or behavior as it might sway too far outside of python tooling (which id rely on your opinion here as I am less in tune to this). But it seems like matching the pip behavior is a fair middle ground? |
Issue description
I am behind a proxy and would like to use pipenv to build my docker images locally. Within my existing setup (using conda and pip) I have a build-argument in my Dockerfile indicating whether I want to disable the SSL certificate checking for conda and pip. This way, when building locally I choose to disable ssl verifications and while building on the CI/CD pipeline I don't provide this build-arg because the CI/CD workers have direct internet.
Using this approach, I can use the same Dockerfile both locally and on the CI/CD system by modifying just the build-arg and I don't have to try and conditionally include PEM certificate chains
I have tried things like:
--extra-pip-argslikepipenv -vvv install --extra-pip-args '--trusted-host "pypi.org files.pythonhosted.org pypi.python.org"' -r requirements.txtPIP_TRUSTED_HOSTenvironment variableI can't seem to be able to instruct pipenv to NOT check the certificates. From the documentation it is not clear how this should be possible
Expected result
I expect pipenv to be able to install packages without checking the SSL certificates if I instruct it to do so. I do know that this negates the purpose of SSL certificates, but this way I am able to not have to manually copy additional items (pem files) into my docker image during local builds
Actual result
Steps to replicate
I am running all the commands below in a clean ubuntu base image that has miniconda3 installed.
requirements.txt file
Pipfile
After running the command
pipenv -v install -r requirements.txt, automatically a Pipfile is generated with the following contents:Even after I change the verify_ssl in this file to false and run the pipenv install again (or pipenv lock), I still get the same CERTIFICATE_VERIFY_FAILED error.
Installing with pip itself works
When running the following command:
the package can be installed without a problem (i.e. no checking of the certificates).
pip config
$ pipenv --support
Pipenv version:
'2023.4.20'Pipenv location:
'/opt/conda/lib/python3.10/site-packages/pipenv'Python location:
'/opt/conda/bin/python'OS Name:
'posix'User pip version:
'23.1'user Python installations found:
3.10.8:/opt/conda/bin/python33.10.8:/opt/conda/bin/python3.10.8:/opt/conda/bin/python3.1PEP 508 Information:
System environment variables:
CONDA_EXE_CE_MHOSTNAMEPWDCONDA_PREFIXHOMELANGLS_COLORSCONDA_PROMPT_MODIFIERhttps_proxyTERM_CE_CONDACONDA_SHLVLSHLVLHTTPS_PROXYHTTP_PROXYhttp_proxyCONDA_PYTHON_EXECONDA_DEFAULT_ENVLC_ALLPATHDEBIAN_FRONTENDOLDPWD_PIP_DISABLE_PIP_VERSION_CHECKPIP_PYTHON_PATHPYTHONDONTWRITEBYTECODEPYTHONFINDER_IGNORE_UNSUPPORTEDPipenv–specific environment variables:
Debug–specific environment variables:
PATH:/opt/conda/bin:/opt/conda/condabin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/binLANG:C.UTF-8PWD:/fooContents of
Pipfile('/foo/Pipfile'):The text was updated successfully, but these errors were encountered: