Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow root CA bundle to be configured #1012

Open
1 of 2 tasks
jobec opened this issue Apr 3, 2019 · 43 comments
Open
1 of 2 tasks

Allow root CA bundle to be configured #1012

jobec opened this issue Apr 3, 2019 · 43 comments
Labels
kind/feature Feature requests/implementations

Comments

@jobec
Copy link

jobec commented Apr 3, 2019

Feature Request

Allow and alternate root CA bundle to be specified from CLI and in the global/project settings file. Similar to how pip does it. This is crucial in environments where direct access to pypi is blocked and when using internal pypi mirrors with certificates signed by an internal root CA. Or with SSL traffic inspecting firewalls.

https://pip.pypa.io/en/stable/reference/pip/?highlight=proxy#cmdoption-cert

Having to set an environment variable like REQUESTS_CA_BUNDLE for an underlying package is not user-friendly and expects the user to know what poetry does under the hood.

@MichalMazurek
Copy link

Hi,
It would be also nice to disable certificate checking, I'm having a lot of issues with a self-signed certificate for private PyPI repo. I can fix that, but first, it would be great to know if this will be welcomed into the project. This can be done together with adding support for the request above.

@floer32
Copy link

floer32 commented Apr 10, 2019

can you confirm/deny whether REQUESTS_CA_BUNDLE actually works?

Having to set an environment variable like REQUESTS_CA_BUNDLE for an underlying package is not user-friendly and expects the user to know what poetry does under the hood.

Arguably that's just a documentation problem.

While encapsulation can be good, requests is extremely common, so much so that most people treat it as if it's basically part of the Python stdlib. REQUESTS_CA_BUNDLE is barely more 'under the hood' of a parameter, than PYTHONUNBUFFERED or other python-official features. So really as long as it is documented and guaranteed that requests is used, that could be enough for immediate future.


The bigger question, I think, is whether/how-much to expose Pip directly through Poetry.

i.e. In pytest there is the env var, PYTEST_ADDOPTS and it appends flags to any pytest invocation. there could be POETRY_PIP_ADDOPTS etc. maybe. Or instead of an env var it could be something in config; with pytest as example again, it has pytest_addopts = ... supported from its config file (.ini in that case, similar to .toml). https://docs.pytest.org/en/latest/customize.html#adding-default-options Of course it won't be exactly like pytest, it's just a worthwhile example to consider.

this under-the-hood-config question is kinda relevant to #558 (though there it is about pip wheel and --find-links configs, rather than something to do with requests or CA bundles)


maybe this relates to what #697 is getting at too... just generally how much to connect Poetry's high-level view to the lower-level nuts and bolts of the tools involved. vs. how much to have known-good ways to "break out" of the typical paths to customize for your needs... while remaining compatible with Poetry

@jobec
Copy link
Author

jobec commented Apr 14, 2019

Yes, using REQUESTS_CA_BUNDLE works.

I don’t think it’s a documentation issue though. Having to set an environment variable while there is the “poetry config” command is counter intuitive.

@drunkwcodes
Copy link

Maybe adding requests library in system requirements will make sense.

Rather than include requests config in poetry configuration, having a script to set env vars in the project sounds better.

@jobec
Copy link
Author

jobec commented Apr 14, 2019

From a UX point of view this isn't true. People use poetry. They shouldn't care about what's underneath.

A lot of tools use this approach (configuration options for the tool, not it's libraries)
Npm, pip, git, vagrant, ... and even requests passes some of it's config on to urllib.

@drunkwcodes
Copy link

drunkwcodes commented Apr 14, 2019

@jobec You are probably right. It should be handled by poetry.

What I considered about is the inconsistency between the env vars and the poetry CA config.
The setting is used by custom certificate users but the majority poetry users only query PyPI.
The setting will be often overlooked and outdated, more of a concern rather than a degree of freedom.

Placing it in env vars is fine. And we can choose handling it by python or the shell. Temporary or not.
https://stackoverflow.com/questions/47285018/how-to-set-environment-variables-using-python-in-windows-linux

@kmray
Copy link

kmray commented Jun 13, 2019

I just ran into this issue and the solution of setting REQUESTS_CA_BUNDLE will be extremely inconvenient.

We are currently using two private repositories: an internal one with a corporate certificate and a SaaS hosted one with a public certificate. If I set the REQUESTS_CA_BUNDLE to the corporate certificate, the SaaS won't work and if I set it to the regular bundle, the private one won't work.

A workaround would be to create a hybrid bundle with both in it and I could manage this on the builds but this would also require that all users set the variable to the hybrid bundle when working on the repo. Having a "cert = " field in source would solve this problem on a per source basis.

@drunkwcodes
Copy link

drunkwcodes commented Jun 13, 2019

@kmray Is the project public to outer space?

There is no need to add field if hybrid bundle is usable to all project members.

@Caligatio
Copy link
Contributor

I'm forgetting if GitHub emails people on cross ticket mentions but I have a PR, #1325, that addresses this request. The developers said in Discord that they're ranking PRs by number of votes so please +1 the PR :)

@stale
Copy link

stale bot commented Nov 13, 2019

Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

@stale stale bot added the wontfix label Nov 13, 2019
@sdispater sdispater added stale and removed wontfix labels Nov 14, 2019
@stale stale bot removed the stale label Nov 14, 2019
@KyleKing
Copy link
Contributor

🤔 With #1325, can we use Poetry to set the CA Certificate path rather than the hackish REQUESTS_CA_BUNDLE variable? I've tried to read through the documentation/tickets and tried a number of options, but still run into the self-signed certificate error

(I'm on Windows in a corporate environment that uses a self-signed certificate)

@Caligatio
Copy link
Contributor

Is this for installing packages? If so, poetry config certificates.foo.cert /path/to/ca.pem should work as long as you previously configured "foo" in your pyproject.toml using something like

[[tool.poetry.source]]
name = "foo"
url = "https://foo.bar/simple/"

There is still an outstanding bug if you have periods in your repo shortname.

@KyleKing
Copy link
Contributor

Is there a user error on my part or is the feature not intended for this use case?


This works:

SET REQUESTS_CA_BUNDLE=C:\Users\king.kyle\certificate.pem
poetry add cerberus

But I'm running errors when trying:

poetry config certificates.pypi.cert C:\Users\king.kyle\certificate.pem
poetry add cerberus -vvv

(Note: also tried adding poetry config certificates.pypi.client-cert C:\Users\king.kyle\certificate.pem)


I added the below snippet to the toml file:

[[tool.poetry.source]]
name = "foo"
url = "https://pypi.org/"

Then tried:

poetry config certificates.foo.cert C:\Users\king.kyle\certificate.pem
poetry add cerberus -vvv

Full error:

(py375) C:\Users\king.kyle\Developer\Project>poetry add cerberus -vvv
Using virtualenv: C:\Users\king.kyle\AppData\Local\Continuum\anaconda2\envs\py375

[SSLError]
HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /pypi/cerberus/json (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1076)')))

Traceback (most recent call last):
  File "C:\Users\king.kyle\.poetry\lib\poetry\_vendor\py3.7\clikit\console_application.py", line 131, in run
    status_code = command.handle(parsed_args, io)
  File "C:\Users\king.kyle\.poetry\lib\poetry\_vendor\py3.7\clikit\api\command\command.py", line 120, in handle
    status_code = self._do_handle(args, io)
  File "C:\Users\king.kyle\.poetry\lib\poetry\_vendor\py3.7\clikit\api\command\command.py", line 171, in _do_handle
    return getattr(handler, handler_method)(args, io, self)
  File "C:\Users\king.kyle\.poetry\lib\poetry\_vendor\py3.7\cleo\commands\command.py", line 92, in wrap_handle
    return self.handle()
  File "C:\Users\king.kyle\.poetry\lib\poetry\console\commands\add.py", line 89, in handle
    packages, allow_prereleases=self.option('allow-prereleases')
  File "C:\Users\king.kyle\.poetry\lib\poetry\console\commands\init.py", line 303, in _determine_requirements
    requirement['name'], allow_prereleases=allow_prereleases
  File "C:\Users\king.kyle\.poetry\lib\poetry\console\commands\init.py", line 333, in _find_best_version_for_package
    name, required_version, allow_prereleases=allow_prereleases
  File "C:\Users\king.kyle\.poetry\lib\poetry\version\version_selector.py", line 29, in find_best_candidate
    package_name, constraint, allow_prereleases=True
  File "C:\Users\king.kyle\.poetry\lib\poetry\repositories\pool.py", line 149, in find_packages
    name, constraint, extras=extras, allow_prereleases=allow_prereleases
  File "C:\Users\king.kyle\.poetry\lib\poetry\repositories\pypi_repository.py", line 112, in find_packages
    info = self.get_package_info(name)
  File "C:\Users\king.kyle\.poetry\lib\poetry\repositories\pypi_repository.py", line 265, in get_package_info
    name, lambda: self._get_package_info(name)
  File "C:\Users\king.kyle\.poetry\lib\poetry\_vendor\py3.7\cachy\repository.py", line 174, in remember_forever
    val = value(callback)
  File "C:\Users\king.kyle\.poetry\lib\poetry\_vendor\py3.7\cachy\helpers.py", line 6, in value
    return val()
  File "C:\Users\king.kyle\.poetry\lib\poetry\repositories\pypi_repository.py", line 265, in <lambda>
    name, lambda: self._get_package_info(name)
  File "C:\Users\king.kyle\.poetry\lib\poetry\repositories\pypi_repository.py", line 269, in _get_package_info
    data = self._get('pypi/{}/json'.format(name))
  File "C:\Users\king.kyle\.poetry\lib\poetry\repositories\pypi_repository.py", line 364, in _get
    json_response = self._session.get(self._url + endpoint)
  File "C:\Users\king.kyle\.poetry\lib\poetry\_vendor\py3.7\requests\sessions.py", line 546, in get
    return self.request('GET', url, **kwargs)
  File "C:\Users\king.kyle\.poetry\lib\poetry\_vendor\py3.7\requests\sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Users\king.kyle\.poetry\lib\poetry\_vendor\py3.7\requests\sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "C:\Users\king.kyle\.poetry\lib\poetry\_vendor\py3.7\cachecontrol\adapter.py", line 53, in send
    resp = super(CacheControlAdapter, self).send(request, **kw)
  File "C:\Users\king.kyle\.poetry\lib\poetry\_vendor\py3.7\requests\adapters.py", line 514, in send
    raise SSLError(e, request=request)


(py375) C:\Users\king.kyle\Developer\Project>

@Caligatio
Copy link
Contributor

If you have

[[tool.poetry.source]]
name = "foo"
url = "https://pypi.org/"

then you must then do poetry config certificates.foo.cert C:\Users\king.kyle\certificate.pem (note the "foo" part). The "foo" is the part that relates a CA to a given custom repo.

@KyleKing
Copy link
Contributor

Yeah, that was what I tried. My last comment could have been ordered in reverse since the third example was this approach.

(py375) C:\Users\king.kyle\Developer\Projectpoetry config --list
certificates.foo.cert = "C:\\Users\\king.kyle\\certificate.pem"  # None
virtualenvs.create = true
virtualenvs.in-project = false
virtualenvs.path = "{cache-dir}\\virtualenvs"  # C:\Users\king.kyle\AppData\Local\pypoetry\Cache\virtualenvs

(py375) C:\Users\king.kyle\Developer\Project

@Caligatio
Copy link
Contributor

Caligatio commented Dec 18, 2019

Apologies for missing that last bit. I think I have a good idea what might be happening; can you try (note the new default line and the non-pypi name):

[[tool.poetry.source]]
name = "foo"
url = "https://pypi.org/"
default = true

Then do poetry config certificates.foo.cert C:\Users\king.kyle\certificate.pem

PyPi is special amongst repository peers in Poetry in that it bypasses my custom certificate logic. It also always gets consulted to see if a package is available. I believe it's probably doing the "foo" lookup correctly but, since you're squatting on the PyPi domain, the special PyPi lookup is failing. The default = true line should disable the default PyPi repo and only use your custom "foo" repo.

@KyleKing
Copy link
Contributor

Thanks, making progress!

I created a new project poetry new TestPyPiSelfSigned, added [[tool.poetry.source]] with default = true to the pyproject.toml, and set certificates.foo.cert

Those steps appear to resolve the self-signed errors, but poetry appears unable to connect to pypi to identify package versions. Unseting the poetry configuration causes the self-signed errors to reappear

(py373) C:\Users\king.kyle\Developer\TestPyPiSelfSigned>poetry install -vvv
Using virtualenv: C:\Users\king.kyle\AppData\Local\Continuum\anaconda2\envs\py373
Updating dependencies
Resolving dependencies...
   1: fact: testpypiselfsigned is 0.1.0
   1: derived: testpypiselfsigned
   1: fact: testpypiselfsigned depends on pytest (^5.2)
   1: fact: testpypiselfsigned depends on pytest (^5.2)
   1: selecting testpypiselfsigned (0.1.0)
   1: derived: pytest (^5.2)
   1: fact: no versions of pytest match >=5.2,<6.0
   1: conflict: no versions of pytest match >=5.2,<6.0
   1: ! pytest (^5.2) is satisfied by pytest (^5.2)
   1: ! which is caused by "testpypiselfsigned depends on pytest (^5.2)"
   1: ! thus: version solving failed
   1: Version solving took 0.328 seconds.
   1: Tried 1 solutions.

[SolverProblemError]
Because testpypiselfsigned depends on pytest (^5.2) which doesn't match any versions, version solving failed.

Traceback (most recent call last):
  File "C:\Users\king.kyle\.poetry\lib\poetry\_vendor\py3.7\clikit\console_application.py", line 131, in run
    status_code = command.handle(parsed_args, io)
  File "C:\Users\king.kyle\.poetry\lib\poetry\_vendor\py3.7\clikit\api\command\command.py", line 120, in handle
    status_code = self._do_handle(args, io)
  File "C:\Users\king.kyle\.poetry\lib\poetry\_vendor\py3.7\clikit\api\command\command.py", line 171, in _do_handle
    return getattr(handler, handler_method)(args, io, self)
  File "C:\Users\king.kyle\.poetry\lib\poetry\_vendor\py3.7\cleo\commands\command.py", line 92, in wrap_handle
    return self.handle()
  File "C:\Users\king.kyle\.poetry\lib\poetry\console\commands\install.py", line 63, in handle
    return_code = installer.run()
  File "C:\Users\king.kyle\.poetry\lib\poetry\installation\installer.py", line 74, in run
    self._do_install(local_repo)
  File "C:\Users\king.kyle\.poetry\lib\poetry\installation\installer.py", line 161, in _do_install
    ops = solver.solve(use_latest=self._whitelist)
  File "C:\Users\king.kyle\.poetry\lib\poetry\puzzle\solver.py", line 36, in solve
    packages, depths = self._solve(use_latest=use_latest)
  File "C:\Users\king.kyle\.poetry\lib\poetry\puzzle\solver.py", line 190, in _solve
    raise SolverProblemError(e)

(py373) C:\Users\king.kyle\Developer\TestPyPiSelfSigned>poetry config certificates.foo.cert --unset

(py373) C:\Users\king.kyle\Developer\TestPyPiSelfSigned>poetry install
Updating dependencies
Resolving dependencies...

[SSLError]
HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /pytest/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1056)')))

@KyleKing
Copy link
Contributor

KyleKing commented Dec 18, 2019

Ooh! Actually solved it, the path should be https://pypi.org/project/

[[tool.poetry.source]]
name = "pypi_with_cert"
url = "https://pypi.org/project/"
default = true
poetry config certificates.pypi_with_cert.cert C:\Users\king.kyle\certificate.pem

Would it be useful to add this to the documentation somewhere?

Edit: see comment below (#1012 (comment)), the url should be url = "https://pypi.org/simple/"

@KyleKing
Copy link
Contributor

Hmm, I'm running into an error with those changes. Seems to work fine if the version specification matches the latest, but poetry can't identify prior versions. For example, changing the pytest = "^5.0" to pytest = "^4.0" causes the install to fail. Is this an issue with the url ("https://pypi.org/project/")?

Create new project

poetry new TestPyPiSelfSigned

Modify the toml file

[tool.poetry]
name = "TestPyPiSelfSigned"
version = "0.1.0"
description = ""
authors = ["Kyle King"]

[[tool.poetry.source]]
name = "pypi_with_cert"
url = "https://pypi.org/project/"
default = true

[tool.poetry.dependencies]
python = "^3.7"

[tool.poetry.dev-dependencies]
pytest = "^4.0"

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

Try to install or update

(py373) C:\Users\king.kyle\Developer\TestPyPiSelfSigned>poetry update -vvv
Using virtualenv: C:\Users\king.kyle\AppData\Local\Continuum\anaconda2\envs\py373
Updating dependencies
Resolving dependencies...
   1: fact: testpypiselfsigned is 0.1.0
   1: derived: testpypiselfsigned
   1: fact: testpypiselfsigned depends on pytest (^4.0)
   1: fact: testpypiselfsigned depends on pytest (^4.0)
   1: selecting testpypiselfsigned (0.1.0)
   1: derived: pytest (^4.0)
pypi_with_cert: 0 packages found for pytest >=4.0,<5.0
   1: fact: no versions of pytest match >=4.0,<5.0
   1: conflict: no versions of pytest match >=4.0,<5.0
   1: ! pytest (^4.0) is satisfied by pytest (^4.0)
   1: ! which is caused by "testpypiselfsigned depends on pytest (^4.0)"
   1: ! thus: version solving failed
   1: Version solving took 0.491 seconds.
   1: Tried 1 solutions.

[SolverProblemError]
Because testpypiselfsigned depends on pytest (^4.0) which doesn't match any versions, version solving failed.

Traceback (most recent call last):
  File "C:\Users\king.kyle\.poetry\lib\poetry\_vendor\py3.7\clikit\console_application.py", line 131, in run
    status_code = command.handle(parsed_args, io)
  File "C:\Users\king.kyle\.poetry\lib\poetry\_vendor\py3.7\clikit\api\command\command.py", line 120, in handle
    status_code = self._do_handle(args, io)
  File "C:\Users\king.kyle\.poetry\lib\poetry\_vendor\py3.7\clikit\api\command\command.py", line 171, in _do_handle
    return getattr(handler, handler_method)(args, io, self)
  File "C:\Users\king.kyle\.poetry\lib\poetry\_vendor\py3.7\cleo\commands\command.py", line 92, in wrap_handle
    return self.handle()
  File "C:\Users\king.kyle\.poetry\lib\poetry\console\commands\update.py", line 49, in handle
    return installer.run()
  File "C:\Users\king.kyle\.poetry\lib\poetry\installation\installer.py", line 74, in run
    self._do_install(local_repo)
  File "C:\Users\king.kyle\.poetry\lib\poetry\installation\installer.py", line 161, in _do_install
    ops = solver.solve(use_latest=self._whitelist)
  File "C:\Users\king.kyle\.poetry\lib\poetry\puzzle\solver.py", line 36, in solve
    packages, depths = self._solve(use_latest=use_latest)
  File "C:\Users\king.kyle\.poetry\lib\poetry\puzzle\solver.py", line 190, in _solve
    raise SolverProblemError(e)


(py373) C:\Users\king.kyle\Developer\TestPyPiSelfSigned>

@KyleKing
Copy link
Contributor

Ok, final answer? Sorry for the spam. I believe we want the URL "https://pypi.org/simple/" (based on documentation from the Link class)

Object representing a parsed link from https://pypi.python.org/simple/*

This now works

[[tool.poetry.source]]
name = "pypi_with_cert"
url = "https://pypi.org/simple/"
default = true

Then run: poetry config certificates.pypi_with_cert.cert C:\Users\Path\To\certificate.pem

@Caligatio
Copy link
Contributor

Caligatio commented Dec 18, 2019

Huzzah! As a rule of thumb, you'll want /simple for most repositories.

@sdispater: Do you have any strong feelings one way or the other about allowing a certificate authority override for PyPI? I'm happy to do a PR with the relevant changes if you're supportive.

@sdispater
Copy link
Member

@Caligatio What is the use case for overriding the certificate authority for PyPI?

@jobec
Copy link
Author

jobec commented Dec 20, 2019

Being behind a corporate firewall that does TLS interception

@Caligatio
Copy link
Contributor

Either what @jobec said or if you're in a corporate environment that squats on external domain names.

My first thought on this is that it seems like a not-great security idea but, on the other hand, it would require explicit settings to enable and thus shouldn't be an accident. We got something working for @KyleKing in this thread but it wasn't exactly obvious what the actual problem was.

@jobec
Copy link
Author

jobec commented Dec 20, 2019

TLS interception is usually done as a security measure. It allows to virus/threath scan what’s being downloaded. Whether it’s effective is a different discussion, but in many companies it’s a policy.

And I agree, this can’t be configured by accident and you need to know how and where to get such root CA bundle. So the chance of abuse is low.

@KyleKing
Copy link
Contributor

Unfortunately, I'm not privy to what changed in the IT Infrastructure. There was a major overhaul in the last two years that caused us to start seeing the error "Self-Signed Certificate" whenever attempting to make a request with pip

Creating a pem file isn't difficult, but you need to know the steps, so probably at least 60% of the people using this workaround would be my coworkers

@jerodg
Copy link

jerodg commented Jan 2, 2020

I couldn't get this to work on mac through the proxy. Pip is configured with our internal certificate and works fine but I can't even get poetry to install due to the self-signed certificate error.

Are there any workarounds to install through proxy?

@Caligatio
Copy link
Contributor

@jerodg Are you having troubles installing poetry or installing packages using Poetry? If the prior, how you installing it? I typically use pip to bootstrap Poetry and that sounds like it works for you.

@jerodg
Copy link

jerodg commented Jan 2, 2020

@Caligatio I can't even install it because of the proxy. I know this thread was mostly about packages but I didn't see any other issues relating to it. It throws a urllib error.

@jobec
Copy link
Author

jobec commented Jan 17, 2020

This doesn't work as expected it seems.

poetry config certificates.private_pypi.cert /project/certificates/ca-bundle.pem --local

It stores the certificate config in /home/user/.config/pypoetry/auth.toml instead of local to the project without giving warnings or anything.

Manually copying the config from that file to the poetry.toml file in the project root works though. Allowing you to configure it at the project level.

@Caligatio
Copy link
Contributor

Caligatio commented Jan 17, 2020 via email

@jobec
Copy link
Author

jobec commented Jan 17, 2020

Then it doesn't match with being able to configure a private repository on the project level but not configure it's root CA.

@Caligatio
Copy link
Contributor

Correct, you don't want to store anything related to custom certificate authorities/client certificates in the project configuration. Certificate authority handling and private certificates are on a per user per repository basis. If I clone your project but want to have the certificate file in a different location, I'm stuck. I can't really modify the pyproject.toml to change the path without possible merge conflicts.

@HuM4NoiD
Copy link

HuM4NoiD commented Sep 6, 2022

Alternatively, poetry should pick up certificates/certificate bundle path from pip config (user/system/site config)

@bkeyser
Copy link

bkeyser commented Oct 11, 2022

The certificate configuration works well when using PyPI. However, if a file is downloaded from files.pythonhosted.org, then the installation fails with the following error

HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Max retries exceeded with url: /packages/1c/f1/612f9aa29f33b25a1034749dde67dfbf6de9b297709d06df71f9bfabfc81/robotframework_selenium2library-3.0.0-py2.py3-none-any.whl (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)')))

Setting the REQUESTS_CA_BUNDLE environment variable enables the download of the wheel referenced above.

The following frame is from the requests library

at ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/requests/adapters.py:563 in send

559│                 raise ProxyError(e, request=request)
560│ 
561│             if isinstance(e.reason, _SSLError):
562│                 # This branch is for urllib3 v1.22 and later.
→ 563│                 raise SSLError(e, request=request)
564│ 
565│             raise ConnectionError(e, request=request)
566│ 
567│         except ClosedPoolError as e:

If the verify parameter of the send() method is set to the certificate file (the same file used in the REQUESTS_CA_BUNDLE environment variable), then the install works correctly.

@neersighted
Copy link
Member

@bkeyser that's something of a misdiagnosis -- of course if you turn off SSL validation, it works. The issue is that Poetry can turn if off for pypi.org but we don't have a good mechanism to do it for files.pythonhosted.org which pypi.org hosts its files at.

@drorata
Copy link

drorata commented Oct 11, 2022

FWIW, I have serious issues with both installing Poetry itself and installing packages using poetry.

As for the 1st part, I was eventually successful after a lot of messing around. I'm now stuck and unable to install packages using poetry. Here's an example of the error I get:

SSLCertVerificationError

  [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)

  at ~\.pyenv\pyenv-win\versions\3.10.2\lib\ssl.py:1341 in do_handshake
      1337│         timeout = self.gettimeout()
      1338│         try:
      1339│             if timeout == 0.0 and block:
      1340│                 self.settimeout(None)
    → 1341│             self._sslobj.do_handshake()
      1342│         finally:
      1343│             self.settimeout(timeout)
      1344│
      1345│     def _real_connect(self, addr, connect_ex):

The following error occurred when trying to handle this error:


  MaxRetryError

  HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /pypi/six/1.16.0/json (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)')))

  at ~\AppData\Roaming\pypoetry\venv\lib\site-packages\urllib3\util\retry.py:592 in increment
      588│             history=history,
      589│         )
      590│
      591│         if new_retry.is_exhausted():
    → 592│             raise MaxRetryError(_pool, url, error or ResponseError(cause))
      593│
      594│         log.debug("Incremented Retry for (url='%s'): %r", url, new_retry)
      595│
      596│         return new_retry

The following error occurred when trying to handle this error:


  SSLError

  HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /pypi/six/1.16.0/json (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)')))

  at ~\AppData\Roaming\pypoetry\venv\lib\site-packages\requests\adapters.py:563 in send
      559│                 raise ProxyError(e, request=request)
      560│
      561│             if isinstance(e.reason, _SSLError):
      562│                 # This branch is for urllib3 v1.22 and later.
    → 563│                 raise SSLError(e, request=request)
      564│
      565│             raise ConnectionError(e, request=request)
      566│
      567│         except ClosedPoolError as e:

Note that I am having these issues when being behind corporate firewall/VPN.

@neersighted
Copy link
Member

@drorata You can solve those by setting poetry config certificates.pypi.cert false. However, you will then hit verification issues for files.pythonhosted.org which is what I referred to above.

You can add a publishing-only repository to 'trick' Poetry into also trusting files.pythonhosted.org, but then you'll hit #3249 (or a specialized variant of it with --trusted-host).

@bkeyser
Copy link

bkeyser commented Oct 11, 2022

@neersighted For those of us who are behind a corporate firewall with a self-signed CA, could a default certificate be set? I don't fully understand the history, but it appears this approach might have been tried in #1325. However, this approach seems to contradict the comment

Certificate authority handling and private certificates are on a per user per repository basis.

The REQUESTS_CA_BUNDLE environment variable can be set as a way to work around the problem (at least for me ;-) )

@neersighted
Copy link
Member

neersighted commented Oct 11, 2022

The default certificate is REQUESTS_CA_BUNDLE. If you want to set the base bundle used by Poetry to authenticate servers, that is where you do it at present.

poetry config repositories.<name>.cert is not captured in your repo and is local to your machine, so there is no contradiction.

@uclatommy
Copy link

I want to share how I resolved this error with poetry. I am in a corporate environment, on windows10 and python 3.10. We have our own root CA certificate. A fresh install of Poetry 1.5 was giving me errors like this:

  SSLError

  HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /pypi/webencodings/0.5.1/json (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)')))

  at ~\AppData\Roaming\pypoetry\venv\lib\site-packages\requests\adapters.py:517 in send
      513│                 raise ProxyError(e, request=request)
      514│ 
      515│             if isinstance(e.reason, _SSLError):
      516│                 # This branch is for urllib3 v1.22 and later.
    → 517│                 raise SSLError(e, request=request)
      518│
      519│             raise ConnectionError(e, request=request)
      520│
      521│         except ClosedPoolError as e:

It's the same issue I faced with pip but I had managed to solve the ssl problem with pip by installing pip-system-certs. If pip is working for you, you can install it by:

pip install pip-system-certs

Otherwise, you'll have to download the wheel and do a manual install first to get pip working.

So what I did to fix Poetry was to go into poetry's venv and pip install pip-system-certs. So for me that looks like this:

cd C:\Users\tchen\AppData\Roaming\pypoetry\venv\Scripts
pip install pip-system-certs

After that, I exited the venv, went into my poetry project directory and used the poetry commands as normal and everything worked.

@TranquilOasis
Copy link

I want to share how I resolved this error with poetry. I am in a corporate environment, on windows10 and python 3.10. We have our own root CA certificate. A fresh install of Poetry 1.5 was giving me errors like this:

  SSLError

  HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /pypi/webencodings/0.5.1/json (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)')))

  at ~\AppData\Roaming\pypoetry\venv\lib\site-packages\requests\adapters.py:517 in send
      513│                 raise ProxyError(e, request=request)
      514│ 
      515│             if isinstance(e.reason, _SSLError):
      516│                 # This branch is for urllib3 v1.22 and later.
    → 517│                 raise SSLError(e, request=request)
      518│
      519│             raise ConnectionError(e, request=request)
      520│
      521│         except ClosedPoolError as e:

It's the same issue I faced with pip but I had managed to solve the ssl problem with pip by installing pip-system-certs. If pip is working for you, you can install it by:

pip install pip-system-certs

Otherwise, you'll have to download the wheel and do a manual install first to get pip working.

So what I did to fix Poetry was to go into poetry's venv and pip install pip-system-certs. So for me that looks like this:

cd C:\Users\tchen\AppData\Roaming\pypoetry\venv\Scripts
pip install pip-system-certs

After that, I exited the venv, went into my poetry project directory and used the poetry commands as normal and everything worked.

This is the comment that finally helped me resolve my issues with poetry. Thank you

@corneels
Copy link

@TranquilOasis can you give more detail on how pip-system-certs resolved your issues?

After installing pip-system-certs in my global python install, as well as within poetry, I am still getting cert errors.

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/feature Feature requests/implementations
Projects
None yet
Development

Successfully merging a pull request may close this issue.