truststore feature does not work when set in a requirement file

[JCapul]

Description

Hi,

After digging a bit, I am quite confident that the truststore feature does not work as expected when the option is placed in a requirement file.

In my org, we have a PyPi server with self-signed certs. I managed to make pip install work with the truststore feature when used in command line option, but when used in a requirement files, I get the typical error unable to get local issuer certificate.

Expected behavior

I am expecting no difference in behaviour of the truststore feature whether it is set as command line option or in a requirement file (since the option use-feature is allowed in requirement files).

pip version

23.2.1

Python version

3.10

OS

Windows 10 and Ubuntu 20.04 (I checked both)

How to Reproduce

Not easy to reproduce, as you might need to set up a self signed pypi server or equivalent to reproduce the bug.
But basically, considering a requirements file like this:

requirements.txt

--use-feature truststore
--extra-index-url https://[custom_pypi_server]

some_package_on_custom_pypi_server

the command pip install -r requirements.txt will fail with an SSLContext error (see output after).

However, the command pip install --use-feature truststore -r requirements.txt works as expected.

After digging, I think the issue comes from the fact that without the commad line option, the PipSession object is instanciated with the default SSLContext. But this default context is not replaced by the truststore SSLContext when the --use-feature truststore line is parsed in the requirement file.

Output

$ pip install -r requirements.txt
Looking in indexes: https://pypi.org/simple, https://[custom_pypi_server]/simple/
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1002)'))': /simple/dlab-tools/

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[SpiderD555]

This is a chicken and egg problem. To use --use-feature truststore option you need to install truststore first.
My recommendation for you is to install truststore first this way pip install --trusted-host [custom_pypi_server] truststore. Only then you can proceed with the rest of requirements. Now keep in mind that each new venv you create won't have truststore installed, so you will need to repeat the procedure again.
If comment sethmlarson/truststore#116 (comment) is true, then the chicken and egg problem will be no more with pip v23.3

[JCapul]

Thanks for you answer.
I forgot to mention that I did install the package truststore beforehand in my environment. And the option --use-feature truststore works as expected when I use it in the pip install command line, but not when embedded in the requirements file.

[uranusjr]

The problem may be the certificate store is initialised too early in the process and it’s too late when the requirements file is parsed. In general I kind of doubt is --use-features is supposed to be allowed in the first place since it introduces a bunch of weird expectations, such as using the legacy resolver for only one requirements file (it would not work).

[pfmoore]

I agree. My first thought here was “--use-feature doesn’t work in requirements files anyway”, and I was surprised to find that it’s documented to work there. Does anyone know why it was allowed in the first place?

I’d prefer that we deprecate the use of --use-feature in requirements files, and immediately desupport specific cases like this that don’t work. I don’t think the effort of trying to make it work in general is worth it.

[JCapul]

Personally, I can live eventually with --use-feature option only in command line. Since it was advertised as possible in documentation and that it looked to be broken for the truststore feature, I raised the issue.