aiohttp cannot make HTTPS (SSL) requests in a Windows container

[doctorpangloss]

Describe the bug

Making an aiohttp request will fail in a Windows Containers on Windows (WCOW) container. It ought to fail on all completely clean, newly booted versions of bare metal Windows 2022.

This is related to microsoft/Windows-Containers#503

The root cause is that Windows containers, like all Windows 2022 images, do not have an equivalent of ca-certificates, the certificate store on Windows does not look like a browser's certificate bundle.

To Reproduce

1. Create a Dockerfile based on a Windows image that makes an aiohttp request to an https URL.
2. Observe it fails. It should not.
3. Use certifi as the ssl context (this example makes sense for fsspec, which expects an async method and ClientSession, but the docs provide an example that make sense for other libraries):

async def get_client(**kwargs):
     """
     workaround for issues with fsspec on Windows
     :param kwargs:
     :return:
     """
     ssl_context = ssl.create_default_context(cafile=certifi.where())
     conn = aiohttp.TCPConnector(ssl=ssl_context)
     return aiohttp.ClientSession(connector=conn, **kwargs)
     
...
# idiosyncratic to fsspec
with fsspec.open("https://www.wikimedia.com/...", get_client=get_client) as f:
    ...

4. Observe these sessions can successfully make requests.
5. curl -I the URL in the container's shell.
6. Make the request without the workaround. Observe it succeeds.

This will not reproduce using docker run, because docker mounts something from the host into the container which propagates the side effects of curl -I.

Expected behavior

aiohttp should behave normally on Windows containers, i.e., on clean copies of Windows.

For the most part, aiohttp has been working on Windows because in a desktop environment, enough side effects have occurred (i.e. calls like curl) to have fixed aiohttp all this time.

Logs/tracebacks

(no logs applicable)

Python Version

$ python --version
3.11.9

aiohttp Version

$ python -m pip show aiohttp
(all versions affected)

multidict Version

$ python -m pip show multidict
(not applicable)

yarl Version

$ python -m pip show yarl
(not applicable)

OS

Windows 2022 Server

Related component

Client

Additional context

No response

Code of Conduct

• I agree to follow the aio-libs Code of Conduct

[webknjaz]

certifi is a hack that the requests maintainers regret having to do. The end-users are responsible for setting up what they trust. Libraries like aiohttp shouldn't be making assumptions or taking over this responsibility.

[Dreamsorcerer]

Yeah, unless you have a specific suggestion of some way we are loading the certificates wrong, then this seems like an issue with the platform.

If you want to use the certifi hack, similar to requests, then that is documented at: https://docs.aiohttp.org/en/stable/client_advanced.html#example-use-certifi

[doctorpangloss]

If you want to use the certifi hack, similar to requests, then that is documented at: https://docs.aiohttp.org/en/stable/client_advanced.html#example-use-certifi

The documentation has flaws, for example:

By default, Python uses the system CA certificates

This may be true. But requests does not only use the system CA certificates by default. It ships with the web certificates bundle. My specific suggestion is that aiohttp should ship with the bundle.

I am using the pattern that I do because other libraries, like fsspec, want a ClientSession object. I didn't choose to use aiohttp, my libraries did.

Libraries like aiohttp shouldn't be making assumptions or taking over this responsibility.

Nobody would be able to install things on Windows or macOS if this strategy were used with requests, because pip uses requests and used to throw this error all the time, for Windows and macOS. The messages were vague - maybe that's another fix, make less vague errors - that they didn't report to bug trackers but on Stack Overflow instead. If any reporting at all.

Many users with aiohttp installed and used in Python didn't choose to use it. Just like with requests. When using pip, nobody chose to use requests. So they added the certificate bundle so that installing packages would work on Windows without vague errors.

[webknjaz]

FTR, there's some good external discussions on the topic:

• Ask users not to raise issues to add random/system certs certifi/python-certifi#72 (comment)
• Consider using system trust stores by default in 3.0.0. psf/requests#2966

The latter also contains a recommendation of the PSF Security Developer-in-Residence to use https://truststore.rtfd.io/#using-truststore-with-aiohttp to pre-configure the CA chain. (though their aiohttp snippet needs to be fixed to avoid resource warnings / cc @sethmlarson — I made a PR to fix them @ sethmlarson/truststore#139)

It also explicitly explains that setting up the cert store is something to be done in applications and not in libraries/frameworks.

Hopefully, PEP 543 resurrection will also get back on track (I noticed that @woodruffw started https://github.com/trailofbits/tlslib.py, and he usually gets the job done) bringing nicer APIs in this field.

[webknjaz]

@doctorpangloss by the way, you don't need an async def in your helper function since you don't actually do anything async inside. JFYI.

[Dreamsorcerer]

The latter also contains a recommendation of the PSF Security Developer-in-Residence to use https://truststore.rtfd.io/#using-truststore-with-aiohttp to pre-configure the CA chain.

Maybe this link can be added to our docs.

[webknjaz]

@Dreamsorcerer yep, that was my thinking. Additionally, we may want to document that it's possible to pass truststore.SSLContext into our interfaces.

[sethmlarson]

Here to give a big ++ to everything @webknjaz said, applications shouldn't be carrying around their own certificate bundles since it's only another layer of headache for operations to keep up-to-date. Delegating to the system is what all other pieces of software do, so Python applications should too.

Pip got mentioned as well, pip already supports using Truststore (recent versions with --use-feature=truststore) and will soon be enabled by default.

[doctorpangloss]

@doctorpangloss by the way, you don't need an async def in your helper function since you don't actually do anything async inside. JFYI.

Yes, I thought the same, but that's what fssync wants, it awaits get_client! So it goes.

I appreciate all the attention to the issue, thank you. I have also raised the issue with the Windows Containers team.

Ultimately, the reason aiohttp doesn't bug out on Windows is because people are using desktop Windows interactively, and eventually, they have made a requests call / curl call or some similar interaction that has some kind of side effects which allow aiohttp to successfully connect to many https:// addresses. I don't understand fully the side effects, and have asked folks at Microsoft to clarify. Specifically, Windows curl.exe seems to somehow activate the specific certificate needed for the specific address requested, which many other URLs will share, and those other URLs sharing that certificate will work with aiohttp. But access a URL that doesn't have a system activated (unknown side effect) certificate, and it will fail.

To summarize, aiohttp on a clean, naked, vanilla Windows, desktop or container or otherwise, will always fail to access https:// URLs.

[doctorpangloss]

Related to #955

To a certain extent, this ticket simply means that aiohttp will always randomly fail on Windows machines, because it has adopted a stance on SSL certificates - not shipping them - on a platform that does not ship with SSL certificates.

One resolution is shown in the Python code here: microsoft/Windows-Containers#503 (comment) which will essentially use certifi as your trust store.

Another solution is the equivalent of installing ca-certificates on Windows which is here: microsoft/Windows-Containers#503 (comment)

Hopefully people who find this ticket with various ssl verify errors will encourage the aiohttp maintainers to reconsider this position from a practical point of view. The only solution right now is using certifi.