Migrate network stack away from requests?

[uranusjr]

I had a couple of conversations regarding Requests that makes me think about replacing it with something else. While Requests is easily the most battle-tested and mature (pure Python) HTTP client implementation available, it does comes with issues.

The most problematic part is licensing—one of its dependencies, chardet, is licensed under LGPL. This means pip’s MIT licensing is considered ambiguous (and potentially invalid) by some since it vendors chardet.

The other is project maintenance. HTTP is an ever-changing landscape these days, and Requests, being a mature lirbary that priorities stability, is slow to adapt new features, such as HTTP/2 (not to mention HTTP/3!) and coroutines (async/await).

Historically, pip does not really have a choice since requests is the only viable option that supports Python 2.7. But now we’re 3.6+ we have a lot more choices, e.g. aiohttp and httpx. I am not sure whether either of them is better than requests in either regards (especially they have a lot more dependencies than requests), but I feel this is a good timing to throw out this thought.

[pradyunsg]

I like the basic idea here, and I think aiohttp might be the better option between the two for our usecases

I think the bigger concern/thought I have is whether it makes sense for us to, well, make the entire codebase async or to keep it sync TBH. Both of these should also have sync variants, but the async API has significantly greater benefits AFAICT.

[uranusjr]

I don’t think aiohttp has a sync interface, but it’s pretty easy to make async functions sync anyway.

[pfmoore]

httpx has a similar interface to requests, AIUI, but I'm not sure how important that is (we only use a small subset of the requests API). It also has both async and sync variants. And aiohttp depends on chardet, so wouldn't actually solve the original problem 🙁

Also, when combined with its dependencies it's significantly smaller than aiohttp.

Sizes:

• requests: 597KB
• httpx: 272KB
• aiohttp: 1122KB

So the size of the pip wheel would change as follows:

• requests: 100%
• httpx: 78%
• aiohttp: 134%

That's a pretty significant size change.

So in summary, +1 from me, I prefer httpx as it doesn't use chardet and it's significantly smaller than the alternatives.

@pradyunsg I'm not sure what you mean by "full async". I'd go async for all network activity because, well why not? But outside of that, what's to gain? I doubt we could easily run backends async, and everything else is either CPU-bound or file operations, which I don't think async does anything with.

But as @uranusjr says, wrapping a chunk of async code in a sync function shouldn't be hard.

[Ousret]

Hi,

Before dropping requests. There is still hope, see the ongoing conversation at psf/requests#5797 at least for python 3. Could be an easy fix.

[uranusjr]

@Ousret That thread is exactly what prompted me to open this discussion, and I am not too hopeful reading requests maintainers' response on the issue.

[potiuk]

We have a very similar discussion happening in Apache Software Foundation https://issues.apache.org/jira/browse/LEGAL-572 right now. The chardet transitive dependency has been brought to our attention as something that is not compliant with the Apache Software Foundation policies and there are ~ 50 projects of the ASF that will have to remove the requests dependency.

Unfortunately chardet maintainers also refused to change the licence several times:

• Changing license chardet/chardet#36
• LGPL and iPhone chardet/chardet#27

However we went a bit different route. @ashb from Apache Airflow proposed a PR : psf/requests#5797 where chardet is replaced (in a very soft way - including fallback to chardet if installed) with MIT-licensed charset-normalizer.

So far the requests maintainers were not willing to merge that requests, quoting 'feature-freeze' (see the discussion in the PR) but maybe if enough people and projects will join the request and they will see the critical mass, they might change their mind.

We have another, alternative proposal if they don't. We could jointly fork the requests library.

The idea is:

• we create an organisation in GitHub (we can even name it "requests-NonL-GPL" or similar).
• we fork requests there and semi-automatically apply the PR from Ash every time they release - and we release our own version right after (also release it to PyPI with requests-non-lgpl name )
• we fork there also any library that we need that uses requests and modify it to use our fork (and release to PyPI) again with modified name. Or we ask the maintainers of the library to switch to our non-lgpl depending version of requests. We can semi-automatically release those libraries when they are released.
• we ask all the ASF projects and anyone else who have problem with LGPL chardet to switch to use those non-lgpl depending libraries

This is a lo of hassle, but it seems doable, and easy to semi-automate, and I am willing to spear-head the effort if we go that route.

[pfmoore]

TBH, I don't want pip to get involved in the debate over requests and chardet. I consider that a decision for the requests maintainers, and not one we (pip) should attempt to influence.

So my vote remains to switch to httpx, but I don't feel that pip should treat this as a matter of significant urgency, just something that we should do in due course. The license question has been around for years now, any sudden urgency is IMO artificial "because we just noticed it".

Regarding going "full async", I've been doing some investigation recently into asyncio, and one potential issue with going that route is that the fast-deps code cannot be used with async http - specifically because the LazyZipOverHTTP class implements a file object's read() function via http, and read() cannot be async, as that's not how the file protocol is defined. However, I'm not an async expert here and I'd love to be proved wrong on this point.

[pradyunsg]

read() cannot be async

Yea, and loop.run_in_executor can be used to wrap blocking I/O operstions: https://docs.python.org/3/library/asyncio-eventloop.html#executing-code-in-thread-or-process-pools

[pfmoore]

Yea, and loop.run_in_executor can be used to wrap blocking I/O operstions

That's the wrong way around. What that code needs to do is essentially equivalent to

def read(self, n):
# Download some bytes over HTTP
return self.buffer[current:current+n]

If the HTTP code is async, it needs to be awaited, so read needs to be async, but it can't be. We can of course use asyncio.run (here), but that section notes "This function cannot be called when another asyncio event loop is running in the same thread", so it'll be tricky to get this right. Not impossible, but certainly tricky.

But this level of detail can wait until someone actually wants to work on going full-async. For the purposes of this issue, I think we can go to httpx initially, and look at async later if we want to.

[sigmavirus24]

Unfortunately chardet maintainers also refused to change the licence several times:

@potiuk I do want to point out that this is a gross misrepresentation of the discussions. We literally - under the terms of the LGPL license - can not change it. The original author is completely unreachable and thus unable to agree to changing the license. We refuse to change the license because we would be violating the terms of the LGPL by doing so, not because we ideologically agree with it more than anything else. Further, the original author (if I remember correctly) licensed the project under LGPL because large swaths of it (the tables of statistics in particular) were taken from Mozilla's LGPL'd universalchardet project that chardet was implemented to mimic in Python. The legal tangles are too deep for anyone to be bothered to deal with.

[pfmoore]

Thanks for the clarification @sigmavirus24.

Can I ask that the debates over licensing not spill over to the pip tracker - it sounds like chardet and requests are already getting enough traffic over this, and it helps no-one to start another extended and unproductive debate here.

The facts are:

1. The situation regarding the chardet license is only one of the reasons we're having this discussion about changing our HTTP library.
2. Nothing will change overnight, so "we're looking at the question" is the only answer anyone will get for the immediate future.
3. Getting us entangled in licensing debates will only slow progress on any change, as people will get burned out and find other, more enjoyable, things to spend their time on.

[sigmavirus24]

Also, for what it's worth, I used to frequently talk to Donald about moving to urllib3. I think that should be an option on the table as well

[potiuk]

Can I ask that the debates over licensing not spill over to the pip tracker - it sounds like chardet and requests are already getting enough traffic over this, and it helps no-one to start another extended and unproductive debate here.

Sure. This is no problem for me. I won't comment here any longer, I just noticed similar debate in PIP and thought it might raise the severity of the problem.

[pfmoore]

Also, for what it's worth, I used to frequently talk to Donald about moving to urllib3. I think that should be an option on the table as well

At 153KB and no dependencies, I agree we should consider it. The main benefit of httpx is that they claim compatibility with the requests API - which may or may not be important (if we're considering going to async, maybe not as much as I thought). I honestly have no idea how complex our needs are here.

[pradyunsg]

FWIW, we already depend on urllib3, through requests.