Replace HTTP client on TestClient from requests to httpx

[Kludex]

Missing:

• Remove asgiref.
• Use the previous import typing
• Fix skipped tests
• Remove docs references to requests.

[Kludex]

How do I send the headers in httpx the same way as requests?

See https://docs.python-requests.org/en/latest/user/quickstart/#post-a-multipart-encoded-file

[Kludex]

Test failing until encode/httpx#1936 is merged.

[tomchristie]

Great bit of work, thanks!

We'll want some careful decision making before we hit go on this. Particularly around what FastAPI users will expect.

Something that we could choose to do here would be switch from requests to httpx in the implementation but make sure that we're not introducing API changes. (Eg. continue to support allow_redirects, at the TestClient level, but perhaps soft-deprecate it.) Not sure.

[Kludex]

About the deprecation, anything else besides the item specified in the description of this PR and the allow_redirects that you have in mind? Because I've run the test suite of both FastAPI and Starlette using this source (on another repository), and only what was specified in the description above was a "problem", and also the fact that cookies parameter don't work the same way on the packages involved.

As a reminder, this PR cannot be concluded without taking a decision on encode/httpx#1936, or giving me orientation e.g. I can skip the test that involves that issue.

[tomchristie]

As a reminder, this PR cannot be concluded without taking a decision on encode/httpx#1936, or giving me orientation e.g. I can skip the test that involves that issue.

Indeed, so for now let's hold on that.

About the deprecation, anything else besides the item specified in the description of this PR and the allow_redirects that you have in mind? Because I've run the test suite of both FastAPI and Starlette using this source (on another repository), and only what was specified in the description above was a "problem", and also the fact that cookies parameter don't work the same way on the packages involved.

Well essentially it'd be interesting to consider two options...

• This PR as-is.
• This PR, but also with backwards compatibility, and without changes to the test suite.

A useful thing to consider here is that any changes to the footprint of our test suite here is a good indicator of what changes our users would also need to make to their test suites. If we're hitting them with a big set of required changes in order to upgrade, then that's not fantastic. (Although there might or might not be good enough reasons why we're prefer to do that.)

I'm mostly saying all this in order to flag up that I think we'll want @tiangolo's perspective, and that we will need to carefully look at the cost of API changes, and how we can mitigate that.

[Kludex]

Until next httpx release. This is a blocker.

[adriangb]

Could httpx cut a release? This is going to cause issues for anyone that depends on httpx right? I realize you can probably just not install the "full" extra, but this still breaks things for those that are already doing it / it's a bit unergonomic and confusing.

[Kludex]

As a reminder, this PR cannot be concluded without taking a decision on encode/httpx#1936, or giving me orientation e.g. I can skip the test that involves that issue.

Indeed, so for now let's hold on that.

About the deprecation, anything else besides the item specified in the description of this PR and the allow_redirects that you have in mind? Because I've run the test suite of both FastAPI and Starlette using this source (on another repository), and only what was specified in the description above was a "problem", and also the fact that cookies parameter don't work the same way on the packages involved.

Well essentially it'd be interesting to consider two options...

• This PR as-is.
• This PR, but also with backwards compatibility, and without changes to the test suite.

A useful thing to consider here is that any changes to the footprint of our test suite here is a good indicator of what changes our users would also need to make to their test suites. If we're hitting them with a big set of required changes in order to upgrade, then that's not fantastic. (Although there might or might not be good enough reasons why we're prefer to do that.)

I'm mostly saying all this in order to flag up that I think we'll want @tiangolo's perspective, and that we will need to carefully look at the cost of API changes, and how we can mitigate that.

• User perspective: we're only breaking their test suite, not their application (assuming they don't use testclient module on their application).
• Starlett's perspective: we're breaking our API, as testclient.TestClient is public.

We usually follow a "try to not add breaking changes" approach, even if we are on "zero version". Considering this, we should make it backwards compatible.

That being said... I'm going to list the set of breaking changes, so it can help us to make the best decision here:

• The methods ("delete", "get", "head", "options") doesn't accept the content, data, json and files parameters.
• allow_redirects is now follow_redirects.
• cookies parameter now is deprecated under the method calls (it should be used on the client instantiation).
• content_type send defaults to "text/plain" when sending file instead of empty string.
• data is now content.

[Kludex]

@tomchristie friendly follow-up ping

[tiangolo]

Awesome! 🚀

Thanks a lot @Kludex for the PR and for the short summary of breaking changes, that's super helpful.

Indeed, breaking changes are not great in general... 😅 .

Nevertheless, the intention has been to move from Requests to HTTPX from the beginning, I would like to do as much as possible to support this.

Some comments about the changes:

allow_redirects ✅

I don't expect a lot of people to have a lot of allow_redirects=False, so I wouldn't expect that migration to be too painful for users.

Body in GET and others ⁉

Starlette and FastAPI actually allow receiving content from GET requests, even if that's not auto documented in OpenAPI, and it's weird in general, it's still supported. I guess mainly because ElasticSearch does it, and to keep compatibility with it. There's a small mention about it in the info box here: https://fastapi.tiangolo.com/tutorial/body/

If the test client doesn't support it, it would mean that users can't test their weird apps. Which are weird, but they should be able to test them.

No Cookies ⁉

I think not being able to pass directly cookie values could be cumbersome, as I would expect some tests to send a cookie value directly from tests, to test authentication flows, independent from setting and retrieving the value. I've seen that and I think I've used that myself. 😬

Default content_type ❓

I'm not sure if this could break anything in FastAPI, but I think the behavior should probably be whatever is the same default behavior from browsers. Unless there's some specification that says that everything should be by default text/plain.

But I would go with whatever browsers do. And if it breaks FastAPI, I should fix it there in FastAPI. 😅 Because I would expect that the great majority of real-life/production clients would be browsers.

Data to Content ❓

This is one of those breaking changes that I guess could be a bit cumbersome but might be worth just going with it... like ripping a band-aid. 😅

Just because this is a design decision in HTTPX, it's quite visible, and we'll want to educate people to use it, instead of having both.

On top, content is a much better name for what it is than data. 😎

Maybe, if it's not too complex, we could have both with a deprecation warning for data for a while, what do you think?

---

Off topic note: Thanks for pinging me @tomchristie and @Kludex!

And thanks @Kludex for pinging me on another channel! I'm swamped in GitHub notifications and it's hard to get to the important stuff like this. 😬

[Kludex]

Thanks for your input @tiangolo ! :)

I'll wait for @tomchristie 's feedback, so we can decide how to continue here in order to have this PR merged.

[br3ndonland]

I didn't get the question 🤔

I asked because the Starlette TestClient is overriding the __enter__ method on the HTTPX Client, but not enforcing opened/closed state in the same way.

starlette/starlette/testclient.py

Line 711 in 3050e9e

def __enter__(self) -> "TestClient":

I don't have a specific problem with it in this context, just wanted to raise the question for discussion.

[tomchristie]

So... the largest impact of this change will be for FastAPI devs. They're the largest install base, they're using the test client, and the change will introduce breaking changes into their test suites and require extra work for their teams.

I guess we might(?) want to have a couple of shims in place in order to minimise the API changes...

• We could have a shim to continue allowing data=....
• We could have a shim to allow content in GET etc... requests.
• The content type change is probably absolutely fine as it is.
• I'm not sure we can easily shim the cookie change.

We could also consider assessing how much negative impact this PR would have by making a pull request to the FastAPI project that uses this branch for the TestClient, and determining how much work is required to change the test suite there.

Anyways, I don't know really - not super-keen on giving teams extra busywork, but yes it would be nicer to have switched to httpx. Perhaps @tiangolo ought to have the final say on if/when we're okay with merging this.

[tiangolo]

This looks great! I'm all in for this. 🚀

---

I just tested it locally in FastAPI, here's the PR with the changes needed after this goes in (and I upgrade the Starlette version pinned in FastAPI): fastapi/fastapi#5350

There are indeed some changes, but I think they are all sensible, and codebases that need to upgrade would have cleaner code after that.

I would think that the main changes I would expect would be:

• Changing data to content where needed
• Changing the style of form data from a list of tuples to a dict, which I think is much more cleaner and understandable
• Changing inline cookies with cookies at client creation

I wouldn't expect many to need to change follow_redirects because I don't think many would even use it.

I also don't expect many to need to send body payloads with GET, DELETE or others, just in corner cases, and they can achieve it as Tom described above (and as done in that FastAPI PR).

---

Also, for almost all of the points, I think except for the cookies in the client, the needed refactor could be automated with a LibCST codemod... but who would have the expertise to end up building that?! 😱 😉

---

I have a couple of minor comments. But this looks great to me. Awesome job @Kludex! 🙇 🍰 ☕

Also, thank you @tomchristie for having me in mind! 🙇 ☕

[michaeloliverx]

Nice job @Kludex

Also, for almost all of the points, I think except for the cookies in the client, the needed refactor could be automated with a LibCST codemod... but who would have the expertise to end up building that?! 😱 😉

This seems like a useful route to take and is quite common in the JavaScript ecosystem.

[Kludex]

Nice job @Kludex

Thanks!

Also, for almost all of the points, I think except for the cookies in the client, the needed refactor could be automated with a LibCST codemod... but who would have the expertise to end up building that?! 😱 😉

This seems like a useful route to take and is quite common in the JavaScript ecosystem.

If no one does it before me, I'll do it before the next release. 👍

[michaeloliverx]

Nice job @Kludex

Thanks!

Also, for almost all of the points, I think except for the cookies in the client, the needed refactor could be automated with a LibCST codemod... but who would have the expertise to end up building that?! 😱 😉

This seems like a useful route to take and is quite common in the JavaScript ecosystem.

If no one does it before me, I'll do it before the next release. 👍

I'd be interested in a blog post on AST transforms 😉

[tiangolo]

Also, for almost all of the points, I think except for the cookies in the client, the needed refactor could be automated with a LibCST codemod... but who would have the expertise to end up building that? 😱 😉

Hehe, I was actually joking with the "who would have the expertise", I knew @Kludex was the - right - person. 😎

I was just putting the "inception" idea seed. 😅

[Kludex]

From the changes:

1. The methods ("delete", "get", "head", "options") doesn't accept the content, data, json and files parameters.
2. allow_redirects is now follow_redirects.
3. cookies parameter now is deprecated under the method calls (it should be used on the client instantiation).
4. content_type send defaults to "text/plain" when sending file instead of empty string.
5. data is now content.

I've implemented 1, 2, and 5. I need to add more tests, and fix a scenario, but in progress.

3. may be tricky or even not needed, because people usually have a TestClient fixture.
4. Not sure if something needs to be done there? 🤔

Code in https://github.com/Kludex/bump-testclient.

EDIT: I thought of a way to do 3. 🤔

[gyKa]

Are there any technical reasons why starlette is starting to use httpx instead of requests?