Version 1.0

[tomchristie]

Update: Everything we need to consider ought to now be milestoned... https://github.com/encode/httpx/milestone/1

---

API Reference

Here's a high-level API reference...

Helper functions

request, stream, get, options, head, post, put, patch, delete

Clients

Client, AsyncClient

Models

Response, Request, URL, QueryParams, Headers, Cookies

Configuration

Limits, Proxy, Timeout

Authentication

Auth, BasicAuth, DigestAuth

Transports

ASGITransport, WSGITransport

Status Codes

codes

---

Exceptions

• HTTPError
  • RequestError
    • TransportError
      • TimeoutException
        • ConnectTimeout
        • ReadTimeout
        • WriteTimeout
        • PoolTimeout
      • NetworkError
        • ConnectError
        • ReadError
        • WriteError
        • CloseError
      • ProtocolError
        • LocalProtocolError
        • RemoteProtocolError
      • UnsupportedProtocol
      • ProxyError
    • DecodingError
    • TooManyRedirects
  • HTTPStatusError
• NotRedirectResponse
• CookieConflict
• StreamError
  • StreamConsumed
  • ResponseNotRead
  • RequestNotRead
  • ResponseClosed

---

Dependencies

• httpcore
  • h11
  • sniffio
• rfc3986
  • idna
• certifi

Optionals: pip install httpx[http2], pip install httpx[brotli]

[florimondmanca]

I’d like to add a discussion point about our position wrt type hints would be. There’s been a couple of issues asking for them to be public API mainly for the use of third party library authors. I don’t have a strong position on this yet, but wondering if type hints (and more generally third party lib authoring experience, although that’s pretty broad) should be in our scope for 1.0?

[tomchristie]

Good call yup, we need to discuss that for sure!

[yeraydiazdiaz]

We have a version 1.0 milestone. Should we formalize these points into issues and add them?

[tomchristie]

The proxy DEFAULT_MODE discussion needs to be part of this too.

[tomchristie]

@yeraydiazdiaz Sure thing, yup.

[pikeas]

Having recently used httpx, I was very surprised that there's no rate-limiting (requests per second, not just max connections) built in. This greatly reduces the utility of the async mode.

[tomchristie]

Okay. Sure we can have a discussion about that, tho it's not strictly a 1.0 blocker.

[florimondmanca]

Something else to potentially consider for 1.0 that came to my mind: #302, aka dropping certifi in favor of the system defaults. I have a different approach in mind that what was originally suggested in the issue, will comment there to see if it makes sense.

[bwright1558]

I've had my eyes on this repository for a while and it looks very promising. The only thing preventing me from using this library is the lack of support for socks5 proxies. Gotta have this for connecting to Tor. It's also a lot easier to spin up a socks proxy server than it is an http proxy server (in my opinion).
There's already an open issue: #203. There's also a PR in httpcore: encode/httpcore#51. Would be nice to see this in a 1.0 release.

[florimondmanca]

@bwright1558 From my POV, SOCKS proxy support is a nice-to-have, but technically not a blocker for 1.0 (we can probably add it later as an incremental new feature, without having to break any existing API).

Note that there's been loads of work by Yeray and @sethmlarson to pull off a sans-io implementation, and as you mentioned Yeray started the integration within httpcore (which is indeed almost only additive). If you'd like to help out I'm sure Yeray will appreciate early-stage user feedback there, e.g. see if the API makes sense, etc. :-)

[yeraydiazdiaz]

I agree with @florimondmanca in that SOCKS support is not a blocker for 1.0, though I certainly see it coming sooner rather than later 🤞

Personally I think the two most requested features for 1.0 are:

• Built-in retries, for which Florimond already opened Retries for connection failures #784)
• Built-in throttling, which is partially addressed by the pool limits but has come up in the chat and Florimond suggested using https://github.com/florimondmanca/aiometer.

To me the latter feels more of a nice-to-have for 1.0, it could even be argued that it's better to leave it as a separate library completely.

[florimondmanca]

I think those are two interesting areas, but are they blockers for 1.0?

I’m still not super convinced we need to have retries built in (libraries like tenacity solve this problem brilliantly). But if we do want some basic functionality then I’d definitely make the case for making retries opt in (which makes it easier to bring in higher control via eg tenacity), in which case this would be a non breaking addition in all cases, so not a 1.0 blocker.

As for throttling, it seems we currently have expectations set wrong wrt pool limits. A lot of people seem to expect those to behave like throttling, when it’s really more about resource limiting. I guess we could address this with some docs revamp about pool limits (I remember Tom drafted a PR for this), and then perhaps recommend focused solutions like aiometer more prominently - at least as a first approach. (FWIW aiometer already has an HTTPX usage example in its README.) Like retries, I don’t see why we wouldn’t be able to bring this in incrementally (throttling should be opt in too), so not a 1.0 blocker in my mind.

[StephenBrown2]

Probably not a strong argument, but retries are "built in" to Requests due to it relying on urllib3 for that, so adding it would allow a smoother transition from requests.

But, I will also concede it's not a requirement for meeting the HTTP client spec (aside from the Retry-After header being a thing...)

[hugovk]

Not directly an httpx issue, but a heads-up that the httpx mock library RESPX doesn't yet support httpx 0.13:

• Require httpx 0.13 lundberg/respx#54

[P0oOOOo0YA]

So you're going to release 1.0 without socks v4/5 support? 👎

[tomchristie]

So you're going to release 1.0 without socks v4/5 support? 👎

Not acceptable. You need to rethink how you approach folks in the community - that’s a massively entitled and disrespectful way to approach your first interaction with a project.

Here’s a better approach:

Q: “Hi folks! Is socks v4/5 a priority? Does that come before or after 1.0?”

A: “1.0 is about API stability. Socks support might come either before or after that, but it’s not a blocker for us indicating that the project is API stable.”

[P0oOOOo0YA]

A: “1.0 is about API stability.

Hi folks! Sorry for not so well first impact but I think 1.0 is about minimum functionality as well. Maybe you're right and i'm a perfectionist to consider supporting proxies (both HTTP(s) & socks v4/5) the minimum requirement of an HTTP client. Anyway congratulations

[hugovk]

Not directly an httpx issue, but a heads-up that the httpx mock library RESPX doesn't yet support httpx 0.13:

• lundberg/respx#54

The newly-released RESPX 0.11 now supports HTTPX 0.13 🎉

[pawamoy]

Hello everyone, how is the 1.0 going 🙂 ? I would like to use HTTPX at work but my client requires software to be in a somewhat "stable" status, and the first note of the first page of the docs indicates that HTTPX should still be considered beta, so right now they won't have it.

Is there a way I could help you releasing version 1.0? Maybe start working on the issues in the 1.0 milestone? I'd appreciate some guidance as I didn't contribute to this project yet, and don't want to be a hindrance to the development 🙂
I'm also not particularly knowledgeable in HTTP protocol matters 😅 Let me know if there's anything I can help with 🙂

[florimondmanca]

Hi @pawamoy,

I understand that the currently advertised beta status could be a hindrance to spreading HTTPX in environments that require stability (I personally would like to start replacing Requests with HTTPX at work 😉), but for now that's how things are. HTTPX will will promoted to "stable" with 1.0.

I hope you'll understand that I'm personally sort of pushing back on asks like this since quality of design requires lots of careful thinking, and that means time. We shouldn't rush things.

So I personally can't even say when 1.0 will be. "Somewhere mid/late 2020" would be my fairest guess for now. We should be releasing 0.14 soon, and that's planned to be our last minor before 1.0.

If you'd like to help, yup, feel free to look over items marked for the 1.0 milestone, since those are required before reaching 1.0. In terms of guidance well, give yourself some time to navigate and understand the code at a high level, then dig deeper as you need. :-) Our contributing guide should help you get up and running as far as tooling and development workflow is concerned.

Cheers!

[pawamoy]

@florimondmanca of course, I was not asking for a 1.0 release just for the sake of it, but rather if I could help to implement the things that are required for the 1.0 release. I totally understand the need for careful thinking, and I don't want you to rush things! Thanks for your answer, I'll then see what I can do 🙂

[aalvrz]

I do not have any requirement to use httpx at work, but I have been watching this project for some time and I must say I am really amazed by the amount of work, discussion and thought put into everything in it ❤️ . I am sure the 1.0 release will be glorious 😃 🚀

[nkitsaini]

Should we update the README.md to reflect new estimate for 1.0 release ?

[florimondmanca]

@nkitsaini "Some time in 2021"? I guess we could aim for that. 😄 Feel free to send a PR if you're up for it!

[manatlan]

I vote for SOCKS proxy support too!
(aiohttp & requests support them ... if it's not here : I doubt that it can be used/useful in the pro world (others programs/tools will not switch to httpx without that (it's my case))
I say that, I say nothing ... I will continue to use aiohttp til httpx supports it ;-(

[emil45]

I vote for SOCKS proxy support too!
(aiohttp & requests support them ... if it's not here : I doubt that it can be used/useful in the pro world (others programs/tools will not switch to httpx without that (it's my case))
I say that, I say nothing ... I will continue to use aiohttp til httpx supports it ;-(

+1

[tomchristie]

I'm very keen on adding SOCK proxy support, and I have been looking at it as part of the revised httpcore work I'm doing. However, there's no need for us that work to block declaring our library as API stable, with a 1.0 release.

I've just raised two discussions, which I'm considering the final blockers for a 1.0 release...

• Do we really need the proxy "mode" argument? #1783
• Reconsidering auto-redirects? #1785

[tomchristie]

@urbanogt - Thanks, yes I'd also like to see a 1.0 release.

Happy to help!

Funding or code collaboration are both very much welcomed.
If you'd like to help with the later, keep your eyes open on the repo and jump in when you can. 😎

[tomchristie]

If pointers are helpful... taking a look at #2788 would be a good one to get resolved.

[tomchristie]

Okay, I'm going to just sketch out the bits that I've had in the back of my mind around this...

---

Bugs that I might currently consider to be 1.0 blockers.

• Gracefully handling async cancellations... httpcore/#785 and httpcore/#756
• ASGI raw_path scope key should not include the query string portion. #2810
• DigestAuthentication not working #2788

Actually only one of those is API changing, so they don't necessarily need to be blockers. Can just be regular minor point bugfix releases. But perhaps they're significant enough to be treated as 1.0 blockers anyways?

Documentation refresh.

Not technically a 1.0 blocker in any sense, but if we end up with lots of exposure from the release it's going to be the first thing newcomers see so? We've clearly got scope for a really nice docs overhaul.

Minimising project dependancies.

Our default dependancies are...

• certifi, idna, sniffio, httpcore, h11, anyio.

We could potentially have a project with a really light default dependancy footprint...

• certifi, httpcore, h11.

Personally I find this a really attractive option, and can talk it through in detail if needed.

API finessing.

There are a couple of areas where our API could fit together more neatly with some changes.

• The proxies=... API is somewhat overloaded. There are neater approaches that we could take if we switched to a single proxy=... API, and use the mounts={...} for more complex cases.
• The SSL configuration using verify/cert/trust_env is somewhat overloaded. I can see that a neater approach would be a single ssl_context=... parameter.
• There's a couple of very minor bits of the URL API that could do with tweaking. .raw_path would probably be better named as .target. Also the .raw property is noted as "to be deprecated".
• There's some bits of client config that are inconsistent with others. mounts superseeds transport which superseeds proxy, verify, cert, http1, http2, limits, app. I think there's bits of neatening up we could do there. For example, dropping app in favour of explicitly requiring transport=httpx.WSGITransport() / transport=httpx.ASGITransport().

We clear don't want to add code churn for our userbase. Perhaps we leave these kinds of changes for a future 2.0 version which also has a really nice gentle deprecation path.

---

Probably none of those are absolutes, and we're in a position for our next releases to be a 1.0 of httpcore and a 1.0 of httpx if we chose to do so.

Thoughts, @encode/maintainers?

[Kludex]

Can we define a versioning policy for the organization, and specify what versions are we going to give support?

Also, given that the previous "1.0 path" didn't go as planed, do you think it would be helpful to be specific on what are the steps to achieve 1.0, in a more objective way?

Since I'm not that involved in HTTPX, I don't have opinions about the feature set.

---

Not technically a 1.0 blocker in any sense, but if we end up with lots of exposure from the release it's going to be the first thing newcomers see so? We've clearly got scope for a really nice docs overhaul.

Completely unrelated to what you said here, but maybe we can add the sponsors in the homepage on the documentation?

[tomchristie]

but maybe we can add the sponsors in the homepage on the documentation?

That's a smart & valid 1.0 request.

[ahopkins]

We could potentially have a project with a really light default dependancy footprint...

Minimizing dependencies is of course always a win. Especially when you consider some that are not strictly necessary (looking at you anyio). Of course, it does make implementation more complicated and there is the maintenance tradeoff. No strong opinions on this, but I would certainly be interested to hear more about what you had in mind to achieve this.

[karpetrosyan]

Mounts and ssl_context makes perfect sense.

• The proxies=... API is somewhat overloaded. There are neater approaches that we could take if we switched to a single proxy=... API, and use the mounts={...} for more complex cases.
• The SSL configuration using verify/cert/trust_env is somewhat overloaded. I can see that a neater approach would be a single ssl_context=... parameter.
• There's a couple of very minor bits of the URL API that could do with tweaking. .raw_path would probably be better named as .target. Also the .raw property is noted as "to be deprecated".

I believe that a documentation refresh is critical for the 1.0.0 release. Motivated newcomers should see beautiful documentation.

Also, I think we should have 0.26.0 before 1.0.0, so that we can test the changes before releasing 1.0.0.

[astrojuanlu]

I'm just a bystander here but I want to spot a chicken-and-egg problem:

Many shops have stated that they're waiting on httpx 1.0 to adopt it. With adoption could (potentially) come more funding and contributors. Without such funding and contributors, getting to 1.0 might get more difficult.

So, if you want to help httpx in the short term, you probably want to help with some of the bugs @tomchristie has stated, or otherwise refrain from making the scope of the 1.0 release bigger.

[samuelcolvin]

Pydantic would be happy to sponsor, we love using httpx (as well as all the other things we've all learnt from @tomchristie).

I'll set up sponsorship tomorrow.

[tomchristie]

That's immensely generous of you, thanks @samuelcolvin ❤️

[tomchristie]

Can we define a versioning policy

We should make it clear that we follow SEMVER.

We should also recommend that users pin their requirements to at least a major version range.

Informally we can follow a stronger-than-SEMVER policy. If we introduce an incompatible API change in a major version, then we can first have a major version bump that issues a warning with the old usage, but does not break. We'll only then remove the old usage after a second major version bump.

Deprecating formal support of older Python versions should require a version bump.

---

My concise "where are we at" is... Let's release a 1.0 version with a neat SEMVER policy immenently.

Following up on #947 (comment) more specifically...

• Bugs that I might currently consider to be 1.0 blockers. Anything we can deal with as a patch release shouldn't be a blocker.
  • The ASGI raw_path is arguably an API change rather than a plain bug. (Ensure that ASGI 'raw_path' does not include query component of URL. #2999)
  • I'd like to review a couple of bits of our URL handling before 1.0. Based on incoming discussions there's a couple of behavioral decisions to make there that wouldn't have any graceful upgrade path.
• Documentation refresh. I don't think we should block a 1.0 release on a major refresh. There's a significant amount of work here, so I'd suggest we'd be better off cutting our 1.0 release and making our versioning policy clear, and then following up with a later docs refresh. A few minor changes would be sufficient...
  • Adding our versioning policy
  • Notes about sponsorship
  • Adding a docs page for our command line client
  • A "why choose httpx" section in the README/docs.
  • Our project tagline could be smarter.
• API finessing. I don't think we should block a 1.0 release on this (we don't want to rush any design here). However, we should address any issues that would prevent later graceful upgrades. Most of the API finessing suggestions have graceful deprecation paths that we could follow if we wanted to adopt them. The one bit that I think needs dealing with upfront is removing any defaults for env-based configuration. I'll expand on this in due course.
  • Ensuring env-based configuration is off by default.
• Minimising project dependancies. This seems worthwhile to me.
  • httpcore version 1.0.
  • Make async dependencies optional. #2858
  • Make idna support optional.

I think once we've got that done we can move to a time-based release schedule, improve our documentation, focus on how we deal with sponsorships, and iteratively improve on the project at a pace that isn't rushed.

[Kludex]

Deprecating formal support of older Python versions should require a version bump.

"version bump" here is not clear if you mean minor or major. By context, it looks like you mean a major?

Informally we can follow a stronger-than-SEMVER policy. If we introduce an incompatible API change in a major version, then we can first have a major version bump that issues a warning with the old usage, but does not break. We'll only then remove the old usage after a second major version bump.

It would be nice if you check https://github.com/encode/starlette/pull/1888/files#diff-8643a6bd412b94d3754ee992dec7139d2db1d1e49902e02584f563f12490f71b , so we can align those thoughts between the encode projects. 🙏

I don't think we should block a 1.0 release on a major refresh. There's a significant amount of work here, so I'd suggest we'd be better off cutting our 1.0 release and making our versioning policy clear, and then following up with a later docs refresh. A few minor changes would be sufficient...

It shouldn't be a blocker, clearly. But... Are we in a hurry?

[tomchristie]

"version bump" here is not clear if you mean minor or major. By context, it looks like you mean a major?

That was my intent here, yup. I'm agnostic on it tho. Maybe a simple policy for both starlette and httpx would be "Older Python versions will always continue to be supported at least until their EOL date. Python versions will only be dropped in either a new major or minor release, and will not be dropped in patch releases." (?)

But... Are we in a hurry?

No. It's just a trade-off we're making if we're pushing back on the timing. A suggestion would be that we time-box this. For instance, from here we could very reasonably commit to releasing 1.0 no later than the end of October.

[Kludex]

I'm afraid the docs never reaching the state mentioned here if we release 1.0 without them.

[rattrayalex]

Hey folks – is there a preliminary/draft migration guide that I could look at as a library maintainer, to ensure we're prepared to support httpx 1.0 as soon as it is ready, and to future-proof the APIs we are designing around httpx?

[tomchristie]

@alex-oleshkevich - No, not currently. The CHANGELOG.md indicates that we don't currently have any breaking API changes, so it depends on what we decide should make the cut for 1.0.

[tomchristie]

So, regarding that API finessing point above... there's one whole slice of work in particular that we could consider for a sharper, simplified but just-as-capable 1.0 release. Let's talk about it...

Existing Transport Configuration

If we consider all the configuration that effects how the transport is instantiated here's what we have...

• proxy (And the deprecated proxies)
• verify
• cert
• http1
• http2
• limits
• trust_env - Effects the SSL config, and auto-proxy configuration.

...and...

• app - If this is used then a WSGITransport or ASGITransport instance is used, and none of the transport options apply.

...and..

• transport - If this is used then it overrides all the transport config options, instead the configuration is directly on the transport class.
• mounts - If this is used then it overrides transport and all the transport config options, and instead multiple transport classes are created.

...and...

• uds - Only available directly on HTTPTransport()
• local_address - Only available directly on HTTPTransport()
• retries - Only available directly on HTTPTransport()
• socket_options - Only available directly on HTTPTransport()

...and...

by the way your transport config also depends on various environment variables that effect the SSL config, and proxy routing.

Ultimately this makes it challenging for users to have a clear mental model of how the transport is being configured, and which options clash. (For example... #1635)

---

Transport Cleanup

Here's how that same functionality could instead look...

• ssl_context = httpx.SSLContext() Add httpx.SSLContext configuration. #3022
• proxy = httpx.Proxy() Add the 'proxy' parameter and deprecate 'proxies'. #2879
• version = httpx.Version("HTTP/1.1") Add httpx.Version configuration. #3051
• limits = httpx.Limits() - Unchanged.
• network_options = httpx.NetworkOptions() - Add httpx.NetworkOptions configuration. #3052

And also...

• transport = httpx.HTTPTransport(...), or transport = ASGITransport(), or transport = WSGITransport().

With a behavior of raise an error if ssl_context or proxy or version is specified, and transport is also specified.

We'd deprecate app = in favor of the requiring the explicit WSGI/ASGI transport style. It's more clear around exposing what's actually happening, an removes an unnecessary point of configuration that clashes with other options. #3050

I think we can also replace mounts=... by having a transport = httpx.Mounts() case for the routing configurations. Ie. routing is handled by a transport if you want that, rather than in the client itself. That'll help since we'll no longer have a transport = .../mounts=... conflict, and we'll be able to expose a client.transport attribute.

(This last one needs a little more thinking about than the others, which are more straightforward.)

---

We'd also need to be refreshing the docs alongside making these changes, and making sure that everything fits together more obviously and neatly, and that users have a gentle set of pointers around bits of upgraded API. I can see how to approach this all in a neatly incremental way, but probably we need to take the design call on the big picture.

There's other more minor points of API tweaking that we might consider for a 1.0, but this is the big "wouldn't it be better if?" option.

We don't have to do any other this for a 1.0 release, but it'd provide a more coherent httpx if we did choose to do so.

Thoughts?

[tomchristie]

@encode/maintainers - So... In order to press on with this I think? we need to start with #3049.

The proposed changes really need to include their associated docs changes in order to be properly reviewable, and making those against the restructured docs will be more clear.

If we're not comfortable with committing ourselves immediately we could choose to make these changes against a 1.0-release branch, rather than directly against master?