Handling back pressure with services

[krishna-kashyap]

Hey,

Possibly I don't understand the complexity completely although I found the thing with service needed to be ready before a request not playing very with routes based on request not completely right.

Here's what I was thinking could be a possible way, there's a router service which routes to other services and needs to be ready for the server to start serving requests. And if the router service needs DB or any other resources then waiting on that service before starting servicing request makes sense. However for the routed services, could they not still be polled on each request and if unavailable (possibly because it serves only certain number of concurrent requests?) return a 503 which tells the client to retry again?

Also there was a MakeService initially in Hyper with similar facility of poll and creating new service instance and felt that played well.with actually load balancing and such but then haven't seen that being actually used as such.

[davidpdrsn]

I'm not quite sure I follow 😕

Since Service::poll_ready doesn't have access to the request axum would have to poll the entire router tree every time to ensure that everything is ready, since we don't know where an eventual request will end up. This is similar Steer from tower. Because most services don't care about backpressure I made the decision for axum just simplify drive readiness as part of Service::call. Its not ideal for sure but I don't know of a better solution.

Note that you can still use ConcurrencyLimit and LoadShed with axum. You just need to handle the errors from LoadShed with handle_error and convert them to a 503.

If parts of your system do care about backpressure the recommendation is to wrap your entire app in a middleware and then drive readiness from there. Similarly to whats shown here.

[krishna-kashyap]

Sorry I wasn't clear. I meant, when we are routing to services, we already have a request and that service can respond if it is okay to handle a request, that is being routed it? Or am I mistaken that isn't how it is supposed to work?

I was envisioning something like this, for sake of naming I call them pico service, let us say it handles just one route, possibly with a capture param meaning some variants but essentially doing the same function(s). In this case, could we not check on poll before calling the call with the request and on not ready send back a 503?

I maybe repeating myself and it might be my poor understanding of the Services design, but I'm understanding there to be multiple services in hierarchical manner, router service on top which delegates the requests to services further lower. In this case the router needs to be ready, but not any of the other services, in fact if we can create services only when needed, that might be great too and helps in cleanly using the service abstraction to handle back pressure.

[davidpdrsn]

I meant, when we are routing to services, we already have a request and that service can respond if it is okay to handle a request, that is being routed it?

The purpose of backpressure in tower::Service to drive readiness before receiving a request, or before allocating space for it rather. This is why poll_ready only receives a std::task::Context and not a request. If we have to receive a request before being able to drive readiness then thats different from how poll_ready works.

I was envisioning something like this, for sake of naming I call them pico service, let us say it handles just one route, possibly with a capture param meaning some variants but essentially doing the same function(s). In this case, could we not check on poll before calling the call with the request and on not ready send back a 503?

The problem is that axum cannot know whether your router contains just one endpoint/service, or many. So we cannot adapt backpressure handling based on that. You can still write your own Service implementation that acquires resources in poll_ready, and optionally fails, and then map those errors to 503 using LoadShed and handle_error.

You can also write an extractor (by implementing FromRequest) that attempts to acquire a resource and returns 503 if there is more capacity. Thats probably the simplest solution, but its different from backpressure propagation at least the way tower::Service defines it.

This guide explains backpressure in more details.

[dartheian]

Hi, the link provided in this discussion points to an old version of axum. I can't find more up to date documentation on the argument. Are the information regarding back pressure contained in this discussion still relevant? Are there any more up to date docs?

EDIT: found it -> routing-to-servicesmiddleware-and-backpressure