Tuning backpressure help

[apenney]

Hi!

We're looking at switching over from Gunicorn to Granian but we ran into a weird performance issue. We're running Django 4.x, and in our development environment baseline latency has gone up from 6ms to 60ms.

This was weird, so I started digging into our apm traces and found one of our middlewares (something custom) has gone from an average of 6ms to 60ms.

It's got some unexciting code that looks like:

    def process_request(self, request):
        """
        Adds person & company to request for easy access
        """
        request.person = None
        request.company = None

        user = request.user
        if user.is_authenticated:
            person = user.person
            company = person.company

            request.person = person
            request.company = company

At this point I couldn't find any obvious reasons for issues, so I started to dig into blocking threads, stuff like that, trying to find anything to tune. From my understanding for WSGI the best option for us is:

• workers = (we've tried 4 and 8)
• workers_lifetime = 3600 (so we don't build up memory leaks)
• threads = default (1)

At that point I would have had backpressure=128 by the default calculation, so I then bumped this to 512 to see if it would help but nothing changed about the latency profile. (We don't use a db connection pool, we just let django do it's thing, so I couldn't scale them based on that).

Questions:

• Is there anything I could look at in a system running gunicorn to get a realistic value for backpressure?
• Is there any reason to touch backlog?
• Is there any other options here? Should I try threading mode instead of workers?

The latency hike was pretty noticeable on the charts so I'm hesitant to try and roll this up into environments with more traffic until I better understand what's going on here. Any suggestions would be gratefully appreciated!

[apenney]

We've spent the day trying all sorts of things, but we still have issues.

We bumped up backpressure, that blew up our database (we ran out of connections). We tried less workers, more workers, all sorts of inbetween configurations.

I think the hardest part is not knowing if granian is contributing to the latency at this point or if we're just wasting our time tuning the wrong things. Are there any metrics or logs (debug logs?) that would help us determine if the io threads/backpressure is too low?

My next plan is to get some continuous profiler running so I can at least see what's going on under the hood.

[gi0baro]

I can't think of anything specific which can increase latency 10x compared to gUnicorn (in fact, every benchmark suggests otherwise).

To me sounds like you're overloading the database, so instead of increasing backpressure I would go the other way around and pick very low numbers (4-16 range).

If that doesn't help, the other possible way would be to limit the blocking threads to a low number as well, but I would first try with a low backpressure.

[apenney]

I’ll give it a shot. From what we could see the database wasn’t really doing at the time (based on trying to decipher apm traces), but it’s also hard to tell for sure without profiling. Even in super low traffic scenarios (just health checks) I still saw really high latency.

Would it be worth us exploring threading instead of workers?

As we’re in kubernetes, I also need to explore going back to one worker per pod and more pods instead of the 4-8 workers * 2 pods we’re trying right now.

[gi0baro]

The threading mode wouldn't probably make a difference, as it involves the Rust side of things only; the Python code interaction would still work in the same way.

And yes, in k8s I generally advise to have 1 worker per pod and scale using replicas, rather than having few fat pods.

If you want Granian to mimic gunicorn, you can kinda forget about backpressure and threat blocking threads as gunicorn threads. Eg: setting blocking threads to 1 will make granian working as the sync gunicorn implementation making requests processing serial.