Nginx fastcgi buffers: set better default

[swalkinshaw]

The value of "8 8k" results in a 64KB buffer size. This buffer size should be slightly bigger than the average response size.

The higher this value is over your average response size, the more memory is wasted on each request.

Reference: #281

[swalkinshaw]

Anyone who wants to help on this PR can run this command and paste the results:

Note: you may need to adjust the name/path of access.log.

echo $(( `awk '($9 ~ /200/)' access.log | awk '{print $10}' | awk '{s+=$1} END {print s}'` / `awk '($9 ~ /200/)' access.log  | wc -l` ))

It will output the average size in bytes of your responses.

[austinpray]

On a couple installs:

• 3508
• 5425
• 28391
• 26684

64KB looks about right

[BrandonShutter]

24757

[swalkinshaw]

@BrandonShutter fyi going back to the original Discourse thread you had set 32 32k as the value. This actual results in 32 * 32k = 1024KB = 1MB which is huge.

[BrandonShutter]

@swalkinshaw I've since downsized to 64kb. Thanks!

[austinpray]

Add a comment with a link to https://rtcamp.com/tutorials/nginx/tweaking-fastcgi-buffers/ ?

[swalkinshaw]

I don't like that URL. It says to do 32 32k which is 1MB. They don't see to understand it multiplies them.

Should add this one: https://gist.github.com/magnetikonline/11312172

[rohmann]

4788
3964

[retlehs]

from production server logs:

28576
42900
44937

[retlehs]

another prod server:

23478

[kalenjohnson]

So I just checked a prod server where I'm getting this: #315

But access.log.1 is 74886

[brocheafoin]

@swalkinshaw :

Anyone who wants to help on this PR can run this command and paste the results

As is mentioned in https://gist.github.com/magnetikonline/11312172, which you referenced later:

Note these are going to report on all access requests with a 200 return code, you might want to split out just FastCGI requests into a seperate Nginx access log for reporting

FWIW, here's the result on a site I'm currently working on:

6520

On said site, some requests were failing with: upstream sent too big header while reading response header from upstream until I set the fastcgi_buffers to 8 8k.

[swalkinshaw]

@brocheafoin thanks! Interesting results.

According to that gist you may actually want to set fastcgi_buffer_size which is specifically for HTTP header size. Your error message leads me to believe it's only talking about the header size and not the entire response size.

[brocheafoin]

@swalkinshaw That's what I tried at first, but I got this error:

"fastcgi_busy_buffers_size" must be less than the size of all "fastcgi_buffers" minus one buffer in /etc/nginx/nginx.conf:128

so I also changed fastcgi_buffers and things worked. I then completely removed the fastcgi_buffer_size directive and only kept fastcgi_buffers 8 8k; and things kept working. I'm a bit over my head with this kind of configuration, but I can run various tests if you think it will help.

[swalkinshaw]

@brocheafoin I don't know that much about it so as long as it's fixed on your end 👍

[swalkinshaw]

8 8k seems like a safe default option so going to merge as is.

[BrandonShutter]

I know this is closed, by why are we not adding the ability to set the buffer_size?

In my main.yml I have nginx_fastcgi_buffer_size: 32k and in my nginx.conf i have fastcgi_buffer_size {{ nginx_fastcgi_buffer_size }};