User data transfers with proxy_buffering spills to disk and fills up small root disks

[jflandry]

Users transferring larger amounts of data can lead to full root disks due to the default nginx proxy_buffering and proxy_max_temp_file_size configuration.

Setting proxy_max_temp_file_size = 0; in jupyterhub.conf.epp under location / { should be an acceptable solution.

https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffering
https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_max_temp_file_size

Syntax: proxy_buffering on | off;
Default:
proxy_buffering on;
Context: http, server, location
Enables or disables buffering of responses from the proxied server.

When buffering is enabled, nginx receives a response from the proxied server as soon as possible, saving it into the buffers set by the proxy_buffer_size and proxy_buffers directives. If the whole response does not fit into memory, a part of it can be saved to a temporary file on the disk. Writing to temporary files is controlled by the proxy_max_temp_file_size and proxy_temp_file_write_size directives.

When buffering is disabled, the response is passed to a client synchronously, immediately as it is received. nginx will not try to read the whole response from the proxied server. The maximum size of the data that nginx can receive from the server at a time is set by the proxy_buffer_size directive.

---

Syntax: proxy_max_temp_file_size size;
Default:
proxy_max_temp_file_size 1024m;
Context: http, server, location
When buffering of responses from the proxied server is enabled, and the whole response does not fit into the buffers set by the proxy_buffer_size and proxy_buffers directives, a part of the response can be saved to a temporary file. This directive sets the maximum size of the temporary file. The size of data written to the temporary file at a time is set by the proxy_temp_file_write_size directive.

The zero value disables buffering of responses to temporary files.

---

[cmd-ntrf]

Excellent write-up and thanks for reporting.

JupyterHub documentation recommends turning off the proxy buffering. I'll update the NGINX config to match the most recent recommendation from them. https://jupyterhub.readthedocs.io/en/stable/howto/configuration/config-proxy.html