Reducing origin server requests

[krizhanovsky]

Currently thundering herd problem is possible when many clients try to request the same resource, in this case Tempesta generates as many request to upstream as it receives from all the clients. Rather it should accurately track requests on the fly and made only one active request for upstream.

All the same concurrent requests must be backloged and sent to a server if currently sent request fails. This is different from Nginx's proxy_cache_lock_age , but it seems we can implement proxy_cache_use_stale update Nginx strategy. Similar feature in Varnish coalesces requests also see nginx-limit-upstream module.

The cache should provide automatic refreshing by using TDB trigger callback for DELETE (see #515).

[krizhanovsky]

The implementation must handle both the cases for absent and invalid cache entries. We need a generic protection against tundering herd, when many equal requests efficiently passes through the cache to a backend server. The only one request must go to a backend server, while others must be postponed.

Technically it could be special in-progress cache entry with a list of requests generating the entry. E.g. if there is no cache entry, then we just create a new one and add the current request to it's list and forward the request to a backend. If following requests find the cache entry, then they just add themelves to the cache entry list and a current routine returns (probably with TFW_POSTPONE or TFW_PASS code). When a response comes, we traverse the list and send the response for the each request (we should just increase page reference counters instead of real data copying).

Since we need to modify cache entries, the issue depends on #788

[vankoven]

Addition to comment above:
Response may not come in time and an error response may be generated. This may happen if the request was evicted for some reason: request timeout, retry timeout or even client disconnection. In this case the error response must be send to client (if applied), other responses for the same cache entry must be evicted if stale, then the latest request must be forwarded to backend server. If all request was evicted for the cache entry, cache entry should be purged.

[krizhanovsky]

Recently we observed not enough memory crash on TDB side due to this issue