Support upstream HTTP connection timeouts

[guydc]

Description:
Envoy supports a variety of upstream connection timeouts that impact persistent HTTP connections establishment and lifecycle:

• Cluster connect timeout: timeout for establishing a TCP connection. If TLS is used, the timeout also applies to TLS session setup. The Envoy default is 5s. The Envoy Gateway default is 10s.
• HTTP proxy idle timeout: Idle connections (without active requests) are closed after this time. Envoy default is 60m.
• HTTP proxy max connection duration: When the max connection duration is reached, HTTP persistent connections are closed (irrespective of the connection's idleness). The default value is unlimited.

Envoy Gateway should support configuration of these timeouts for upstream connections. Below are some examples and use cases:

• Cluster connect timeout:
  • Envoy Gateway default values prioritize security. However, other proxies like nginx use 60s as a default for connect timeout, indicating that longer connection establishment times are possible.
  • Users may need to increase this value. For example, when the latency to the backend is significant, the TCP/TLS handshake may not finish in time.
• HTTP Idle timeout:
  • Users may want to change idle timeout settings to better reflect their performance vs. resource consumption needs.
  • It could be necessary to align the Envoy idle timeout with the backend idle timeout, if the backend does not send the Connection: close header before closing the connection. Otherwise, a race condition can occur where envoy will use a half-closed connection and fail with a reset. For example, see here: Envoy intermittently responds with 503 UC (upstream_reset_before_response_started{connection_termination}) envoy#14981.
• HTTP max connection duration:
  • While the default value is not disruptive, Envoy docs mention that setting a maximum age is beneficial for cleanup of connections to targets that are no longer discoverable with DNS.