Overload Manager: allow certain requests even when envoy.overload_actions.stop_accepting_requests is triggered

[ysun-square]

Title: being able to skip certain requests when envoy.overload_actions.stop_accepting_requests is triggered in overload manager

Description:
We want to be able to configure overload manager to always let certain requests pass through either based on headers or paths when envoy.overload_actions.stop_accepting_requests is triggered.

Our use case is we are considering to always pass through F5 load balancer's health check traffic. One incident we encountered is large nodes of our envoy cluster behind F5 are overloaded. Overload manager in these nodes failed F5 health check which cause a drastic increase of load in healthy nodes and cascading the failure to entire cluster.

[daixiang0]

The title mislead me :(

[KBaichoo]

Is the F5 load balancer health check traffic to the Envoy or through the Envoy to a backend? If it's to the Envoy itself, how are you getting the health check. I know of #18678 which opted out admin interface from global_downstream_max_connections.

[ysun-square]

Is the F5 load balancer health check traffic to the Envoy or through the Envoy to a backend? If it's to the Envoy itself, how are you getting the health check. I know of #18678 which opted out admin interface from global_downstream_max_connections.

thanks for the reply. Health check traffic is sent directly to Envoy currently. We created a special local_cluster in one https listener which proxies the traffic to /ready in admin interface.

[KBaichoo]

I wonder if you invoked the admin directly, whether you'd get affected by Overload Manager:

envoy/source/server/admin/admin.cc

Line 481 in 83f0506

Http::Code AdminImpl::request(absl::string_view path_and_query, absl::string_view method,

[ysun-square]

if invoked the admin directly, it will not be affected by Overload Manager. But F5 has a limitation, if we expose the admin port to it, the entire port will be public. We do not want that.

What we are doing currently is we build a simple go sidecar proxying F5 traffic to the envoy admin port.

[KBaichoo]

Sorry for the late reply, I think ideally if you could shim a way to access the admin port securely and make request through that shim, would be best.

Otherwise, I think if creating a workaround the question becomes how to ensure the access point from the workaround is secure. One way of implementing this could be using debug headers cross referencing that this comes from a trusted IP address.