Dynamically select upstream cluster in TCP Proxy network filter based on Proxy Protocol TLV headers

[gitanuj]

I'm trying to explore how I can use a TLV Header value from Proxy Protocol to select upstream cluster in TCP Proxy. So far I haven't found any documentation or code doing something similar. Is this currently supported? Is writing a custom Wasm network filter the way to go? Looks like if I can set envoy.tcp_proxy.cluster from my custom filter it will work.

Background:
I'm using Envoy (using Istio Ingress Gateway) as a reverse proxy and I want to automatically route connections coming through AWS Private Link. We can extract VPC Endpoint ID from Proxy Protocol TLV Headers and I want to use that to route to an upstream cluster (which is a local kubernetes Service). For public connections I'm using SNI based routing however for Private Links I want to route based on VPC Endpoint ID (to avoid creating a private DNS record).

[gitanuj]

I tried using Unified Matching API but looks like envoy.extensions.matching.common_inputs.network.v3.FilterStateInput just returns the underlying state's serializeAsString representation which for proxy protocol listener filter doesn't include TLV headers.

I'm now trying to see if I can somehow use Unified Matching on dynamic metadata (since proxy protocol filter can emit TLV header value as dynamic metadata).

[soulxu]

The Cluster's endpoint can be selected by the metadata https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/endpoint/v3/endpoint_components.proto#config-endpoint-v3-lbendpoint, is that help for your case if you only use one cluster but with multiple endpoints, and choice the endpoint based on the metadata?

[gitanuj]

I'm using Istio Ingress Gateway to route to local kubernetes services based on a PPv2 TLV Header. As I understand Istio uses VirtualService to add a Kubernetes Service as an upstream cluster to Envoy and it doesn't define any endpoints in this case (may be because these are kubernetes services so routing is handled by kube-proxy). So I can't rely on metadata match for endpoints.

[kyessenov]

Unified matching cannot set a dynamic metadata or a filter state to control tcp_proxy. What it can do is to select a tcp_proxy filter chain from a list given a metadata for TLV. If you want the former, you'd need a custom filter (or a Wasm/lua/golang script modulo the limitations of the runtimes).

[gitanuj]

@kyessenov Currently proxy_protocol filter allows emitting TLVs as dynamic metadata. Would it make sense if proxy_protocol filter could set filter state using a custom key? (another filter that does something similar is sni_cluster filter which always sets the filter state to SNI value) Tcp_proxy filter only relies on filter state envoy.tcp_proxy.cluster to route to upstream cluster. This way proxy_protocol filter can influence upstream cluster.

Otherwise the only other solution without having to recompile Envoy currently is writing a custom WASM filter (Lua isn't supported at L4 and Golang L4 filter doesn't support continueReading() to chain other filters yet).

[gitanuj]

Looks like this is currently not possible with all the extension mechanisms

1. Wasm doesn't support setting filter state (created issue here)
2. Golang network filter doesn't work as a non terminal filter. Also it is a WIP and doesn't even compile without directly referencing the code. (cc @antJack)
3. Lua doesn't support L4

Is there any other way I could intercept onNewConnection before tcp_proxy filter and set envoy.tcp_proxy.cluster filter state? One option is to do what sni_cluster filter is doing by adding a native filter and recompiling the proxy. However I'm trying to see if it is possible to achieve this without recompiling envoy.

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted" or "no stalebot". Thank you for your contributions.

[guybrushthre]

@gitanuj have you ever figured this out?