diff --git a/content/en/blog/2023/egress-sni/egress-sni-flow.png b/content/en/blog/2023/egress-sni/egress-sni-flow.png new file mode 100644 index 000000000000..81cdbb969644 Binary files /dev/null and b/content/en/blog/2023/egress-sni/egress-sni-flow.png differ diff --git a/content/en/blog/2023/egress-sni/index.md b/content/en/blog/2023/egress-sni/index.md new file mode 100644 index 000000000000..e097dbc9581d --- /dev/null +++ b/content/en/blog/2023/egress-sni/index.md @@ -0,0 +1,344 @@ +--- +title: [WIP] Routing HTTPS/TLS traffic to arbitrary remote destinations +description: A generic approach to set up egress gateways and use them to route traffic to a restricted set of target remote hosts dynamically, supporting wildcard domains. +publishdate: 2023-12-10 +attribution: Gergő Huszty (IBM) +keywords: [traffic-management,gateway,mesh,mtls,egress,remote] +--- + +If you are using Istio to handle application originated outbound traffic, you are probably familiar with the concept of +egress gateways. Those are specific in a way that they are terminating and routing mesh application originated traffic +and not exposing their services to the external world. This is a useful feature if your system is operating in a restricted +environment and you want to control what can be reached on the public internet from your mesh. + +## Background +The use-case is documented in the official Istio docs until 1.13, but parts related to arbitrary target domains got removed +as it caused unfortunate confusions. +However, the old solution was still usable until now, when Istio 1.20 dropped some code from Envoy which made the approach work. +This post attempts to describe how we resolved the issue and fill the gap with an as-similar-as-possible approach. We will use +generic and Istio version independent components and Envoy features. That should allow users of the old solution to seamlessly +migrate before their systems are facing the breking changes in the old solution. + +## Problem to solve +The currently documented egress traffic routing use-cases are all building on the assumption that the target of the traffic +(the hostname) is statically determined by a `VirtualService`, that tells Envoy in the egress gateway pod where to TCP proxy +the matching outbound connections. You can use multiple and even wildcard DNS names to match the routing criteria, but you +are not able to route the traffic exactly where the application wanted to. For example you can match traffic that targets +`*.foo.com`, but you can direct everything into a single final target, i.e. `bar.foo.com`. If there would be a +service on `api.foo.com` that is not hosted by the same server as `bar.foo.com`, your traffic to that arbitrary host will +fail. In fact the connection would go to `bar.foo.com` servers, while the target hostname in the TLS handshake and the +HTTP payload would contain `api.foo.com`, which may not be served by the same servers. +The solution on high level is to inspect the original server name (SNI extension) in the application TLS handshake (that is sent +in plain-text, so no TLS termination or other man-in-the-middle operation is needed) in every new connection and use it as +a target dynamically to TCP proxy the traffic when leaves the gateway. + +## Further security measures +Now that we are restricting the traffic, we should be able to lock down the egress gateways in a way, that they can be used +only from within the mesh. This is achieved by enforcing the `ISTIO_MUTUAL` or mTLS peer authentication between the application +sidecar and the gateway. That means, there will be two layers of TLS on the application L7 payload. One that is the application +originated end-to-end TLS session terminated by the final remote target, and another one that is the istio mTLS session. + +Also, in order to mitigate application pod corruption, not only the application sidecar performs hostname list checks, but +the gateway as well. So that any compromised application pod can only still access the allowed targets, nothing more. + +## Low-level Envoy programming to the rescue +Recent Envoy releases implements a specific, dynamic TCP forward proxy solution that is using the SNI header on a per +connection basis to determine the target. While `VirtualService` cannot express the target like this, we are able to use +`EnvoyFilter`s to alter the Istio generated routing instructions to match the needs. + +The idea is to create a custom `Gateway` instance in the egress gateway pod(s) to listen for the outbound traffic. With +a `DestinationRule` and a `VirtualService` we can instruct the application sidecars to route the traffic (to a selected +list of hostnames) to that `Gateway`, using istio mTLS. On the gateway pod side we build the SNI forwarder with the mentioned +`EnvoyFilter`s by introducing internal Envoy listeners and clusters that will make it happen. The last thing is to patch the +internal destination of the gateway implemented TCP proxy to the internal SNI forwarder. + +## Solution + +{{< image width="90%" link="./egress-sni-flow.png" alt="Egress SNI routing with arbitrary domain names" title="Egress SNI routing with arbitrary domain names" caption="Egress SNI routing with arbitrary domain names" >}} + +In order to deploy the sample configuration, create the `istio-egress` namespace, allow injection in it and deploy the following manifest. + +{{< text yaml >}} +# New k8s cluster service to put egressgateway into the Service Registry +# so application sidecars can route traffic towards it within the mesh. +apiVersion: v1 +kind: Service +metadata: + name: egressgateway + namespace: istio-egress +spec: + type: ClusterIP + selector: + istio: egressgateway + ports: + - port: 443 + name: tls-egress + targetPort: 8443 +--- +# Gateway deployment with injection method +apiVersion: apps/v1 +kind: Deployment +metadata: + name: istio-egressgateway + namespace: istio-egress +spec: + selector: + matchLabels: + istio: egressgateway + template: + metadata: + annotations: + inject.istio.io/templates: gateway + labels: + istio: egressgateway + sidecar.istio.io/inject: "true" + spec: + containers: + - name: istio-proxy + image: auto # The image will automatically update each time the pod starts. + securityContext: + capabilities: + drop: + - ALL + runAsUser: 1337 + runAsGroup: 1337 +--- +# Set up roles to allow reading credentials for TLS +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: istio-egressgateway-sds + namespace: istio-egress +rules: +- apiGroups: [""] + resources: ["secrets"] + verbs: ["get", "watch", "list"] +- apiGroups: + - security.openshift.io + resourceNames: + - anyuid + resources: + - securitycontextconstraints + verbs: + - use +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: istio-egressgateway-sds + namespace: istio-egress +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: istio-egressgateway-sds +subjects: +- kind: ServiceAccount + name: default +--- +apiVersion: v1 +kind: List +items: +# Define a new listener that enforces Istio mTLS on inbound connections. +# This is where sidecar will route the application traffic, wrapped into +# Istio mTLS. + - apiVersion: networking.istio.io/v1alpha3 + kind: Gateway + metadata: + name: egressgateway + namespace: istio-system + spec: + selector: + istio: egressgateway + servers: + - port: + number: 8443 + name: tls-egress + protocol: TLS + hosts: + - "*" + tls: + mode: ISTIO_MUTUAL +# VirtualService that will instruct sidecars in the mesh to route the outgoing +# traffic to the egress gateway Service if the SNI target hostname matches + - apiVersion: networking.istio.io/v1alpha3 + kind: VirtualService + metadata: + name: direct-wildcard-through-egress-gateway + namespace: istio-system + spec: + hosts: + - "*.foo.com" + gateways: + - mesh + tls: + - match: + - gateways: + - mesh + port: 443 + sniHosts: + - "*.foo.com" + route: + - destination: + host: egressgateway.istio-egress.svc.cluster.local + subset: wildcard +# Dummy routing instruction. If omitted, no reference will point to the Gateway +# definition, and istiod will optimise the whole new listener out. + - apiVersion: networking.istio.io/v1alpha3 + kind: VirtualService + metadata: + name: direct-wildcard-through-egress-gateway2 + namespace: istio-system + spec: + hosts: + - "*" + gateways: + - egressgateway + tcp: + - match: + - gateways: + - egressgateway + port: 8443 + route: + - destination: + host: "dummy.local" + weight: 100 +# Instruct sidecars to use Istio mTLS when sending traffic to the egress gateway + - apiVersion: networking.istio.io/v1alpha3 + kind: DestinationRule + metadata: + name: egressgateway + namespace: istio-system + spec: + host: egressgateway.istio-egress.svc.cluster.local + subsets: + - name: wildcard + trafficPolicy: + tls: + mode: ISTIO_MUTUAL +# Put the remote targets into the Service Registry + - apiVersion: networking.istio.io/v1alpha3 + kind: ServiceEntry + metadata: + name: wildcard + namespace: istio-system + spec: + hosts: + - "*.foo.com" + ports: + - number: 443 + name: tls + protocol: TLS +# Access logging for the gateway + - apiVersion: telemetry.istio.io/v1alpha1 + kind: Telemetry + metadata: + name: mesh-default + namespace: istio-system + spec: + accessLogging: + - providers: + - name: envoy +# And finally, the configuration of the SNI forwarder, +# it's internal listener, and the patch to the original Gateway +# listener to route everything into the SNI forwarder. + - apiVersion: networking.istio.io/v1alpha3 + kind: EnvoyFilter + metadata: + name: sni-magic + namespace: istio-system + spec: + configPatches: + - applyTo: CLUSTER + match: + context: GATEWAY + patch: + operation: ADD + value: + name: sni_cluster + load_assignment: + cluster_name: sni_cluster + endpoints: + - lb_endpoints: + - endpoint: + address: + envoy_internal_address: + server_listener_name: sni_listener + - applyTo: CLUSTER + match: + context: GATEWAY + patch: + operation: ADD + value: + name: dynamic_forward_proxy_cluster + lb_policy: CLUSTER_PROVIDED + cluster_type: + name: envoy.clusters.dynamic_forward_proxy + typed_config: + "@type": type.googleapis.com/envoy.extensions.clusters.dynamic_forward_proxy.v3.ClusterConfig + dns_cache_config: + name: dynamic_forward_proxy_cache_config + dns_lookup_family: V4_ONLY + + - applyTo: LISTENER + match: + context: GATEWAY + patch: + operation: ADD + value: + name: sni_listener + internal_listener: {} + listener_filters: + - name: envoy.filters.listener.tls_inspector + typed_config: + "@type": type.googleapis.com/envoy.extensions.filters.listener.tls_inspector.v3.TlsInspector + + filter_chains: + - filter_chain_match: + server_names: + - "*.foo.com" + filters: + - name: envoy.filters.network.sni_dynamic_forward_proxy + typed_config: + "@type": type.googleapis.com/envoy.extensions.filters.network.sni_dynamic_forward_proxy.v3.FilterConfig + port_value: 443 + dns_cache_config: + name: dynamic_forward_proxy_cache_config + dns_lookup_family: V4_ONLY + - name: envoy.tcp_proxy + typed_config: + "@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy + stat_prefix: tcp + cluster: dynamic_forward_proxy_cluster + access_log: + - name: envoy.access_loggers.file + typed_config: + "@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog + path: "/dev/stdout" + log_format: + text_format_source: + inline_string: '[%START_TIME%] "%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% + %PROTOCOL%" %RESPONSE_CODE% %RESPONSE_FLAGS% %RESPONSE_CODE_DETAILS% %CONNECTION_TERMINATION_DETAILS% + "%UPSTREAM_TRANSPORT_FAILURE_REASON%" %BYTES_RECEIVED% %BYTES_SENT% %DURATION% + %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% "%REQ(X-FORWARDED-FOR)%" "%REQ(USER-AGENT)%" + "%REQ(X-REQUEST-ID)%" "%REQ(:AUTHORITY)%" "%UPSTREAM_HOST%" %UPSTREAM_CLUSTER% + %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% + %REQUESTED_SERVER_NAME% %ROUTE_NAME% + + ' + - applyTo: NETWORK_FILTER + match: + context: GATEWAY + listener: + filterChain: + filter: + name: "envoy.filters.network.tcp_proxy" + patch: + operation: MERGE + value: + name: envoy.tcp_proxy + typed_config: + "@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy + stat_prefix: tcp + cluster: sni_cluster +{{< /text >}} +