Connect: Expose HTTP health checks

[banks]

Currently we recommend that apps that use a Connect proxy should protect themselves from clients bypassing the proxy by only listening on localhost.

On a VM-based setup where the Consul agent and application are on same host and share a network namespace this works fine. The app can register TCP/HTTP health checks and agent can perform them locally even though the app instance is not exposed to the network.

In Kubernetes where we run agents in a separate pod and they can only talk to the application pod via it's public IP, there is currently no way to have the app register network-based health checks with Consul and protect itself by listening only on loopback within the pod.

By extension, for applications that specify Kube liveness/readyness checks, these stop working once they stop listening publicly on the Pod IP and the same issue occurs for the Kubelet.

It's possible to work around this now if you can modify your app or manually configure the proxy in some way to expose only the health check endpoint but this is not obvious or easy to do.

We should do something like:

• When setting up a sidecar proxy, if the target app instance has network-based health checks defined, we should (optionally) configure an additional listener on the sidecar that is exposed to the public and does not require Connect certificate authentication (although still over TLS), but only exposes the exact endpoint(s) necessary for the defined health check. It is assumed that these don't pose a major risk in being exposed since they must be to have the agent or kubelet which are not specifically authorized by intentions to access them.

Note that we will likely need to setup a similar listener anyway for exposing Prometheus metrics publicly from Envoy (proxying to the insecure internal admin interface on loopback).

Another potential option would be if we can configure Envoy to perform the active health check itself on the local app instance and then just have a simple TCP check to the public port. This would only work if Envoy rejects new connections when it has no healthy backend instances which I think is true but would need validating. It also requires that any network check possible in Consul or kube needs to be supported by active health checks in Envoy too but I thing that is the case.