Add TCP Keepalives to Envoy -> Contour xDS comms

[youngnick]

By default, Envoy will not tear down half open TCP connections for xDS, and will believe that it's connected.
See envoyproxy/envoy#6323 and envoyproxy/envoy#5173 for more background.

We should configure tcp keepalives for the Envoy -> Contour connection to ensure that half-open connections are flushed.

A sample config from envoyproxy#5173:

    static_resources:
      clusters:
      - name: {{ cluster_name }}_ads
        connect_timeout: { seconds: 1 }
        dns_refresh_rate: { seconds: 10 }
        type: STRICT_DNS
        ...
        upstream_connection_options:
          tcp_keepalive:
            keepalive_probes: 3
            keepalive_time: 30
            keepalive_interval: 5

Thanks to @ravilr for noting this one.

It's also possible that #1523 might be related to this.

[bgagnon]

@youngnick I was just about to comment on #1523 to say we were successful in reproducing the issue by turning off our forced EDS updates for ~48 hours -- this caused 15 out of 16 envoys to stop receiving updates from Contour.

I immediately started looking into timeout issues and found out the bootstrap was missing TCP keepalive options in 0.15. I was going to open the exact same merge request at #1746 next.

In any case, I'll add more details to #1523 but it's looking more and more like you've fixed it with b183d04 🎉

[bgagnon]

@youngnick sorry for asking, but do you think there's a chance this could be back-ported to the 0.15.x series? we'd like to test this important bug fix prior to our beta and 1.0 changes

[bgagnon]

Should the PermitWithoutStream option be enabled as well? This is what we've put in place in our XDS proxy:

	grpcServer := grpc.NewServer(
		grpc.KeepaliveEnforcementPolicy(keepalive.EnforcementPolicy{
			PermitWithoutStream: true, // Allow pings even when there are no active streams
		}),
		grpc.KeepaliveParams(keepalive.ServerParameters{
			Time:    60 * time.Second, // Ping the client if it is idle for 60 seconds to ensure the connection is still active
			Timeout: 5 * time.Second,  // Wait 5 seconds for the ping ack before assuming the connection is dead
		}))

Given the design of XDS I guess we almost always have a stream open... so I am not sure about this.

[davecheney]

@bgagnon this probably won't hurt.

[youngnick]

@bgagnon thanks, I missed that. D'oh.