Consider using long loadbalancer connection timeouts to avoid api object re-lists

[aaronlevy]

When clients are watching api resources, if the connection times out, the client might miss updates, which then means it has to do a relist of all the watched objects and this could potentially be an expensive operation. (This was brought up in a discussion of adding HA to kubeadm)

[aaronlevy]

This also might affect kubectl logs and kubectl exec timeouts

[sym3tri]

@s-urbaniak thoughts on this?

[s-urbaniak]

So there is indeed one way to reproduce the behavior like the above on AWS, if you simply kubectl exec into a pod and do nothing, the default 60s idle timeout kicks in and disconnects after that timeout:

$ time kubectl exec -ti echopod-1961818357-ncr4j /bin/sh
/ # 
real    1m2.778s
user    0m0.210s
sys     0m0.030s

Note that his affects only idle connection, where no data stream happens to the ELB. What we can do here from our side is to make this idle timeout configurable: https://www.terraform.io/docs/providers/aws/r/elb.html#idle_timeout.

Additionally what falls into this discussion bucket is handling connection draining. Currently this feature is disabled by default but could also be made configurable:

• https://www.terraform.io/docs/providers/aws/r/elb.html#connection_draining
• https://www.terraform.io/docs/providers/aws/r/elb.html#connection_draining_timeout