Stolon-proxy has no TCP keepalive

[Dirbaio]

We had some issues in prod due to stolon-proxy not closing connections to the master when the incoming connection is closed. This caused all the available connections in the master to run out, and nothing would fix it other than killing stolon-proxy.

Here you can see the Conntrack output (100.92.128.134 is the IP of stolon-proxy, 100.122.206.245 is the IP of the current master keeper.

There are 49 ESTABLISHED connections from the proxy to the master, but just 1 ESTABLISHED connection from somewhere else to the proxy. Shouldn't the proxy close all the connections?

root@nodes-89nq:~# conntrack -L | grep 100.92.128.134
tcp      6 79360 ESTABLISHED src=100.92.128.134 dst=100.122.206.245 sport=58640 dport=5432 src=100.122.206.245 dst=100.92.128.134 sport=5432 dport=58640 [ASSURED] mark=0 use=1
tcp      6 82703 ESTABLISHED src=100.92.128.134 dst=100.122.206.245 sport=35460 dport=5432 src=100.122.206.245 dst=100.92.128.134 sport=5432 dport=35460 [ASSURED] mark=0 use=1
tcp      6 95 TIME_WAIT src=100.92.128.134 dst=100.122.206.245 sport=53774 dport=5432 src=100.122.206.245 dst=100.92.128.134 sport=5432 dport=53774 [ASSURED] mark=0 use=1
tcp      6 84472 ESTABLISHED src=100.92.128.134 dst=100.122.206.245 sport=35672 dport=5432 src=100.122.206.245 dst=100.92.128.134 sport=5432 dport=35672 [ASSURED] mark=0 use=1
tcp      6 83391 ESTABLISHED src=100.92.128.134 dst=100.122.206.245 sport=43076 dport=5432 src=100.122.206.245 dst=100.92.128.134 sport=5432 dport=43076 [ASSURED] mark=0 use=1
tcp      6 83358 ESTABLISHED src=100.92.128.134 dst=100.122.206.245 sport=42902 dport=5432 src=100.122.206.245 dst=100.92.128.134 sport=5432 dport=42902 [ASSURED] mark=0 use=1
tcp      6 85029 ESTABLISHED src=100.92.128.134 dst=100.122.206.245 sport=41708 dport=5432 src=100.122.206.245 dst=100.92.128.134 sport=5432 dport=41708 [ASSURED] mark=0 use=1
tcp      6 79786 ESTABLISHED src=100.92.128.134 dst=100.122.206.245 sport=52778 dport=5432 src=100.122.206.245 dst=100.92.128.134 sport=5432 dport=52778 [ASSURED] mark=0 use=1
tcp      6 91 TIME_WAIT src=100.92.128.134 dst=100.122.206.245 sport=53700 dport=5432 src=100.122.206.245 dst=100.92.128.134 sport=5432 dport=53700 [ASSURED] mark=0 use=1
tcp      6 41 TIME_WAIT src=10.132.0.6 dst=100.92.128.134 sport=55402 dport=5432 src=100.92.128.134 dst=10.132.0.6 sport=5432 dport=55402 [ASSURED] mark=0 use=1
tcp      6 81556 ESTABLISHED src=100.92.128.134 dst=100.122.206.245 sport=60890 dport=5432 src=100.122.206.245 dst=100.92.128.134 sport=5432 dport=60890 [ASSURED] mark=0 use=1
tcp      6 86399 ESTABLISHED src=100.92.128.134 dst=100.122.206.245 sport=46450 dport=5432 src=100.122.206.245 dst=100.92.128.134 sport=5432 dport=46450 [ASSURED] mark=0 use=1
tcp      6 111 TIME_WAIT src=10.132.0.6 dst=100.92.128.134 sport=56600 dport=5432 src=100.92.128.134 dst=10.132.0.6 sport=5432 dport=56600 [ASSURED] mark=0 use=1
tcp      6 95 TIME_WAIT src=100.92.223.202 dst=100.92.128.134 sport=39078 dport=5432 src=100.92.128.134 dst=100.92.223.202 sport=5432 dport=39078 [ASSURED] mark=0 use=2
tcp      6 81 TIME_WAIT src=100.92.128.134 dst=100.122.206.245 sport=53524 dport=5432 src=100.122.206.245 dst=100.92.128.134 sport=5432 dport=53524 [ASSURED] mark=0 use=1
tcp      6 83325 ESTABLISHED src=100.92.128.134 dst=100.122.206.245 sport=42280 dport=5432 src=100.122.206.245 dst=100.92.128.134 sport=5432 dport=42280 [ASSURED] mark=0 use=1
tcp      6 81818 ESTABLISHED src=100.92.128.134 dst=100.122.206.245 sport=57736 dport=5432 src=100.122.206.245 dst=100.92.128.134 sport=5432 dport=57736 [ASSURED] mark=0 use=1
tcp      6 31 TIME_WAIT src=100.92.128.134 dst=100.122.206.245 sport=52688 dport=5432 src=100.122.206.245 dst=100.92.128.134 sport=5432 dport=52688 [ASSURED] mark=0 use=1
tcp      6 25 TIME_WAIT src=100.92.128.134 dst=100.122.206.245 sport=52576 dport=5432 src=100.122.206.245 dst=100.92.128.134 sport=5432 dport=52576 [ASSURED] mark=0 use=1
tcp      6 84931 ESTABLISHED src=100.92.128.134 dst=100.122.206.245 sport=59060 dport=5432 src=100.122.206.245 dst=100.92.128.134 sport=5432 dport=59060 [ASSURED] mark=0 use=1
tcp      6 79557 ESTABLISHED src=100.92.128.134 dst=100.122.206.245 sport=50450 dport=5432 src=100.122.206.245 dst=100.92.128.134 sport=5432 dport=50450 [ASSURED] mark=0 use=1
tcp      6 84472 ESTABLISHED src=100.92.128.134 dst=100.122.206.245 sport=35658 dport=5432 src=100.122.206.245 dst=100.92.128.134 sport=5432 dport=35658 [ASSURED] mark=0 use=1
tcp      6 86399 ESTABLISHED src=100.92.128.134 dst=100.92.128.133 sport=42472 dport=2379 src=100.92.128.133 dst=100.92.128.134 sport=2379 dport=42472 [ASSURED] mark=0 use=1
tcp      6 83358 ESTABLISHED src=100.92.128.134 dst=100.122.206.245 sport=42904 dport=5432 src=100.122.206.245 dst=100.92.128.134 sport=5432 dport=42904 [ASSURED] mark=0 use=1
tcp      6 79655 ESTABLISHED src=100.92.128.134 dst=100.122.206.245 sport=51558 dport=5432 src=100.122.206.245 dst=100.92.128.134 sport=5432 dport=51558 [ASSURED] mark=0 use=1
tcp      6 83424 ESTABLISHED src=100.92.128.134 dst=100.122.206.245 sport=43692 dport=5432 src=100.122.206.245 dst=100.92.128.134 sport=5432 dport=43692 [ASSURED] mark=0 use=1
tcp      6 86383 ESTABLISHED src=100.92.128.134 dst=100.68.192.16 sport=39196 dport=2379 src=100.92.128.133 dst=100.92.128.134 sport=2379 dport=39196 [ASSURED] mark=0 use=2
tcp      6 84734 ESTABLISHED src=100.92.128.134 dst=100.122.206.245 sport=42016 dport=5432 src=100.122.206.245 dst=100.92.128.134 sport=5432 dport=42016 [ASSURED] mark=0 use=1
tcp      6 41 TIME_WAIT src=100.92.128.134 dst=100.122.206.245 sport=52870 dport=5432 src=100.122.206.245 dst=100.92.128.134 sport=5432 dport=52870 [ASSURED] mark=0 use=1
tcp      6 105 TIME_WAIT src=100.92.128.134 dst=100.122.206.245 sport=53964 dport=5432 src=100.122.206.245 dst=100.92.128.134 sport=5432 dport=53964 [ASSURED] mark=0 use=1
tcp      6 84472 ESTABLISHED src=100.92.128.134 dst=100.122.206.245 sport=35670 dport=5432 src=100.122.206.245 dst=100.92.128.134 sport=5432 dport=35670 [ASSURED] mark=0 use=1
tcp      6 83424 ESTABLISHED src=100.92.128.134 dst=100.122.206.245 sport=43690 dport=5432 src=100.122.206.245 dst=100.92.128.134 sport=5432 dport=43690 [ASSURED] mark=0 use=1
tcp      6 85062 ESTABLISHED src=100.92.128.134 dst=100.122.206.245 sport=41996 dport=5432 src=100.122.206.245 dst=100.92.128.134 sport=5432 dport=41996 [ASSURED] mark=0 use=1
tcp      6 61 TIME_WAIT src=100.92.128.134 dst=100.122.206.245 sport=53200 dport=5432 src=100.122.206.245 dst=100.92.128.134 sport=5432 dport=53200 [ASSURED] mark=0 use=1
tcp      6 101 TIME_WAIT src=100.92.128.134 dst=100.122.206.245 sport=53896 dport=5432 src=100.122.206.245 dst=100.92.128.134 sport=5432 dport=53896 [ASSURED] mark=0 use=1
tcp      6 81720 ESTABLISHED src=100.92.128.134 dst=100.122.206.245 sport=56900 dport=5432 src=100.122.206.245 dst=100.92.128.134 sport=5432 dport=56900 [ASSURED] mark=0 use=1
tcp      6 21 TIME_WAIT src=100.92.128.134 dst=100.122.206.245 sport=52514 dport=5432 src=100.122.206.245 dst=100.92.128.134 sport=5432 dport=52514 [ASSURED] mark=0 use=1
tcp      6 81163 ESTABLISHED src=100.92.128.134 dst=100.122.206.245 sport=56786 dport=5432 src=100.122.206.245 dst=100.92.128.134 sport=5432 dport=56786 [ASSURED] mark=0 use=1
tcp      6 55 TIME_WAIT src=100.92.128.134 dst=100.122.206.245 sport=53102 dport=5432 src=100.122.206.245 dst=100.92.128.134 sport=5432 dport=53102 [ASSURED] mark=0 use=1
tcp      6 82604 ESTABLISHED src=100.92.128.134 dst=100.122.206.245 sport=34256 dport=5432 src=100.122.206.245 dst=100.92.128.134 sport=5432 dport=34256 [ASSURED] mark=0 use=1
tcp      6 81 TIME_WAIT src=10.132.0.6 dst=100.92.128.134 sport=56056 dport=5432 src=100.92.128.134 dst=10.132.0.6 sport=5432 dport=56056 [ASSURED] mark=0 use=1
tcp      6 11 TIME_WAIT src=100.92.128.134 dst=100.122.206.245 sport=52346 dport=5432 src=100.122.206.245 dst=100.92.128.134 sport=5432 dport=52346 [ASSURED] mark=0 use=1
tcp      6 85062 ESTABLISHED src=100.92.128.134 dst=100.122.206.245 sport=41994 dport=5432 src=100.122.206.245 dst=100.92.128.134 sport=5432 dport=41994 [ASSURED] mark=0 use=1
tcp      6 84931 ESTABLISHED src=100.92.128.134 dst=100.122.206.245 sport=59058 dport=5432 src=100.122.206.245 dst=100.92.128.134 sport=5432 dport=59058 [ASSURED] mark=0 use=1
tcp      6 81261 ESTABLISHED src=100.92.128.134 dst=100.122.206.245 sport=57926 dport=5432 src=100.122.206.245 dst=100.92.128.134 sport=5432 dport=57926 [ASSURED] mark=0 use=1
tcp      6 84243 ESTABLISHED src=100.92.128.134 dst=100.122.206.245 sport=52464 dport=5432 src=100.122.206.245 dst=100.92.128.134 sport=5432 dport=52464 [ASSURED] mark=0 use=1
tcp      6 71 TIME_WAIT src=10.132.0.6 dst=100.92.128.134 sport=55894 dport=5432 src=100.92.128.134 dst=10.132.0.6 sport=5432 dport=55894 [ASSURED] mark=0 use=1
tcp      6 79426 ESTABLISHED src=100.92.128.134 dst=100.122.206.245 sport=59306 dport=5432 src=100.122.206.245 dst=100.92.128.134 sport=5432 dport=59306 [ASSURED] mark=0 use=1
tcp      6 111 TIME_WAIT src=100.92.128.134 dst=100.122.206.245 sport=54068 dport=5432 src=100.122.206.245 dst=100.92.128.134 sport=5432 dport=54068 [ASSURED] mark=0 use=1
tcp      6 61 TIME_WAIT src=10.132.0.6 dst=100.92.128.134 sport=55732 dport=5432 src=100.92.128.134 dst=10.132.0.6 sport=5432 dport=55732 [ASSURED] mark=0 use=1
tcp      6 115 TIME_WAIT src=100.92.128.134 dst=100.122.206.245 sport=54142 dport=5432 src=100.122.206.245 dst=100.92.128.134 sport=5432 dport=54142 [ASSURED] mark=0 use=1
tcp      6 86383 ESTABLISHED src=100.92.128.134 dst=100.68.192.16 sport=39198 dport=2379 src=100.92.128.133 dst=100.92.128.134 sport=2379 dport=39198 [ASSURED] mark=0 use=1
tcp      6 31 TIME_WAIT src=10.132.0.6 dst=100.92.128.134 sport=55220 dport=5432 src=100.92.128.134 dst=10.132.0.6 sport=5432 dport=55220 [ASSURED] mark=0 use=1
tcp      6 115 TIME_WAIT src=100.92.223.202 dst=100.92.128.134 sport=39160 dport=5432 src=100.92.128.134 dst=100.92.223.202 sport=5432 dport=39160 [ASSURED] mark=0 use=1
tcp      6 79360 ESTABLISHED src=100.92.128.134 dst=100.122.206.245 sport=58638 dport=5432 src=100.122.206.245 dst=100.92.128.134 sport=5432 dport=58638 [ASSURED] mark=0 use=1
tcp      6 86399 ESTABLISHED src=100.92.223.244 dst=100.92.128.134 sport=46422 dport=5432 src=100.92.128.134 dst=100.92.223.244 sport=5432 dport=46422 [ASSURED] mark=0 use=1
tcp      6 79557 ESTABLISHED src=100.92.128.134 dst=100.122.206.245 sport=50452 dport=5432 src=100.122.206.245 dst=100.92.128.134 sport=5432 dport=50452 [ASSURED] mark=0 use=1
tcp      6 1 TIME_WAIT src=100.92.128.134 dst=100.122.206.245 sport=52168 dport=5432 src=100.122.206.245 dst=100.92.128.134 sport=5432 dport=52168 [ASSURED] mark=0 use=1
tcp      6 51 TIME_WAIT src=10.132.0.6 dst=100.92.128.134 sport=55572 dport=5432 src=100.92.128.134 dst=10.132.0.6 sport=5432 dport=55572 [ASSURED] mark=0 use=1
tcp      6 84734 ESTABLISHED src=100.92.128.134 dst=100.122.206.245 sport=42018 dport=5432 src=100.122.206.245 dst=100.92.128.134 sport=5432 dport=42018 [ASSURED] mark=0 use=1
tcp      6 82441 ESTABLISHED src=100.92.128.134 dst=100.122.206.245 sport=57934 dport=5432 src=100.122.206.245 dst=100.92.128.134 sport=5432 dport=57934 [ASSURED] mark=0 use=2
tcp      6 85193 ESTABLISHED src=100.92.128.134 dst=100.122.206.245 sport=34074 dport=5432 src=100.122.206.245 dst=100.92.128.134 sport=5432 dport=34074 [ASSURED] mark=0 use=1
tcp      6 21 TIME_WAIT src=10.132.0.6 dst=100.92.128.134 sport=55046 dport=5432 src=100.92.128.134 dst=10.132.0.6 sport=5432 dport=55046 [ASSURED] mark=0 use=1
tcp      6 85062 ESTABLISHED src=100.92.128.134 dst=100.122.206.245 sport=60944 dport=5432 src=100.122.206.245 dst=100.92.128.134 sport=5432 dport=60944 [ASSURED] mark=0 use=1
tcp      6 101 TIME_WAIT src=10.132.0.6 dst=100.92.128.134 sport=56428 dport=5432 src=100.92.128.134 dst=10.132.0.6 sport=5432 dport=56428 [ASSURED] mark=0 use=1
tcp      6 51 TIME_WAIT src=100.92.128.134 dst=100.122.206.245 sport=53040 dport=5432 src=100.122.206.245 dst=100.92.128.134 sport=5432 dport=53040 [ASSURED] mark=0 use=1
tcp      6 91 TIME_WAIT src=10.132.0.6 dst=100.92.128.134 sport=56232 dport=5432 src=100.92.128.134 dst=10.132.0.6 sport=5432 dport=56232 [ASSURED] mark=0 use=1
tcp      6 79426 ESTABLISHED src=100.92.128.134 dst=100.122.206.245 sport=59310 dport=5432 src=100.122.206.245 dst=100.92.128.134 sport=5432 dport=59310 [ASSURED] mark=0 use=1
tcp      6 105 TIME_WAIT src=100.92.223.202 dst=100.92.128.134 sport=39118 dport=5432 src=100.92.128.134 dst=100.92.223.202 sport=5432 dport=39118 [ASSURED] mark=0 use=1
tcp      6 1 TIME_WAIT src=10.132.0.6 dst=100.92.128.134 sport=54700 dport=5432 src=100.92.128.134 dst=10.132.0.6 sport=5432 dport=54700 [ASSURED] mark=0 use=1
tcp      6 79360 ESTABLISHED src=100.92.128.134 dst=100.122.206.245 sport=58694 dport=5432 src=100.122.206.245 dst=100.92.128.134 sport=5432 dport=58694 [ASSURED] mark=0 use=1
tcp      6 81720 ESTABLISHED src=100.92.128.134 dst=100.122.206.245 sport=34376 dport=5432 src=100.122.206.245 dst=100.92.128.134 sport=5432 dport=34376 [ASSURED] mark=0 use=1
tcp      6 5 TIME_WAIT src=100.92.223.202 dst=100.92.128.134 sport=38670 dport=5432 src=100.92.128.134 dst=100.92.223.202 sport=5432 dport=38670 [ASSURED] mark=0 use=1
tcp      6 81261 ESTABLISHED src=100.92.128.134 dst=100.122.206.245 sport=57908 dport=5432 src=100.122.206.245 dst=100.92.128.134 sport=5432 dport=57908 [ASSURED] mark=0 use=1
tcp      6 82637 ESTABLISHED src=100.92.128.134 dst=100.122.206.245 sport=34472 dport=5432 src=100.122.206.245 dst=100.92.128.134 sport=5432 dport=34472 [ASSURED] mark=0 use=1
tcp      6 35 TIME_WAIT src=100.92.223.202 dst=100.92.128.134 sport=38810 dport=5432 src=100.92.128.134 dst=100.92.223.202 sport=5432 dport=38810 [ASSURED] mark=0 use=1
tcp      6 81621 ESTABLISHED src=100.92.128.134 dst=100.122.206.245 sport=33410 dport=5432 src=100.122.206.245 dst=100.92.128.134 sport=5432 dport=33410 [ASSURED] mark=0 use=1
tcp      6 83358 ESTABLISHED src=100.92.128.134 dst=100.122.206.245 sport=42872 dport=5432 src=100.122.206.245 dst=100.92.128.134 sport=5432 dport=42872 [ASSURED] mark=0 use=1
tcp      6 25 TIME_WAIT src=100.92.223.202 dst=100.92.128.134 sport=38768 dport=5432 src=100.92.128.134 dst=100.92.223.202 sport=5432 dport=38768 [ASSURED] mark=0 use=1
tcp      6 82801 ESTABLISHED src=100.92.128.134 dst=100.122.206.245 sport=36370 dport=5432 src=100.122.206.245 dst=100.92.128.134 sport=5432 dport=36370 [ASSURED] mark=0 use=1
tcp      6 35 TIME_WAIT src=100.92.128.134 dst=100.122.206.245 sport=52746 dport=5432 src=100.122.206.245 dst=100.92.128.134 sport=5432 dport=52746 [ASSURED] mark=0 use=1
tcp      6 83555 ESTABLISHED src=100.92.128.134 dst=100.122.206.245 sport=44876 dport=5432 src=100.122.206.245 dst=100.92.128.134 sport=5432 dport=44876 [ASSURED] mark=0 use=1
tcp      6 55 TIME_WAIT src=100.92.223.202 dst=100.92.128.134 sport=38896 dport=5432 src=100.92.128.134 dst=100.92.223.202 sport=5432 dport=38896 [ASSURED] mark=0 use=1
tcp      6 11 TIME_WAIT src=10.132.0.6 dst=100.92.128.134 sport=54878 dport=5432 src=100.92.128.134 dst=10.132.0.6 sport=5432 dport=54878 [ASSURED] mark=0 use=1
tcp      6 81818 ESTABLISHED src=100.92.128.134 dst=100.122.206.245 sport=57734 dport=5432 src=100.122.206.245 dst=100.92.128.134 sport=5432 dport=57734 [ASSURED] mark=0 use=1
tcp      6 79393 ESTABLISHED src=100.92.128.134 dst=100.122.206.245 sport=58898 dport=5432 src=100.122.206.245 dst=100.92.128.134 sport=5432 dport=58898 [ASSURED] mark=0 use=1
tcp      6 5 TIME_WAIT src=100.92.128.134 dst=100.122.206.245 sport=52234 dport=5432 src=100.122.206.245 dst=100.92.128.134 sport=5432 dport=52234 [ASSURED] mark=0 use=1
tcp      6 71 TIME_WAIT src=100.92.128.134 dst=100.122.206.245 sport=53362 dport=5432 src=100.122.206.245 dst=100.92.128.134 sport=5432 dport=53362 [ASSURED] mark=0 use=2
tcp      6 86399 ESTABLISHED src=100.92.128.134 dst=100.92.128.133 sport=39290 dport=2379 src=100.92.128.133 dst=100.92.128.134 sport=2379 dport=39290 [ASSURED] mark=0 use=1
tcp      6 81621 ESTABLISHED src=100.92.128.134 dst=100.122.206.245 sport=33408 dport=5432 src=100.122.206.245 dst=100.92.128.134 sport=5432 dport=33408 [ASSURED] mark=0 use=1
tcp      6 79754 ESTABLISHED src=100.92.128.134 dst=100.122.206.245 sport=52512 dport=5432 src=100.122.206.245 dst=100.92.128.134 sport=5432 dport=52512 [ASSURED] mark=0 use=1
tcp      6 79754 ESTABLISHED src=100.92.128.134 dst=100.122.206.245 sport=52508 dport=5432 src=100.122.206.245 dst=100.92.128.134 sport=5432 dport=52508 [ASSURED] mark=0 use=1

[Dirbaio]

In case it helps: I killed stolon-proxy with SIGABRT, which caused it to print this goroutine dump:

https://gist.github.com/Dirbaio/6d25d3baa843662cb44821513815a9d6

[Dirbaio]

Okay, I think I know what's going on. Running ss inside the docker container does show incoming connections as ESTABLISHED, but they're to a pod that no longer exists.

My theory is this:

• Some pod was opening tons of connections to stolon-proxy
• I killed the pod's deployment, it caused the pod to be deleted
• Somehow deleting the pod didn't send any RST or FIN to stolon-proxy
• stolon-proxy is stuck with the connections forever: they don't timeout (or they take a LONG time to timeout, long enough to exhaust all postgres connection slots anyway)

Maybe this could be solved enabling some sort of keepalive, so stolon-proxy can notice a client is dead?

[sgotti]

@Dirbaio. Thanks for the detailed report. I'll tale a detailed look at it tomorrow.

I never noticed the stolon proxy leaking connection (though we should add some integration tests in pollon to catch possible regressions). What version of stolon are you using (git master, a specific commit, a released version? we should add a version option to print it...)?

If instead it's a problem in the k8s netfilter based service proxy or in the pod network between nodes I'm not sure we can do something inside stolon (we can add an option to set custom socket keepalives values but that will be a workaround to the real problem). I'll try to trigger it to see if I can reproduce this behavior on a multi node cluster. What k8s pod network communication type are you using? (static, flannel, custom like GCE etc...).

[Dirbaio]

Some details on my setup: K8s 1.5.2 on GCE Ubuntu 16.04 VMs. Pod network is Calico with IP-IP tunneling and NAT enabled. EDIT: And the Stolon version is 0.5.0 plus the patch for #257

Anyway, I think in the general case you just can't rely on the remote end to properly shut down connections. This could also be caused by the VM on the remote end to be forcibly shut down or crash, or by network issues, or many other causes. IMO stolon-proxy should simply time out these connections and move on.

Also I'm pretty sure Postgres server itself enables TCP keepalive for this reason :)

[sgotti]

Anyway, I think in the general case you just can't rely on the remote end to properly shut down connections. This could also be caused by the VM on the remote end to be forcibly shut down or crash, or by network issues, or many other causes. IMO stolon-proxy should simply time out these connections and move on.

I agree. I think we should add to the proxy some options to enable tcp keepalive (or perhaps enable it by default on the socket like postgres does) and to set the related parameter (since the os defaults, if not changed, should be too high) like postgres already permits. I'll open a new issue since there're some implementation detail related on how to do this with golang.

Some details on my setup: K8s 1.5.2 on GCE Ubuntu 16.04 VMs. Pod network is Calico with IP-IP tunneling and NAT enabled. EDIT: And the Stolon version is 0.5.0 plus the patch for #257

I tried to simulate missing connection reset on a little k8s cluster but when I delete a pod (also forcing it without a grace period) the process is always stopped before removing the container virtual interface so a FIN/RST is always sent to the proxy. Obviously I can reproduce this if I detach the k8s node network cable or abruptly shutdown the node.

@Dirbaio Since looks like the issue isn't the stolon-proxy leaking connection can you please change the title? Thanks.

[Dirbaio]

Changed the title :)

Yeah, I tried to reproduce it too, but couldn't. Shutting down a pod the regular way always sends FIN. I don't know why it happened to me, maybe there's something else going on, but it did happen.

IMO adding keepalive by default to the proxy is a great fix

[sgotti]

Fixed by #357