no connections between containers on GKE

[rade]

We've got a cluster on GKE, and scope shows pod connections fine, e.g.

but it does not show container connections

except when they are in the host netns. Neither does it show process connections unless they are on the host netns.

[rade]

I've looked at the report. The containers do not have any IPs associated with them. That's because docker doesn't know about container IPs in the context of k8s. However, the probes should detect connections at the process level, which should then "bubble up" to the container level. See also #2586. So the question is why that isn't working here. Note that on GKE we don't get eBPF, so connections are tracked via conntrack, netstat and proc walking.

[rade]

@bboreham has suggested that the probes could obtain container IPs in a similar fashion how they determine weave IPs, via weaveutil container-addrs <bridge> ...
.

[rade]

So the question is why that isn't working here. Note that on GKE we don't get eBPF, so connections are tracked via conntrack, netstat and proc walking.

The latter (which is #2769) is certainly part of the reason; some short-lived connections, e.g. from load-test to front-end and front-end to catalogue aren't associated with processes because of that. However, we are also missing long-lived connections. The reason for that is #2668, which turns out to exclude more connections than it should, namely NATed connections.