Fix deletion of interface on container delete in weave docker proxy mode

[murali-reddy]

• on container delete perform veth interface delete
• use consistent way to generate the name for veth pair on container create and delete

Fixes #3406 weave not deleting network interfaces

[bboreham]

So what changed? When I last looked at this area the veth would disappear if the netns it was attached to disappeared.

[murali-reddy]

So what changed?

From what i see handlers for ContainerDied and ContainerDestroyed were never implemented. I am not sure under what docker version and weave version combinations this was working earlier. I could not really trace how it might have worked earlier.

[bboreham]

Right, I meant what changed outside of Weave Net.

[bboreham]

Trying various versions reveals the veth goes away with the container when run with Weave Net 2.3.0, and does not go away with 2.4.0.

This is the result of ip link show with 2.3.0:

89: vethwepl1869@if88: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1376 qdisc noqueue master weave state UP mode DEFAULT group default 
    link/ether f2:2f:46:37:1e:55 brd ff:ff:ff:ff:ff:ff link-netnsid 0

and 2.4.0:

100: vethwepl3241@if99: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1376 qdisc noqueue master weave state UP mode DEFAULT group default 
    link/ether 0e:74:97:80:97:d7 brd ff:ff:ff:ff:ff:ff link-netnsid 0

Can't spot any relevant difference.

Looking at the changes in 2.4.0, I suspect #3291 - where we change namespace differently. I see the veth does go away if I stop the weaver process.

[bboreham]

I tried this branch out, and while it achieves the desired effect I see a number of error messages in the log:

INFO: 2019/01/03 15:57:10.148046 Command line options: map[datapath:datapath ipalloc-range:10.32.0.0/12 nickname:vagrant proxy:true resolv-conf:/var/run/weave/etc/resolv.conf status-addr:127.0.0.1:6782 weave-bridge:weave host-root:/host http-addr:127.0.0.1:6784 H:[unix:///var/run/weave/weave.sock] plugin:true port:6783 dns-listen-address:172.17.0.1:53 docker-bridge:docker0]
INFO: 2019/01/03 15:57:10.148251 weave  git-f910ce0f5635
INFO: 2019/01/03 15:57:10.149826 Docker API on unix:///var/run/docker.sock: &[Components=[{"Details":{"ApiVersion":"1.35","Arch":"amd64","BuildTime":"2017-12-27T20:09:53.000000000+00:00","Experimental":"false","GitCommit":"c97c6d6","GoVersion":"go1.9.2","KernelVersion":"4.4.0-92-generic","MinAPIVersion":"1.12","Os":"linux"},"Name":"Engine","Version":"17.12.0-ce"}] Version=17.12.0-ce GitCommit=c97c6d6 Os=linux BuildTime=2017-12-27T20:09:53.000000000+00:00 KernelVersion=4.4.0-92-generic Platform={"Name":""} ApiVersion=1.35 MinAPIVersion=1.12 GoVersion=go1.9.2 Arch=amd64]
INFO: 2019/01/03 15:57:10.161735 proxy listening on unix:///var/run/weave/weave.sock
INFO: 2019/01/03 15:57:10.304959 Bridge type is bridged_fastdp
INFO: 2019/01/03 15:57:10.305115 Communication between peers is unencrypted.
INFO: 2019/01/03 15:57:10.312108 Our name is 3a:76:44:de:f0:0d(vagrant)
INFO: 2019/01/03 15:57:10.312245 Launch detected - using supplied peer list: []
INFO: 2019/01/03 15:57:10.322628 Docker API on unix:///var/run/docker.sock: &[Components=[{"Details":{"ApiVersion":"1.35","Arch":"amd64","BuildTime":"2017-12-27T20:09:53.000000000+00:00","Experimental":"false","GitCommit":"c97c6d6","GoVersion":"go1.9.2","KernelVersion":"4.4.0-92-generic","MinAPIVersion":"1.12","Os":"linux"},"Name":"Engine","Version":"17.12.0-ce"}] GitCommit=c97c6d6 GoVersion=go1.9.2 Os=linux BuildTime=2017-12-27T20:09:53.000000000+00:00 Platform={"Name":""} Version=17.12.0-ce ApiVersion=1.35 MinAPIVersion=1.12 Arch=amd64 KernelVersion=4.4.0-92-generic]
INFO: 2019/01/03 15:57:10.323183 Checking for pre-existing addresses on weave bridge
INFO: 2019/01/03 15:57:10.348004 [allocator 3a:76:44:de:f0:0d] No valid persisted data
INFO: 2019/01/03 15:57:10.364597 [allocator 3a:76:44:de:f0:0d] Initialising via deferred consensus
INFO: 2019/01/03 15:57:10.391793 Listening for DNS queries on 172.17.0.1:53
INFO: 2019/01/03 15:57:10.398831 Sniffing traffic on datapath (via ODP)
INFO: 2019/01/03 15:57:10.411413 Listening for HTTP control messages on 127.0.0.1:6784
INFO: 2019/01/03 15:57:10.412332 Listening for status+metrics requests on 127.0.0.1:6782
INFO: 2019/01/03 15:57:10.420412 Waiting for Weave API Server...
INFO: 2019/01/03 15:57:10.429115 Finished waiting for Weave API Server
INFO: 2019/01/03 15:57:10.429338 Listening on /run/docker/plugins/weave.sock for global scope
INFO: 2019/01/03 15:57:10.430150 Listening on /run/docker/plugins/weavemesh.sock for local scope
INFO: 2019/01/03 15:57:10.430674 Creating default "weave" network
INFO: 2019/01/03 15:57:10.451661 [ipam] RequestPool weavelocal 10.32.0.0/12  map[]
INFO: 2019/01/03 15:57:10.452044 [net] CreateNetwork 35c712ee0dfad5e4acc17bd0f49cdec3fd95f0f4c5a4d4072bba6b46fec45984
ERRO: 2019/01/03 15:57:10.717056 error removing veth pairs created for deleted container: invalid argument
ERRO: 2019/01/03 15:57:55.690333 error removing veth pairs created for deleted container: invalid argument
ERRO: 2019/01/03 15:58:09.586653 error removing veth pairs created for deleted container: invalid argument
INFO: 2019/01/03 15:58:09.652372 Creating container with WEAVE_CIDR ""
INFO: 2019/01/03 15:58:10.025039 Attaching container ee41d5991d5f52991ea43af8259a1f3467a3a08fa0fdb1801d17f1619a0c276d with WEAVE_CIDR "" to weave network
INFO: 2019/01/03 15:58:10.027391 Assuming quorum size of 1
ERRO: 2019/01/03 15:58:10.444092 error removing veth pairs created for deleted container: invalid argument
INFO: 2019/01/03 15:58:10.577078 [nameserver 3a:76:44:de:f0:0d] Ignoring registration ee41d5991d5f. 10.32.0.1 ee41d5991d5f52991ea43af8259a1f3467a3a08fa0fdb1801d17f1619a0c276d (not a subdomain of weave.local.)

Some of them look like they come from containers dying that were never attached to the Weave network. The first one comes just from weave launch.

I think it would be best to remain silent if the veth name does not exist.

[murali-reddy]

thanks for the feedback, i will make the necessary changes.

Looking at the changes in 2.4.0, I suspect #3291 - where we change namespace differently. I see the veth does go away if I stop the weaver process.

Interesting. As I understand so from 2.4, weaver process switch to container namespace perform operations (instead of separate process) and switch back to its own namespace? But some how this holds back the veth pair? Let me check if there is any leakage of network namespaces as well.

[MikeMichel]

@murali-reddy can we support somehow (testing) here? will this also fix #3432 ?

[murali-reddy]

@MikeMichel I did not had a chance to investigate #3432, but does seems like a related. Will check if its due to same root cause.

can we support somehow (testing) here? will this also fix #3432 ?

Certainly could use help with testing. I will update the PR and share a private image

[murali-reddy]

Unfortunately fix in this PR does not work consistently. There is some latent problem after this change how network namespaces are switched. I need to rework on proper fix. Closing this PR.