No network inside containers launched inside dind on CentOS 7 host #466
Comments
|
Hmm, those (I'm not sure how |
|
The version of libseccomp on the troubled host: |
|
Yeah, thanks for testing -- it's probably not My best guess now is that the CentOS 7 kernel supports |
|
Also, to be clear, it's not just DNS that is failing. I'm also seeing ICMP and TCP failures. |
|
We are also experiencing this problem which makes all our CI build jobs failed. Changing all tags from latest to the previous version on hundred of repos is not that ideal :( |
|
Can you run the following one-liner on affected infrastructure and provide the full output?
It should look something like this: $ docker run -it --rm --privileged docker:dind sh -euxc 'modprobe nf_tables > /dev/null 2>&1 || :; if ! iptables -nL > /dev/null 2>&1; then modprobe ip_tables || :; /usr/local/sbin/.iptables-legacy/iptables -nL > /dev/null 2>&1; echo success legacy; else echo success nftables; fi'
+ modprobe nf_tables
+ :
+ iptables -nL
+ echo success nftables
success nftablesor: $ docker run -it --rm --privileged docker:dind sh -euxc 'modprobe nf_tables > /dev/null 2>&1 || :; if ! false iptables -nL > /dev/null 2>&1; then modprobe ip_tables || :; /usr/local/sbin/.iptables-legacy/iptables -nL > /dev/null 2>&1; echo success legacy; else echo success nftables; fi'
+ modprobe nf_tables
+ :
+ iptables -nL
+ modprobe ip_tables
ip: can't find device 'ip_tables'
ip_tables 36864 0
x_tables 53248 8 ip_tables,xt_mark,xt_nat,xt_tcpudp,xt_conntrack,xt_MASQUERADE,xt_addrtype,nft_compat
modprobe: can't change directory to '/lib/modules': No such file or directory
+ :
+ /usr/local/sbin/.iptables-legacy/iptables -nL
+ echo success legacy
success legacy |
|
|
Any chance you could test #468? 👀
|
This did not resolve it, unfortunately. Still no network inside the inner containers. |
|
That's absolutely flabbergasting. 😭 I spun up my own CentOS 7 instance to try and debug further, and I managed to replicate immediately. What I've found is that the host is definitely still using the legacy iptables/xtables, and we have zero means of detecting that reliably inside the container that I've found so far. So, as far as I can tell, there's something deficient in either the network namespaces or The best I've come up with is checking whether |
|
Is this something we could allow the user to specify via a config ENV var? I also wonder how much of a concern this needs to be, as CentOS 7 goes EOL at the end of June. If this is a pain for maintenance I think a documented config setting for near end of life / past end of life setups is reasonable. |
|
Ok, #468 is probably on hold until the new year (#468 (comment)), but here are some workarounds if you need to fix this before we can resolve that: FROM docker:dind
ENV PATH /usr/local/sbin/.iptables-legacy:$PATHor:
|
We're seeing issues where dind images after
docker:24.0.6-dindhave issues with networking inside the inner containers. The following logs are using thedocker@sha256:8f9c4d8cdaa2f87b5269d4d6759711c843c37e34a02b8bb45653e5b8f4e2f0a2image, which I believe should have the updates from #463 (please let me know if ti doesn't).I can reproduce our issues by launching dind with
docker run --rm -ti --privileged --name docker -e DOCKER_TLS_CERTDIR= -p 2375:2375 docker@sha256:8f9c4d8cdaa2f87b5269d4d6759711c843c37e34a02b8bb45653e5b8f4e2f0a2I believe the
bridgewarning to be a red-herring as I see that indocker:24.0.6-dindwhich works.I then run a
debiancontainer and try aapt-get updateNetwork works if I run the inner container with
--net=host.Host info:
The text was updated successfully, but these errors were encountered: