Weave on a single centos7 host, Containers cant resolve each other #1266
Comments
|
You've missed out the |
|
I did that as well Should i be using |
|
I see what the problem is. You are setting the hostnames to You shouldn't need to set the hostname at all since weave will derive it automatically from the container name, placing it in the |
|
[root@dockerHostCentOS7 docker-dns-build]# docker exec webnode1 ping -c 5 webnode2 Still no luck!!, not sure if this helps |
|
are you sure you have run the |
|
Oops, i wrote a small bash script to start weave (and its containers) so the eval gets executed within the bash script. so in essence the docker run shell doesn't see the
|
|
still no luck Is it in any case related to this issue - #1033 |
|
The above works absolutely fine for me; the ping succeeds :( |
|
What do |
|
I dont know if this has any bearing i start docker with the google dns servers here is the Here is the output for |
|
That status/ps output looks fine. Please repeat the test, but this time launch weavedns with |
i.e. 1.7.1? |
Here are the debug messages for weavedns
|
|
So it looks like the DNS queries never make it to weavedns. On the host, run
That is not the latest version. Any chance you could try with 1.7.1? |
|
Ok got Here is the ping test, no luck yet. Docker version Here are the logs This is a minimal install of centos doesn't have dig, let me install and try that command. |
|
Here is the output of the dig command. dig for webnode2 get the NXDOMAIN |
That looks good. What do you get from |
|
not sure i did it right, there was no output |
|
So somehow name resolution from inside the containers just doesn't work. does the output of |
The output of iptables-save |
docker exec ... |
|
I corrected it again. ping to host IP is working |
I want to know whether pinging that specific IP, which is the docker bridge IP, is working from inside containers. Does it? |
|
yes |
|
Any chance you can either a) give me access to that machine, or b) tell me how to create a VM with the exact same configuration as you have? |
|
If you have teamviewer, |
|
tcpdumping docker0 while running made things work! So the question is a) why are those rules there, and b) how exactly do they break things. |
rade
changed the title
|
Even kubernetes, a project to which RH contributes, currently requires firewalld to be disabled. See: https://access.redhat.com/articles/1198103 While firewalld does seem to have support for adding arbitrary iptables rules (https://fedoraproject.org/wiki/FirewallD#Direct_options), that suggests you invoke So the firewalld cure seems worse than the disease right now. If RHEL insists on going down that road, maybe we'll have to do it at some point. But it looks unpleasant, and I'd like to see some more projects like k8s go through the pain first. Maybe the folks inside RH who do that will be able to convince their colleagues responsible for firewalld to make it less objectionable. |
|
Ok, I just read through this: kubernetes/kubernetes#12396 In particular:
So what docker (and now k8s) have done is more like tolerating firewalld than supporting it. They are not using it in a way that yields the benefits firewalld is supposed to provide (i.e. coordination of access to iptables). But even that requires use of dbus, which is a problem for us. |
|
I'm fine with the warning, and applying a fix manually I'm just saying the lack of offering a resolution using the tools at hand (firewalld/firewall-cmd) either alongside the warning message or in your documentation is problematic In the scenario where I just want to go get up and running with Docker + Weave, it turns into a two-command install into a multi-hour foray of researching WTF firewalld is and how it interop's with iptables I'm just saying, if you know a way to manually fix this using firewalld/firewall-cmd, you should make that info easy to find Thanks, |
I don't think there is a simple manual resolution other than disabling firewalld, and I appreciate that is not very satisfying to you. It's not satisfying to me either. And I take your point that the warning message is not very specific. But the obvious fix for the would be to make it say how to disabled firewalld. |
|
Can you add the command to send to firewalld to remove the 2 offending rules? |
Set the iptables to allow for weave to work - #1266 |
We didn't find any way to ask firewalld to do this. And if you remove them with iptables, but you don't disable firewalld, then it will put them back when it restarts. |
|
If the new rules are saved, they would persist, |
|
@miztiik: Are you suggesting that weave should use firewalld to add the rules? |
|
The commands |
I don't think there is any way to do this. |
|
@dpw We can't do it automatically, but it could be put in an init script on the host, right? |
|
It means that the firewalld command is not able to remove some of the rule it manages???? |
It could. But that's not necessary a robust approach: If firewalll d restarts for any reason, it will restore the rules. (I expect that any system with firewalld is using systemd, so strictly speaking it would involve a service file, not an init script.) |
|
No I mean that there should be a way to remove permanently that rules using firewalld. |
|
@gioppoluca I think we're all agreed that it should allow these things, but unfortunately at present it appears not to. Do RedHat have anything to say on the matter? |
|
In case anyone else comes across this, and has the same issue and desire: "As someone using firewalld, I want to use weaveDNS, without disabling firewalld, and i am willing to run a manual command to set up the firewall to make it work" Here is the firewall-cmd answer: First, you select a zone, and add your weave and docker interfaces to it: Then, add the dns service to those interfaces:
when you launch weave, you will still get the message complaining about the reject rule. But in fact, it will still work because firewall-cmd has inserted a rule that will allow the DNS traffic. Hope this helps someone else! |
|
@dcowden dude still not working, weaveDNS cant resolve a name on my containers, any ideas? |
|
Hmm, sorry no other ideas... does it start working when you disable firewalld completely?
That test will tell u if it is a firewall issue or not |
|
yeah! with the firewalld is down works fine!, but in your comment of 17 MAR, you wrote that works for you ? |
|
@joserivca did you follow the exact steps @dcowden posted? |
|
@rade yes, but still appears that message '-A FORWARD -j REJECT --reject-with icmp-host-prohibited' will block name resolution via weaveDNS - please reconfigure your firewall. will block name resolution via weaveDNS - please reconfigure your firewall. this is my iptables-save Generated by iptables-save v1.4.21 on Wed Apr 20 16:42:34 2016*nat Completed on Wed Apr 20 16:42:34 2016Generated by iptables-save v1.4.21 on Wed Apr 20 16:42:34 2016*mangle Completed on Wed Apr 20 16:42:34 2016Generated by iptables-save v1.4.21 on Wed Apr 20 16:42:34 2016*security Completed on Wed Apr 20 16:42:34 2016Generated by iptables-save v1.4.21 on Wed Apr 20 16:42:34 2016*raw Completed on Wed Apr 20 16:42:34 2016Generated by iptables-save v1.4.21 on Wed Apr 20 16:42:34 2016*filter Completed on Wed Apr 20 16:42:34 2016 |
@dcowden wrote:
|
|
one note-- even in my setup, i still got the warning message saying it wouldnt work. that's because the warning message is based on the default rule, but when you make the changes I posted, you are adding rules ahead of the default deny, so it works. I dont know iptables well enough to really dive through your entire list above.. |
|
@dcowden @rade , i know about the message , thats why im tried that solution, for example [root@pmbrklnxd02 S56490]# /database/weave/weave status dns [root@hdfs-namenode-main /]# telnet 10.40.0.0 22 |
|
Note in #1266 (comment) the interface You can find this name by running: On many systems the name is |
|
OK, really thanks for the help , but my solution was put this on iptables |
I have a single weave host, with two containers running on it.
I started weave
weave launch && weave launch-dns && weave launch-proxyand then started the containers with docker commands,
docker run -d -ti -h webnode1 --name webnode1 -p 80:80 docker.io/httpd:latestdocker run -d -ti -h webnode2 --name webnode2 -p 8080:80 docker.io/httpd:latestbut i am not able to ping them from one another and then the traffic to the outside world is not passed through.
docker exec webnode1 ping -c 5 webnode2results in unknown host webnode2Running the latest version of weave 1.0.1, docker is also latest version.
The text was updated successfully, but these errors were encountered: