Feature for rewrite source-nat ipv4 address

[askanhesse]

Hi there!

In this feature I created an small docker container that is able to setup an iptables rule for the virtual network of mailcow_dockerized to change the source nat for outgoing request since this is currently not easily possible with docker.

The reason I created this container is, that I wanted to do on one server multiple services that not depend on each other and wanted to use multiple not depending letsencrypt acme instances.
For this approach I use different IP addresses per service, but since docker overwriting iptables rules and the DOCKER-USER chain for personal rules does not be reached in case of post-routing, so this is hard to do. To add an post-routing iptables rule right after start of the composed containers this is an easy and working solution.
For sure I could (in case of acme container) just use the SKIP_IP_CHECK parameter, but to me its a good approach to let a service use the same ip for outgoing and ingoing traffic since other services identify it by this (e.g. acme container, postfix) and I try to avoid inconsistency there.

If you don't like my idea or approach, it is for sure no problem... But I'm thinking im not the only one with this case scenario. 😄

With some feedback I adding also the code for config update.

[mkuron]

I‘d have expected that Docker supports this out of the box via an argument to docker network create. But it looks like you‘re right and there is none. In fact, there is a feature request for that: moby/moby#30053.

On the other hand, there is com.docker.network.bridge.host_binding_ipv4. Maybe that can get the job done?

Making iptables modifications while Docker is running or a host firewall is enabled can be problematic though. There may be conflicting rules, so you need to make sure that your rules are inserted in the right place.

Also, for this to be mergeable at all (which requires further discussion), it would need IPv6 support too.

[askanhesse]

Hey @mkuron!

Thanks for you answer. Yes, you're right. There are several issues about this topic, but since its already long time and not clear when it will be solved, I decided for this.

As far as I see com.docker.network.bridge.host_binding_ipv4 is in relation of the --ip flag, so for local network (of the user network under docker) binding, correct me if I am wrong. It will not change the external ip source natting. 🤔

True, I totally agree. it can in really specific cases collide with other natting rules - thats why the script will not be enabled by default. In this case an user can choose to use this contained script and modify/define custom files or define iptables that work for his case.

To add IPv6 support should be possible. Will do that when I get some time really soon 😄

[mkuron]

According to the documentation, com.docker.network.bridge.host_binding_ipv4 is supposed to set the IP for exposed ports to listen on. Because replies need to be sent back out over the same IP, I guess this needs source NAT too. Have you tried it? If it doesn’t do that, maybe adding that feature to Docker itself would be better. You need to have a script like the one in this pull request for every one of your services, so it would be nicer to solve it centrally.

[askanhesse]

Okay, I see what you mean. Then this does not work out, I tied it already - It does not set SNAT rules.
I also bound all the ports already on a fixed IP instead of default 0.0.0.0 and got the same result.

Sure, I totally agree this should be an docker feature - but there is no integration planned to far and I am not sure about integrating it by myself. As long this is not solved this feature could be more an temporary solution - but an working one that is:

• much less complicated on setting up and SNAT for an user of this project (e.g. on every restart of the cluster it would get overwritten by docker auto-generated rules)
• it is consistent in one place with the configuration and not somewhere on the host
• it can be replaced by native feature really fast when it comes up (config parameters can be reused)

I agree on your points, but still think this feature could be an good idea. I would like to get an decision out for this. 🎱

[askanhesse]

Any decision on that? @mkuron / cc @andryyy

[mkuron]

Definitely needs IPv6 support. Also you should remove the depends_on From the docker-compose.yml as it‘s fine if the rules are added after the other containers are brought up.

Is there anyone else here who needs this feature? I‘m a bit reluctant on adding more iptables stuff to mailcow as the intoduction of ipv6nat and fail2ban also came with some conflicts with host firewalls that needed to be sorted out...

[askanhesse]

Hey there! I added ipv6 support. I still left one depends_on at position of mailcow_acme, since there is the possibility that acme is ready before the natting container as docker-compose starting containers randomly. This would lead into certificate pulling errors.
I'm even not that sure if it is smart to not have it at the mailcow_postfix container, since SPF-rules need to contain the servers main IP also in strict mode, when there is still a mail in the queue when the container is started and you don't want it to be dropped.

Otherwise, what would you say on that @mkuron?

[lenovouser]

If I remember correctly there were some issues with not being able to resolve DNS if you use SNAT (see #302 (comment)) - it seems like @Ringelnatz has found a solution though, so you might want to incorporate that. I haven't tested if it works though.

[askanhesse]

Could be an firewall issue there. I just tried his scenario and it worked out for me fine and there are no issues at all.
I had something kind of related at #497, but I can only guess on this.

Otherwise, you are right. His script can detect container states and can trigger re installation of the rules, but I am not sure if this is needed since iptables rules will only be touched when docker compose up/start/restart/stop triggered.

Edit: So would you like to also filter 53/tcp and 53/udp out of this? My current iptables rules will only affect outgoing traffic for an network interface (no changes for unbound needed e.g.).

[askanhesse]

Hey @mkuron, anything new yet? Will it be accepted or dropped? 😄

[mkuron]

Well, that's for @andryyy to decide, he's the boss. Since nobody else chimed into this discussion, saying that they need this feature, I'm currently not so much in favor of it. Its use will almost certainly interact with firewalls in ways we don't anticipate and it will take more effort down the road to work that out. I suspect this is a rather niche feature because nowadays the IPv4 address is about as expensive as the cheapest virtual server you can get... Thus few people have the luxury of not having to share an IPv4 address between multiple services.

Since your patch is pretty much independent of the rest of mailcow (the only existing file it modifies is docker-compose.yml, and you could put those changes into a docker-compose.override.yml), I suppose it's not too difficult for you to keep your local Mailcow running with the patch until we decide whether we want to merge it.

[askanhesse]

Okay, also fine for me. Thats how I currently does it. I was just wondering if anybody else could need that. 😄

[mkuron]

Ok great. And just to be clear, we really appreciate your contribution!

[Ringelnatz]

I personally ordered an additional IPv4 address dedicated for mailcow and then started figuring out how to tell docker which one is the IP address i want for hours. In my opinion the possibility to specify the outgoing IP address is a really important feature for a mail server. If the address mails are sent from and the one that is specified by rDNS/SPF/... do not match, mails quickly are classified as spam.

+1 for this PR until docker provides this directly.

I'm even not that sure if it is smart to not have it [the depends_on option] at the mailcow_postfix container, since SPF-rules need to contain the servers main IP also in strict mode, when there is still a mail in the queue when the container is started and you don't want it to be dropped.

Also +1 for that.

Its use will almost certainly interact with firewalls in ways we don't anticipate and it will take more effort down the road to work that out.

How about providing this as an opt_in feature which has to be enabled in the config? And explaining possible side effects regarding the firewall in the documentation?

[askanhesse]

@Ringelnatz Thanks for that.

How about providing this as an opt_in feature which has to be enabled in the config? And explaining possible side effects regarding the firewall in the documentation?

Currently it already is optional since the commands only will be runned when the flag in config file is enabled (default its disabled). I would avoid to don't run the container itself since the depends_on flag should wait for this container to let the outgoing IP be for 100% sure consistent.

Otherwise I definitely support the suggestion to add some documentation there, sure 👍

If this is going to be merged by the maintainers here, I would do the changes. But I wait for some feedback first.

[extremeshok]

Any possibility this can be resurrected to support multiple outgoing ip's for email ?

We have extremely clean IP's (more than 4years) and want to keep them rotated to keep the reputations
active.

[mkuron]

You should be able to just change the address in the mailcow.conf file and do docker-compose up -d. You can easily automate that with a cronjob on the host.

[andryyy]

And if you start to send spam (for whatever reason), all four IPs are not that clean anymore.

This is not difficult, but mailcow shouldnt really be used to handle more complex NAT. Don’t know...

[slintes]

Hi, I'd also like to have this feature. My usecase:
I migrated today from a bare metal server at Hetzner to one of their new cloud servers. In order to be flexible (and that helped during migration too) I'm also using one of their "floating ip"s, which is an ip address which can dynamically be assigned to a cloud server. I use this ip in DNS configurations.
The problem now was: when I send a mail, the receiving mail servers compared the ip address of the DNS entry of my mailserver (which is the floating ip) with the actual ip address of the sending server (the server's main ip address). The mismatch resulted in rejected mails.
I solved this by doing sth similar like this PR, following instructions on https://medium.com/@havloujian.joachim/advanced-docker-networking-outgoing-ip-921fc3090b09 (which I found earlier than this one).

PS: yes, this should be a docker feature, but I don'see any progress in related issues, e.g. moby/moby#30053

[andryyy]

You can set SNAT_TO_SOURCE in mailcow.conf to set an outgoing IP. Everything else is still just common networking. Not sure if it should be even a task of Docker itself.

mailcow shouldn’t fit every custom network config. Sometimes you just need some lines of iptables magic.

[slintes]

wasn't aware of SNAT_TO_SOURCE, thanks. Will give it a try when I update my mailcow :)

[andryyy]

I think it is missing in the docs. We need to add so much more. :-(

…

Am 28.04.2018 um 21:53 schrieb Marc Sluiter ***@***.***>: wasn't aware of SNAT_TO_SOURCE, thanks. Will give it a try when I update my mailcow :) — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub, or mute the thread.

[DipenduDas73]

Multiple IPs are needed if one ISP is down, then we can send email from another IP, this is how actual server work I think.