Swarm cluster with high latency due to overlay network

[amitripshtos]

Hey ! Thanks for doing AMA.
I saw that you have some experience with docker swarm, and I want to share my story with it and see if I'm missing something..

I deployed my application on a swarm (physical servers) and at the beginning everything went well.
After a nightmare of a week, I had the following issues:

1. The overall latency got bigger , especially in a latency-sensitive application (with about 100-150K RPMs on peaks). I guess that's because IPVS and the load balancing between servers.
2. I had postgres database attached to the manager node, and the other services connected to it. I had disconnections while the TCP socket was alive - I know now that's because timeout issues in IPVS (it's a bug but nobody calls that and it's weird) so I adjusted the kernel paramaters and it should have fix that issue.

So after having these kind of issues (with postgres and rabbitmq for IPVS timeout issues) and latency issue (IPVS load balancing) I ditched it and came back to manually deploying the application using docker compose and bridge networking.

I'm looking at the issues at the libnetwork quite freqenly, and I know that overlay network is not mature enough to work well (as well as the IPVS thingy with the load balancing)

My questions are:

1. Do I still have any kind of hope with docker swarm before I move to kubernetes ? I don't really want to add more work to myself using kubernetes , but I can use calico network driver which is more fast.
2. Why docker does not have any kind of benchmark for their network stacks ? It's like you can't find any performance benchmarks anywhere.
3. What do you suggest me to do in that case ? I got burned quite hard for that week in docker swarm, but I do want some kind of clustering.

Thanks,
Amit.

[BretFisher]

There are plenty of people using Swarm full time, but I think the "problem" is that it's so easy to use, people just default to overlay with VIP and then think any limits around that are "core swarm issues" rather than just a lack of education around when to use what networking features. So thanks for reaching out:

1. If you're not running multiple backend tasks in a single swarm service, then there is no reason to use VIP. Whenever an app needs to talk to a database or key/value service, which is almost always a single-task-per-service, use the DNSRR endpoint mode --endpoint-mode=dnsrr. Then you won't have to deal with VIP/IPVS in the middle or any of its timeout settings.
2. Teams I know use overlay with VIP fine for hundreds of connections a minute or more. I'm not sure the upper limit of its scalability. You should be aware of the 10-30% performance hit overlay+VIP will give you. It gives you ease-of-use out of the box (a docker design principle) and trade in some top-level performance, so don't use slow networking on AWS (many instances are sub-1Gb networking), which I don't recommend for prod latency-sensitive systems anyway. I'd much rather pay for 10G networking in big projects and use big fast servers to get over the overlay cost then try to squeeze every ounce out of "slow networking" on cheaper servers. In general with orchestration, it's better to scale out to 5-10 servers and then start scaling up to gain faster processors, I/O, and networking if those are required.
3. You can always use bridge, host, macvlan, or even plugins like weave networking drivers in Swarm if overlay just won't work for your requirements.
4. Don't use IPSec (encrypted overlay), which I think adds 3-10% perf hit.

I hope that helps.

[amitripshtos]

Thank you for your comment, I really like that AMA concept and I feel this may help for people over the internet ! :)

I want to respond to some of the good points you mentioned:

1. At some point during the "hell" week, I updated the postgres and rabbitmq services to run as dnsrr, which made things better. But, I wanted the postgres to publish it's port to outside world, so I can deploy services outside the swarm with that postgres. So currently the only solution is to configure sysctl to work well with the hosts (the sysctl PR per-service is about to come out so it's a good thing)
2. How would you recommand me to scale my network-sensitive services (in the case of docker swarm) ? Thing on them like game-servers: they are latency sensitive and burst data. I though about either deploy them in bridge mode using HAProxy (so each physical host will have a bridge network with the services and one HAProxy) or use MACVLAN (I'm not sure how it works yet and what's the right way to use it).
3. Does docker stack supports deploying services that are in the swarm but outside of the overlay network?

Thanks for your time !!!

[BretFisher]

On your points:

1. I don't see what --endpoint-mode=dnsrr has to do with port publishing. You can publish the port just like with VIP mode.
2. A. this is beyond what I could advise here. If overlay options don't work and you're talking about azure/aws clouds, then either host network driver or bridge is the next option. host will be the fastest, and you can maybe team it with Docker for AWS/Azure to auto-update a front-end LB like AWS ELB. There's lots of combo's based on requirements. B. macvlan doesn't use NAT, and gives each container a IP on the same network as the host, so you usually can't do this in the cloud, where they don't allow you to set multiple IP's to the real host NIC. It's great for datacenters where you can control your IP subnets.
3. absolutely. Overlay is just the default network driver, but many use other options.