Consul does not initiate RPC connections from the 'bind' address or support 'source' address

[rileyje]

consul version for both Client and Server

Server: Consul v0.7.5
Client: NA

consul info for Server

Server:

agent:
        check_monitors = 0
        check_ttls = 0
        checks = 0
        services = 1
build:
        prerelease =
        revision = '21f2d5a
        version = 0.7.5
consul:
        bootstrap = false
        known_datacenters = 3
        leader = false
        leader_addr = 172.18.1.69:8300
        server = true
raft:
        applied_index = 11568
        commit_index = 11568
        fsm_pending = 0
        last_contact = 56.345518ms
        last_log_index = 11568
        last_log_term = 2
        last_snapshot_index = 8192
        last_snapshot_term = 2
        latest_configuration = [{Suffrage:Voter ID:172.18.1.69:8300 Address:172.18.1.69:8300} {Suffrage:Voter ID:172.18.1.133:8300 Address:172.18.1.133:8300} {Suffrage:Voter ID:172.18.1.5:8300 Address:172.18.1.5:8300}]
        latest_configuration_index = 1
        num_peers = 2
        protocol_version = 1
        protocol_version_max = 3
        protocol_version_min = 0
        snapshot_version_max = 1
        snapshot_version_min = 0
        state = Follower
        term = 2
runtime:
        arch = amd64
        cpu_count = 8
        goroutines = 62
        max_procs = 8
        os = linux
        version = go1.7.5
serf_lan:
        encrypted = true
        event_queue = 0
        event_time = 4
        failed = 0
        health_score = 0
        intent_queue = 0
        left = 0
        member_time = 3896
        members = 6
        query_queue = 0
        query_time = 1
serf_wan:
        encrypted = true
        event_queue = 0
        event_time = 1
        failed = 0
        health_score = 0
        intent_queue = 0
        left = 0
        member_time = 13
        members = 9
        query_queue = 0
        query_time = 1

Operating system and Environment details

CentOS Linux release 7.3.1611 (Core)
Linux ip-172-18-1-175 3.10.0-514.10.2.el7.x86_64 #1 SMP Fri Mar 3 00:04:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc pfifo_fast state UP qlen 1000
    link/ether 12:f6:c5:ab:61:f6 brd ff:ff:ff:ff:ff:ff
    inet 172.18.1.175/26 brd 172.18.1.191 scope global dynamic eth0
       valid_lft 3200sec preferred_lft 3200sec
    inet6 fe80::10f6:c5ff:feab:61f6/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc pfifo_fast state UP qlen 1000
    link/ether 12:63:ed:78:44:74 brd ff:ff:ff:ff:ff:ff
    inet 172.18.1.133/26 brd 172.18.1.191 scope global dynamic eth1
       valid_lft 2407sec preferred_lft 2407sec
    inet6 fe80::1063:edff:fe78:4474/64 scope link
       valid_lft forever preferred_lft forever

Description of the Issue (and unexpected/desired result)

When the -bind ip_addr argument is passed to Consul, Consul still creates RPC connections to other servers from the default host address. For multi-homed hosts behind firewalls, especially if the default interface is dynamically configured, it would be useful if Consul initiated connections from the 'bind' address or supported a 'source' address option.

Reproduction steps

Create a multi-server Consul cluster where at least one Consul server has multiple IP addresses on the same subnet. Pass Consul the '-bind ip_addr' argument where ip_addr is the second/non-default host address. On a non-leader server, query local Consul agent and observe network traffic to leader from default IP.

Command line:

consul agent -bind 172.18.1.133 -node ip-172-18-1-133 -config-dir /opt/consul/consul.d -datacenter aws-us-east-1

Config:

{
  "data_dir": "/opt/consul/data",
  "disable_update_check": true,
  "dns_config": {
    "allow_stale": true,
    "max_stale": "5s"
  },
  "log_level": "warn"
  "server": true,
  "bootstrap_expect": 3,
  "telemetry": {
    "dogstatsd_addr": "127.0.0.1:8125"
  }
  "encrypt": "<consul keygen>"
}

[centos@ip-172-18-1-175 ~]$ sudo ss -tulna |grep 8300
tcp    LISTEN     0      128    172.18.1.133:8300                  *:*
tcp    ESTAB      0      0      172.18.1.175:38012              172.18.1.69:8300
tcp    ESTAB      0      0      172.18.1.133:8300               172.18.1.94:48244
tcp    ESTAB      0      0      172.18.1.133:8300               172.18.1.122:53730
tcp    ESTAB      0      0      172.18.1.133:8300               172.18.1.122:53736

[slackpad]

Hi @rileyje thanks for opening an issue. It seems like we should be able to use the bind address so we don't need to introduce any additional configuration.

[magiconair]

Tested this as follows:

Vagrantfile

Vagrant.configure("2") do |config|
  config.vm.define "consul1" do |c|
    c.vm.box = "ubuntu/trusty64"
	c.vm.hostname = "consul1"
    c.vm.box_check_update = false
    c.vm.network "private_network", ip: "192.168.33.11"
    c.vm.network "private_network", ip: "192.168.33.111"
  end
  config.vm.define "consul2" do |c|
    c.vm.box = "ubuntu/trusty64"
	c.vm.hostname = "consul2"
    c.vm.box_check_update = false
    c.vm.network "private_network", ip: "192.168.33.12"
    c.vm.network "private_network", ip: "192.168.33.112"
  end
  config.vm.define "consul3" do |c|
    c.vm.box = "ubuntu/trusty64"
	c.vm.hostname = "consul3"
    c.vm.box_check_update = false
    c.vm.network "private_network", ip: "192.168.33.13"
    c.vm.network "private_network", ip: "192.168.33.113"
  end
end

build consul

GOOS=linux GOARCH=amd64 go build -tags consul -o ~/vagrant/consul-3node-cluster/consul

consul nodes

vagrant@consul1:~$ /vagrant/consul agent -server -data-dir data -bootstrap-expect 3 -bind 192.168.33.111
vagrant@consul2:~$ /vagrant/consul agent -server -data-dir data -join 192.168.33.111 -bind 192.168.33.112
vagrant@consul3:~$ /vagrant/consul agent -server -data-dir data -join 192.168.33.111 -bind 192.168.33.113

On consul2 before fix

vagrant@consul2:~$ ss -tulna | grep 8300 | grep ESTA
tcp    ESTAB      0      0          192.168.33.12:57646    192.168.33.111:8300
tcp    ESTAB      0      0         192.168.33.112:8300      192.168.33.13:39525
tcp    ESTAB      0      0          192.168.33.12:57649    192.168.33.111:8300
tcp    ESTAB      0      0          192.168.33.12:54823    192.168.33.113:8300
tcp    ESTAB      0      0         192.168.33.112:8300      192.168.33.11:40555
tcp    ESTAB      0      0          192.168.33.12:54824    192.168.33.113:8300

On consul2 with fix

vagrant@consul2:~$ ss -tulna | grep 8300 | grep ESTA
tcp    ESTAB      0      0         192.168.33.112:33024    192.168.33.111:8300
tcp    ESTAB      0      0         192.168.33.112:8300     192.168.33.111:45034
tcp    ESTAB      0      0         192.168.33.112:8300     192.168.33.111:46201