EC2 discovery doesn't work when master not running on port 9300

[ejain]

I've had a long-running, hard to reproduce problem where an elasticsearch cluster would sometimes get into a state where new nodes would no longer see existing nodes. The only solution would be to first shut down all nodes, then bring them back up.

It now looks like this happens when I run both a data node and a client-only node on each machine, and the (master) data node ends up bound to port 9301 instead of 9300.

This is not a firewall issue as I have ports 9200-9400 open, and can telnet to port 9301 on the machine running the master node from any other machine in the cluster.

[dadoonet]

Hi Eric

I can confirm what you are saying.

Basically, we limit the discovery to the first port in a given range. So 9300 here. You can see that here: https://github.com/elasticsearch/elasticsearch-cloud-aws/blob/master/src/main/java/org/elasticsearch/discovery/ec2/AwsEc2UnicastHostsProvider.java#L163

One thing we could probably do is to increase a bit the number of ports we should try to ping but without trying to scan all ports in that range.
I think that having more than 3 elasticsearch nodes on the same physical instance does not really make sense.

So we could think of changing the code to something like:

TransportAddress[] addresses = transportService.addressesFromString(address);
// we only limit to 3 addresses, makes no sense to ping 100 ports
for (int i = 0; (i < addresses.length && i < 3); i++) {
        logger.trace("adding {}, address {}, transport_address {}", instance.getInstanceId(), address, addresses[i]);
        discoNodes.add(new DiscoveryNode("#cloud-" + instance.getInstanceId() + "-" + i, addresses[i], Version.CURRENT));
}

What do you think?

BTW we have the same concern with other cloud plugins.

[ejain]

Should default to at least 2 (to handle a standalone instance plus a client running on the same machine), and perhaps make this value configurable? Also, would be good to have some mention of this in the docs!

[kimchy]

we discussed on elasticsearch side to change the value of UnicastZenPing.LIMIT_PORTS_COUNT to 2, it make it better out of the box. I think its better to do it on that end. Also, the way we build the list of unicast nodes should change in that case, where we should have all the 9300 first, and the 9301 later

[ejain]

Is there already an open issue for changing the default value of UnicastZenPing.LIMIT_PORTS_COUNT (or making it configurable), or should I create one?

[dadoonet]

@ejain I just opened elastic/elasticsearch#7090 to track this.

[ejain]

Thanks, closing this issue... Are there other issues that need to be resolved before this plugin can be released for elasticsearch 1.3?

[jerry4]

I already asked in elastic/elasticsearch#7090, but is there any reason that this hasn't been merged in? I'd really like to have the limit_port_count configurable (and don't want to fork this plugin).