SRV records that point to names that don't resolve to the service's IP

[treed]

I'm having an issue where:

• Software is using a SRV record via consul to find a service, which returns the appropriate port for the service, and the FQDN for the relevant agent's node. The additional info section includes a record for that name, mapping to the service's IP.
• The software is disregarding the additional info and doing a lookup for the FQDN itself, which resolves to a totally different IP, which is not the one the service is listening on.

It seems reasonable to me that SRV records could respond with an FQDN that leads to the service itself, although I'm not sure how that would look.

I'm also unsure if the behavior of this software (srv-router in this case) is typical, but it would not surprise me if this came up in other instances.

[armon]

I guess I'm a bit confused here. We are already returning the A record of the node along with the SRV record. Is the issue that the service address is different than the node address?

[treed]

The issue is related to the fact that the service address is different from the node address.

The SRV response includes the service address under the node's name, but in this instance I have software (srv-router) that ignores that and just does a separate lookup for the node address returned by the SRV request, which is not correct for the service.

It's possible that this is a bug in how srv-router is handling the lookup, but it would not surprise me if this were a common thing in other software too, so it's probably better to handle this more systematically.

One thought occurred to me for something like having the SRV response change to <service-id>.<node>.node.consul and to have that map directly to the service IP, but I don't know if maybe there's something else I could be doing or if you have better ideas for how to handle this.

[treed]

Here's some output that illustrates what I'm talking about.

% dig kibana-http.service.consul SRV

; <<>> DiG 9.9.5-3ubuntu0.2-Ubuntu <<>> kibana-http.service.consul SRV
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 33446
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; QUESTION SECTION:
;kibana-http.service.consul.    IN      SRV

;; ANSWER SECTION:
kibana-http.service.consul. 0   IN      SRV     1 1 5601 core-01.node.dc1.consul.

;; ADDITIONAL SECTION:
core-01.node.dc1.consul. 0      IN      A       10.200.0.26

;; Query time: 1 msec
;; SERVER: 10.200.0.1#53(10.200.0.1)
;; WHEN: Mon Mar 30 17:05:03 PDT 2015
;; MSG SIZE  rcvd: 152

% dig core-01.node.dc1.consul

; <<>> DiG 9.9.5-3ubuntu0.2-Ubuntu <<>> core-01.node.dc1.consul
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 58845
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;core-01.node.dc1.consul.       IN      A

;; ANSWER SECTION:
core-01.node.dc1.consul. 0      IN      A       172.17.8.101

;; Query time: 2 msec
;; SERVER: 10.200.0.1#53(10.200.0.1)
;; WHEN: Mon Mar 30 17:05:51 PDT 2015
;; MSG SIZE  rcvd: 80

When I try to use srv-router to get to the kibana-http service, it tries connecting to 172.17.8.101:5601, rather than 10.200.0.26:5601

It seems like the answer shouldn't change depending on how the user asks.

[gbelur]

The node name consul returns in the answer section represents the consul agent and not the domain name of the host running the service. Shouldn't that be the case?

[treed]

IMO, it should be the case that it returns a name representing the service itself, which might (and in this case does) have a different IP for various reasons.

In the original answer, it gives this IP for that node's hostname in the additional info section, which is... kinda right:

core-01.node.dc1.consul. 0 IN A 10.200.0.26

But then if anything else asks for that hostname in other contexts, it gets the node's actual IP:

core-01.node.dc1.consul. 0 IN A 172.17.8.101

I think it would be better to have services resolve to a hostname representing the service, which would always resolve to the service's IP.

[armon]

@treed Sorry about the delay. I agree. I guess for the SRV lookups we should use a special FQDN that is
distinct from the node lookup to avoid the ambiguity. The issue as you said is that the IP address depends on the lookup context (node -> agent IP, service -> service IP).

I'll mark this as a thinking bug.

[prisamuel]

I'm seeing the exact same behaviour with Consul - also using the above mentioned srv-lb for doing dns lookup.

bash-4.3# dig mt-content-blogs.service.consul. SRV

;; QUESTION SECTION:
;mt-content-blogs.service.consul. IN    SRV

;; ANSWER SECTION:
mt-content-blogs.service.consul. 5 IN   SRV 1 1 9494 consul-server.economist.local.node.dc1.consul.

;; ADDITIONAL SECTION:
consul-server.economist.local.node.dc1.consul. 5 IN A 172.18.0.4

but the subsequent A record lookup returns

bash-4.3# dig consul-server.economist.local.node.dc1.consul. A

;; QUESTION SECTION:
;consul-server.economist.local.node.dc1.consul. IN A

;; ANSWER SECTION:
consul-server.economist.local.node.dc1.consul. 0 IN A 172.18.0.5

which is different from the A record returned in the 'ADDITIONAL SECTION' in the first SRV lookup.

[foxel]

Any update on this? Really disappointing bug.

Why not return service address in SRV answer section?

[weiwei04]

Any update on this?

[piotrkowalczuk]

This bug makes consul unusable with go's net.LookupSRV.

[perplexes]

This also makes consul somewhat unusable with node's native dns.resolveSrv

[slackpad]

A simple fix for this would be something like a new Consul DNS handler for names like 10.200.0.26.ipv4.addr.consul. which returns the corresponding A or AAAA record (for .ipv6.addr.consul.). This would be simple and not need to tie back to the original service in any way.