Is Caravan compatible with consul 1.7+

[commarla]

HI,

We just upgraded from consul 1.6.4 to 1.9.1 and caravan is not working anymore with this error :

11:04:27.414 [warn]  [libcluster:caravan] unable to connect to :"conversations-xmpp-tech-4370@0a3e155b.addr.dev-eu_de_1.consul"
11:04:34.418 [warn]  [libcluster:caravan] unable to connect to :"conversations-xmpp-tech-4370@0a3e172b.addr.dev-eu_de_1.consul"
11:04:39.423 [error] 
** Cannot get connection id for node :"conversations-xmpp-tech-4370@app-10-62-21-91.node.dev-eu_de_1.consul"

I have done some digging on the consul side and found they have changed the SRV response format hashicorp/consul#7360 and hashicorp/consul#7360

Instead of returning the node fqdn app-10-62-21-91.node.dev-eu_de_1.consul. it's returning this 0a3e172b.addr.dev-eu_de_1.consul.

And it broke Caravan.

[uberbrodt]

Interesting! I'll take a look and see what we can do to support both formats

…

On Thu, Dec 24, 2020 at 1:12 AM Laurent Commarieu ***@***.***> wrote: HI, We just upgraded from consul 1.6.4 to 1.9.1 and caravan is not working anymore with this error : 11:04:27.414 [warn] [libcluster:caravan] unable to connect to ***@***.***_de_1.consul" 11:04:34.418 [warn] [libcluster:caravan] unable to connect to ***@***.***_de_1.consul" 11:04:39.423 [error] ** Cannot get connection id for node ***@***.***_de_1.consul" I have done some digging on the consul side and found they have changed the SRV response format hashicorp/consul#7360 <hashicorp/consul#7360> and hashicorp/consul#7360 <hashicorp/consul#7360> Instead of returning the node fqdn app-10-62-21-91.node.dev-eu_de_1.consul. it's returning this 0a3e172b.addr.dev-eu_de_1.consul. And it broke Caravan. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#5>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAHO3IYJ3AQBMZH62LTQIWDSWLSXFANCNFSM4VH4LR2Q> .

[commarla]

Hi, I have linked twice the same issue, the second one is hashicorp/consul#6792

[uberbrodt]

@commarla Alright, had a chance to dig in. Basically, it was written with pre 1.7.x Consul in mind. I don't frankly understand what it's supposedly returning now. I don't have a readily available test env since switching jobs, but setting it up following this guide I seem to get the old style responses (which is confusing):

🏁cbrodt@tails [~] > dig @127.0.0.1 -p 8600 counting.service.consul srv

; <<>> DiG 9.11.25-RedHat-9.11.25-2.fc32 <<>> @127.0.0.1 -p 8600 counting.service.consul srv
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 31232
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 3
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;counting.service.consul.	IN	SRV

;; ANSWER SECTION:
counting.service.consul. 0	IN	SRV	1 1 9001 client-1.node.dc1.consul.

;; ADDITIONAL SECTION:
client-1.node.dc1.consul. 0	IN	A	172.17.0.3
client-1.node.dc1.consul. 0	IN	TXT	"consul-network-segment="

;; Query time: 1 msec
;; SERVER: 127.0.0.1#8600(127.0.0.1)
;; WHEN: Sun Jan 10 18:04:22 CST 2021
;; MSG SIZE  rcvd: 148

I'll try and figure out if there's something that I can do using DNS with the new versions of Consul, but don't know when I'll get to it. In the meantime, have you looked at libcluster_consul? Looks like it uses the HTTP API to get you the nodes. You can still use the EPMD module as long as the port is in the name short part of the name.

[commarla]

Hi @uberbrodt thanks for your answer. I'll definitely give libcluster_consul a try when I have time.

[johnnyplaydrums]

Hi @uberbrodt! Just wondering if you have any plans to make Caravan work with Consul 1.7+? We are trying to deploy an elixir service to our nomad/consul cluster and we're looking for solutions. Should we look elsewhere or do you have plans to work on it? I'd offer to do the work myself but I'm a total Elixir noob. I've checked out libcluster_consul, does that cover the same ground as caravan? Like the ability to use a dynamic port for epmd that's different for each instances of the service. When reading the docs on epmd it says All nodes in a cluster must use the same epmd port number, which wouldn't work in Nomad/Consul land since all port assignments need to be (or at least should be) dynamic. Caravan removes the need for the default epmd, but I can't tell if libcluster_consul is the same. Thanks for the help!

[johnnyplaydrums]

@commarla just wondering if consul 1.7+ support is planned or if this library is deprecated? Thank you.

[commarla]

@commarla just wondering if consul 1.7+ support is planned or if this library is deprecated? Thank you.

I am not the maintainer. I closed this issue because I didn't use consul anymore.

[johnnyplaydrums]

Got it! I'm curious, did you replace Consul with some other product, and why?

[commarla]

Got it! I'm curious, did you replace Consul with some other product, and why?

No I quit my previous job. Consul is still in use there.