Best way to be notified about "everything" in the cluster

[sayden]

Hi,

We are implementing Consul on a potentially large cluster (over 500 nodes, close to 1000). We are struggling trying to figure out out to provide an efficient way to be notified about changes (node down, service down, service up...) in the cluster on some central HA nodes.

We have thought of capturing the moment that the masters are synced with the new state and publish the information through a websocket to all connected clients.

Reading here #603 and in our own research it seems pretty reasonable to make this approach in Consul code. We even manage to do it with local changes in the agent adding a small amount of logic in watch/plan.go where it handles the updated result.

The question is, did we miss something in the docs? were in the code could we do it in master? Do you have a better idea?

Thank you very much

[slackpad]

Hi @sayden you are correct that Consul doesn't provide a single, centralized feed of cluster events.

Watches under the hood are just doing blocking queries (https://www.consul.io/docs/agent/http.html) against some of the HTTP endpoints, so it is possible to write your websocket server completely outside the Consul core. Queries to the catalog and health endpoints can be done as a long poll, getting notified only when there are potential changes. Your server can consolidate these updates in any way it chooses and then pass the information along to clients. Our Atlas product has a similar implementation under the hood - it makes blocking queries against the catalog of nodes and the health of services and then exposes that to our application to power the dashboard, alerts, and things like chat notifications.

[slackpad]

Closing for now - please let me know / reopen if you have more questions.

[huyjack178]

Hi slackpad, I heard about Blocking Query very first time when I study Consul. I had just known about web socket to support watching before. I don't know the reason why Consul uses Blocking Query to watch K/V change and doesn't use websocket. Is there any comparison between them, which one is better for network?

[slackpad]

I'm not well versed enough in websockets to make a good comparison. Websocket-based systems are probably more efficient at sending an initial dump + delta updates, where Consul usually sends the whole payload back each time.