Docker scheduler and service discovery with Nomad+Consul+Consult_template+haproxy+connectable

How to use nomad and consul setup a docker scheduler across cluster host as well as docker service discovery

External face loadbalancer using haproxy and consul-template

Internal discovery using Consul DNS and gliderlabs/connectable

Nomad & Consul setup

3 VMs : Docker1 Docker2 Docker3

Docker1 : Nomad server/client + Consul server/client + connectable

Docker2 : Nomad server/client + Consul server/client + connectable

Docker3 : Nomad server/client + Consul server/client + connectable

+-------------------------+   +-------------------------+   +-------------------------+
| +------+   +------+     |   | +------+   +------+     |   | +------+   +------+     |
| |Consul|   |Nomad |     |   | |Consul|   |Nomad |     |   | |Consul|   |Nomad |     |
| +------+   +------+     |   | +------+   +------+     |   | +------+   +------+     |
| +-------+ +-----------+ |   | +-------+ +-----------+ |   | +-------+ +-----------+ |
| |Haproxy| |Connectable| |   | |Haproxy| |Connectable| |   | |Haproxy| |Connectable| |
| +-------+ +-----------+ |   | +-------+ +-----------+ |   | +-------+ +-----------+ |
|    |        ^     |     |   |    |        ^     |     |   |    |        ^     |     |
|    |        |     |     |   |    |        |     |     |   |    |        |     |     |
|    |        |     |     |   |    |        |     |     |   |    |        |     |     |
|    v        |     ^     |   |    v        |     ^     |   |    v        |     ^     |
|       +-----+  +-----+  |   |       +-----+  +-----+  |   |       +-----+  +-----+  |
| +--+6379       |Cache|  |   | +--+6379       |Cache|  |   | +--+6379       |Cache|  |
| |Web  +        +-----+  |   | |Web  +        +-----+  |   | |Web  +        +-----+  |
| +-----+                 |   | +-----+                 |   | +-----+                 |
+-------------------------+   +-------------------------+   +-------------------------+

Bootstrap

$ vagrant up

Fire up a Job

$ vagrant ssh docker1

vagrant@docker1:~$ cd /vagrant/services

vagrant@docker1:/vagrant$ nomad run loadbalancer.nomad

vagrant@docker1:/vagrant$ nomad run web.nomad

View nodes & service in consul

http://192.168.0.20:8500

Get docker instance meta-data from consul

$ curl http://192.168.0.20:8500/v1/catalog/service/web-nodejs

• In two servers configuration you can use both servers to action above operations

Check job status from nomad client

vagrant@docker1:~$ nomad node-status

vagrant@docker1:$ nomad status loadbalancer vagrant@docker1:$ nomad status web

Check Haproxy config generated by consul_template

Go any of vm , like docker1 , docker 2 ,docker 3

vagrant@docker1:~$ docker exec haproxy_container_id cat /haproxy/haproxy.cfg

vagrant@docker2:~$ docker exec haproxy_container_id cat /haproxy/haproxy.cfg

vagrant@docker3:~$ docker exec haproxy_container_id cat /haproxy/haproxy.cfg

Access the web application

http://web-nodejs.192.168.0.20.xip.io/

Watch the hostname in the page , it actually a container ID

Hello Http :46847201c374

Check the internal port within web docker instance

docker exec web-nodejs_container_id nc -vz localhost 6379

beer!

Vagrant 1.8.5 Nomad 0.4.0 Consul 0.6.4