command/agent: Add simple HTTP check type

[nicholascapo]

We are running 20+ services that all register using consul. Consul then checks each service using curl every second (1s). On some of our lower power (dev and test) machines we are seeing a lot of CPU load from consul forking curl 20 times per second.

Add a simple HTTP check type that makes an HTTP GET request every Interval to the specified URL.
The status of the service depends on the HTTP Response Code.
200 is passing, 503 is warning and anything else is failing.

An HTTP based check looks like this:
{
"check": {
"id": "api",
"name": "HTTP API on port 5000 with a /health route",
"http": "http://localhost:5000/health",
"interval": "10s"
}
}

[blalor]

I believe this type of check was considered in the early days of consul and rejected. I forget on what grounds. I would personally like to see it added. I ended up writing a utility to poll http end points and submit check results to the agent.

[armon]

I think is a good idea. @blalor it's possible this was rejected early on, but I've had a change of heart on it.
Some feedback I have:

• Any 2XX status code should be treated as OK
• Let's use 429 "Too Many Requests" for WARN
• Anything else is critical.
• Tests for the new functionality would be great :)
• Super pedantic, some log statements are missing the appropriate "agent: " prefix.

[armon]

My reasoning for not using 503 is that in many application servers already implement to indicate various critical conditions. e.g. Phusion or uWSGI failing to start the underlying Rails/Django app causes 503. Using 429 requires an app to actually implement it since it's generally not a builtin default, so you cannot accidentally get into the warning state.

[sean-]

(small request: the ability to see the entire HTTP Request/Response so it can be compared or replayed with telnet(1)).

[ryanuber]

I agree with @armon on using a 429 for warnings. It also agree with @sean- that writing the response into the output field would be helpful, and it's probably not too complicated to implement. Looking good!

[nicholascapo]

Fixed: Any 2XX status code should be treated as OK
Fixed: Let's use 429 "Too Many Requests" for WARN
Fixed: Anything else is critical.
Fixed: Tests for new functionality
Fixed: Super pedantic, some log statements are missing the appropriate "agent: " prefix.
Fixed: HTTP Request/Response in output

Example debug log:
2015/01/13 00:28:24 [DEBUG] agent: http check 'service:foo' is passing: HTTP GET http://127.0.0.1:60522/health: 200 OK Output: {"JSON data from service"}

Example from http://localhost:8500/v1/health/state/passing
{
"Node": "bar",
"CheckID": "service:foo",
"Name": "Service 'foo' check",
"Status": "passing",
"Notes": "",
"Output": "HTTP GET http://127.0.0.1:37728/health: 200 OK Output: {"JSON data from service"}",
"ServiceID": "foo",
"ServiceName": "foo"
}

[armon]

LGTM! Thanks!