Active health checks do not support https

[AndyBan]

I have reverse proxy setup going to a couple of https only local services however the active health check doesn't seem to respect the tls flag.

Route:

{
              "match": [],
              "handle": [
                {
                  "handler": "reverse_proxy",
                  "transport": {
                    "protocol": "http",
                    "tls": {
                      "insecure_skip_verify": true
                    },
                    "keep_alive": {
                      "enabled": true
                    }
                  },
                  "health_checks": {
                    "active": {
                      "path": "/",
                      "expect_status": 200
                    }
                  },
                  "load_balancing": {
                    "selection_policy": {
                      "policy": "least_conn"
                    }
                  },
                  "upstreams": [
                    { "dial": "localhost:8030" },
                    { "dial": "localhost:8031" }
                  ]
                }
              ]
            }

2020/05/26 10:20:44.998 �[34mINFO�[0m   http.handlers.reverse_proxy.health_checker.active       HTTP request failed
        {"host": "localhost:8030", "error": "Get \"http://localhost:8030/\": EOF"}

Have I missed something or perhaps this check needs improving?

https://github.com/caddyserver/caddy/blob/master/modules/caddyhttp/reverseproxy/healthchecks.go#L201

[fr33tux]

Hello,

I face a similar issue here, where I am unable to use the reverse proxy active healthcheck properly when https is enabled between the RP and its backend.

I have the following setup, where the Caddy Load balancer is exposed by HTTPS and talks to its upstreams servers using HTTPS. It needs to make a health check on a specific endpoint (/v1/api), and overwrite the Host header.
Then the group of Caddy Reverse Proxy also listens on HTTPS, and proxy the requests to their local service, running locally:

                                                                                                                           +----------------------+
                                                                                                                           |                      |
                                                                                                                           |                      |
                                                                              +-----------------------------+    +-------->+        Service       |
                                                                              |                             |    |         |                      |
                                                                              |                             |    |         |                      |
                                                                              |      Caddy Reverse Proxy    |    |         +----------------------+
                                                                              |     reverse_proxy 127.0.0.1:3030 {
                                                       +--------------------->+         health_path /v1/api +----+
                                                       |                      |         health_interval 15s |
                                                       |                      |         health_timeout 5s   |
                                                       |                      |         health_status 200   |
                                                       |                      |                             |
+---------------------------------+                    |                      +-----------------------------+
|                                 |                    |
|                                 |                    |
|     Caddy load balancer         |                    |
|                                 |                    |
|                                 |                    |                      +------------------------------+             +----------------------+
|                                 |                    |                      |                              |             |                      |
|                                 |                    |                      |    Caddy Reverse Proxy       |             |                      |
|        reverse_proxy {          +-----------------------------------------> |                              |       +---->+        Service       |
|         to https://rp1:8080     |                    |                      |     reverse_proxy 127.0.0.1:3030 {   |     |                      |
|         to https://rp2:8080     |                    |                      |         health_path /v1/api  +-------+     |                      |
|         to https://rp3:8080     |                    |                      |         health_interval 15s  |             +----------------------+
|                                 |                    |                      |         health_timeout 5s    |
|        lb_policy least_conn     |                    |                      |         health_status 200    |
|        health_path /v1/api      |                    |                      |                              |
|        health_interval 15s      |                    |                      +------------------------------+
|        health_timeout 5s        |                    |
+--------header_up Host {http.reverse_proxy.upstream.hostport}
                                                       |
                                                       |
                                                       |
                                                       |                      +-------------------------------+
                                                       |                      |                               |            +----------------------+
                                                       |                      |       Caddy Reverse Proxy     |            |                      |
                                                       |                      |                               |            |                      |
                                                       |                      |      reverse_proxy 127.0.0.1:3030 {+------>+        Service       |
                                                       +--------------------> |          health_path /v1/api  |    |       |                      |
                                                                              |          health_interval 15s  |    |       |                      |
                                                                              |          health_timeout 5s    +----+       +----------------------+
                                                                              |          health_status 200    |
                                                                              |                               |
                                                                              |                               |
                                                                              |                               |
                                                                              +-------------------------------+

1. It works fine between the Caddy Reverse Proxy and their services in http.
2. It works fine between the Caddy load balancer and the services (directly in http, by bypassing the group of Caddy Reverse proxy)
3. It doesn't work between the Caddy Load Balancer and the group of Caddy Reverse Proxy in https when Active health check is enabled. It looks like the requests made by the active health check don't benefit the header manipulation directives, so the host header they send is incorrect, throwing an unhealthy status. The connection between the load balancer and the reverse proxy works fine if I disable the active health check.

Build: https://github.com/caddyserver/caddy/releases/tag/v2.0.0

Hopefully that helps,

[francislavoie]

Are you able to compile from source? Could you try a couple things?

Compile from the latest on master and see if it works first, we've made a handful of changes slated for v2.1 that may or may not already resolve it.

If not, could you try to make this change:

https://github.com/caddyserver/caddy/blob/aef560c/modules/caddyhttp/reverseproxy/healthchecks.go#L239-L242

Change this section to this, to show a bit more info. This should confirm whether the line you linked to is properly setting the scheme to HTTPS.

		h.HealthChecks.Active.logger.Info("HTTP request failed",
			zap.String("scheme", scheme),
			zap.String("host", hostAddr),
			zap.String("path", h.HealthChecks.Active.Path),
			zap.Error(err),
		)

If it's not showing the scheme as https, then I think you could try to change the line you linked to this (frankly I don't think this shouldn't be any different, but it's a hunch):

	if ht, ok := h.Transport.(TLSTransport); ok && ht.TLSEnabled() {

Another thing to try would be to disable keep_alive in your config (set it to false). I'm reading that some people had issues running into EOFs when using KeepAlive: https://stackoverflow.com/questions/17714494/golang-http-request-results-in-eof-errors-when-making-multiple-requests-successi

[AndyBan]

The code change fixed it, sent pull

[francislavoie]

I'll keep this open until it's merged. Thanks!