Consul does not present any client certificate when doing https healthchecks

[thomas-maurice]

consul version for both Client and Server

Server: v0.8.4

consul info for both Client and Server

Server:

agent:
	check_monitors = 0
	check_ttls = 0
	checks = 2
	services = 3
build:
	prerelease = 
	revision = f4360770
	version = 0.8.4
consul:
	bootstrap = true
	known_datacenters = 1
	leader = true
	leader_addr = 127.0.0.1:8300
	server = true
raft:
	applied_index = 19008
	commit_index = 19008
	fsm_pending = 0
	last_contact = 0
	last_log_index = 19008
	last_log_term = 30
	last_snapshot_index = 16384
	last_snapshot_term = 25
	latest_configuration = [{Suffrage:Voter ID:127.0.0.1:8300 Address:127.0.0.1:8300}]
	latest_configuration_index = 1
	num_peers = 0
	protocol_version = 2
	protocol_version_max = 3
	protocol_version_min = 0
	snapshot_version_max = 1
	snapshot_version_min = 0
	state = Leader
	term = 30
runtime:
	arch = amd64
	cpu_count = 4
	goroutines = 96
	max_procs = 4
	os = linux
	version = go1.8.1
serf_lan:
	coordinate_resets = 0
	encrypted = true
	event_queue = 1
	event_time = 30
	failed = 0
	health_score = 0
	intent_queue = 0
	left = 0
	member_time = 1
	members = 1
	query_queue = 0
	query_time = 1
serf_wan:
	coordinate_resets = 0
	encrypted = true
	event_queue = 0
	event_time = 1
	failed = 0
	health_score = 0
	intent_queue = 0
	left = 0
	member_time = 1
	members = 1
	query_queue = 0
	query_time = 1

Operating system and Environment details

Debian Sid

Description of the Issue (and unexpected/desired result)

Apparently Consul does not send a client certificate when doing an HTTPS healthcheck. I would like it to present the cert_file I specified in the configuration when performing HTTPS healthchecks.

Reproduction steps

Register an HTTPS healthcheck, the remote endpoint using a private PKI for TLS and verifying the client certs. Configure Consul for using TLS with a certificate signed by the private CA ca_file, cert_file and key_file
It will fail on the client showing
2017/08/06 17:48:19 http: TLS handshake error from 127.0.0.1:38878: tls: client didn't provide a certificate
On consul, we see :
2017/08/06 16:48:49 [WARN] agent: http request failed 'https://127.0.0.1:6663/metrics': Get https://127.0.0.1:6663/metrics: remote error: tls: bad certificate

[slackpad]

The plan for this is to add a new tls_use_agent_config Boolean option to the health check definition. If this is true, we will use the agent's configured key_file, cert_file, ca_file, and ca_path for the health check's HTTPS client, so we will present that client certificate and use the CA configuration to verify the server. This doesn't allow using different TLS settings per-check, but should be simpler to configure for the most common use case of using the agent's TLS config.

This will default to false to retain backwards compatibility (not presenting a client certificate and using the system's CA info), and will fail validation if set to true and the agent isn't configured to use TLS.

[slackpad]

Actually after some internal discussion (and since the previous proposal is kind of agent-level anyway in terms of using the agent's TLS config), I'm thinking we will make this an agent-level config instead of per-check. The new config option will be called enable_agent_tls_for_checks and will have the same semantics as suggested previously. If this is true it will apply for all HTTPS health checks running on that agent.

[sonicsandy]

We have a similar situation where we are facing issues when we are trying to do a 2-way mutual auth during the HTTPS health checks. 1-way TLS works though.

Let me try to explain the setup:

Server agent cluster:

• Server Agent 1 (leader) - HTTP: 9500, HTTPS: 9543
• Server Agent 2 - HTTP: 10500, HTTPS: 10543
• Server Agent 3 - HTTP: 11500, HTTPS: 11543

Client agent

• Client Agent 1 - HTTP: 8500, HTTPS: 8543

Spring boot app

• Hello-world-app – HTTPS: 443

Case 1: Spring boot app (Hello-world-app) works in 1-way mode (doesn’t need client’s cert to establish connection)
Spring Configuration:

# HTTP Server (Tomcat) Port
server: 
  port: 443
  ssl:
    key-store: C:\app.p12
    key-store-password: changeit
    key-store-type: JKS

Result: Application health url passes in Consul

Case 2: Spring boot app (Hello-world-app) works in 2-way mode (needs client’s cert to establish connection)
Spring Configuration:

# HTTP Server (Tomcat) Port
server: 
  port: 443
  ssl:
    key-store: C:\app.p12
    key-store-password: changeit
    key-store-type: JKS
    trust-store: C:\Users\z233123\workspace\consulcluster\app-ts.p12
    trust-store-password: cvshealth
    trust-store-type: JKS
    client-auth: need
    enabled: true

Consul Configuration:

{
  "datacenter": "dc1",
  "data_dir": "C:\\consul\\data_dir1",
  "log_level": "INFO",
  "node_name": "server1",
  "encrypt": "mrTAh3MJ9Sma6YB74uH8EQ==",
  "ca_file": "C:\\Users\\z233123\\workspace\\consulcluster\\ca.cer",
  "cert_file": "C:\\Users\\z233123\\workspace\\consulcluster\\consul.cer",
  "key_file": "C:\\Users\\z233123\\workspace\\consulcluster\\consul.key",
  "verify_incoming": true,
  "verify_outgoing": true,
  "server": true,
  "bootstrap" : true,
  "ports" : {
    "dns" : -1,
    "http" : 9500,
    "https" : 9543,
    "serf_lan" : 9301,
    "serf_wan" : 9302,
    "server" : 9300
  }
}

Result: Application health url fails in Consul with the errors as seen below:

Consul client agent logs:

2017/10/16 13:44:31 [WARN] agent: http request failed 'https://IBMT440PC03M1G5.caremarkrx.net:443/health': Get https://IBMT440PC03M1G5.caremarkrx.net:443/health: remote error:
tls: bad certificate
    2017/10/16 13:45:01 [WARN] agent: http request failed 'https://IBMT440PC03M1G5.caremarkrx.net:443/health': Get https://IBMT440PC03M1G5.caremarkrx.net:443/health: remote error:
tls: bad certificate
    2017/10/16 13:45:31 [WARN] agent: http request failed 'https://IBMT440PC03M1G5.caremarkrx.net:443/health': Get https://IBMT440PC03M1G5.caremarkrx.net:443/health: remote error:
tls: bad certificate

Spring boot app logs:

ServerHelloDone
https-jsse-nio-443-exec-3, WRITE: TLSv1.2 Handshake, length = 1887
https-jsse-nio-443-exec-4, READ: TLSv1.2 Handshake, length = 7
*** Certificate chain
<Empty>
***
https-jsse-nio-443-exec-4, fatal error: 42: null cert chain
javax.net.ssl.SSLHandshakeException: null cert chain
%% Invalidated:  [Session-17, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256]
https-jsse-nio-443-exec-4, SEND TLSv1.2 ALERT:  fatal, description = bad_certificate
https-jsse-nio-443-exec-4, WRITE: TLSv1.2 Alert, length = 2
https-jsse-nio-443-exec-4, fatal: engine already closed.  Rethrowing javax.net.ssl.SSLHandshakeException: null cert chain
https-jsse-nio-443-exec-4, called closeOutbound()
https-jsse-nio-443-exec-4, closeOutboundInternal()