#10621 #6364 broke the ability to use prepared queries for connect upstreams

[dschaaff]

When filing a bug, please include the following headings if possible. Any example text in this template can be deleted.

Overview of the Issue

Using prepared queries for connect upstream fails on agent version 1.10.1 while it works on 1.10.0. This is presumably due to #10621 .

Reproduction Steps

• create connect service a
• create connect service b
• ensure intentions allow communication between service a and service b
• create a prepared query referencing the upstream service

{
  "name": "service-b-query",
  "Service": {
    "Service": "service-b",
    "Tags": [],
    "Connect": true
  }
}

• set the upstreams annotation service a to use the prepared query

consul.hashicorp.com/connect-service-upstreams: prepared_query:service-b-query:1500

• attempt to connect to service b and notice envoy connection error

 curl -vvv 127.0.0.1:1500
* Expire in 0 ms for 6 (transfer 0x558aab21bfb0)
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x558aab21bfb0)
* Connected to 127.0.0.1 (127.0.0.1) port 1500 (#0)
> GET / HTTP/1.1
> Host: 127.0.0.1:1500
> User-Agent: curl/7.64.0
> Accept: */*
> 
* Recv failure: Connection reset by peer
* Closing connection 0
curl: (56) Recv failure: Connection reset by peer

I believe this is because the upstream's cert does not contain the prepared query name, only the service name spiffe:///ns//dc//svc/.

After downgrading the consul agent to 1.10.0 the request to the prepared_query upstream will succeed.

Consul info for both Client and Server

Client info

agent:
	check_monitors = 0
	check_ttls = 0
	checks = 4
	services = 4
build:
	prerelease =
	revision = db839f18
	version = 1.10.1
consul:
	acl = enabled
	known_servers = 3
	server = false
runtime:
	arch = arm64
	cpu_count = 2
	goroutines = 195
	max_procs = 2
	os = linux
	version = go1.16.6
serf_lan:
	coordinate_resets = 0
	encrypted = true
	event_queue = 0
	event_time = 205547
	failed = 0
	health_score = 0
	intent_queue = 0
	left = 0
	member_time = 647721
	members = 45
	query_queue = 0
	query_time = 1

Server info

agent:
	check_monitors = 0
	check_ttls = 0
	checks = 2
	services = 2
build:
	prerelease =
	revision = db839f18
	version = 1.10.1
consul:
	acl = enabled
	bootstrap = false
	known_datacenters = 1
	leader = false
	leader_addr = 10.20.208.112:8300
	server = true
raft:
	applied_index = 373096103
	commit_index = 373096103
	fsm_pending = 0
	last_contact = 4.978615ms
	last_log_index = 373096103
	last_log_term = 388662
	last_snapshot_index = 373087177
	last_snapshot_term = 388662
	latest_configuration = [{Suffrage:Voter ID:6cd485d0-1ccc-5049-8105-79abb943105e Address:10.20.209.189:8300} {Suffrage:Voter ID:b254668f-b57e-f763-5f3e-3f5ca38ecbc5 Address:10.20.208.112:8300} {Suffrage:Voter ID:e0e2704e-96c9-589d-c4eb-02f0ecf11bd2 Address:10.20.202.159:8300}]
	latest_configuration_index = 0
	num_peers = 2
	protocol_version = 3
	protocol_version_max = 3
	protocol_version_min = 0
	snapshot_version_max = 1
	snapshot_version_min = 0
	state = Follower
	term = 388662
runtime:
	arch = arm64
	cpu_count = 2
	goroutines = 1478
	max_procs = 2
	os = linux
	version = go1.16.6
serf_lan:
	coordinate_resets = 0
	encrypted = true
	event_queue = 0
	event_time = 205547
	failed = 1
	health_score = 0
	intent_queue = 0
	left = 1
	member_time = 647725
	members = 47
	query_queue = 0
	query_time = 1
serf_wan:
	coordinate_resets = 0
	encrypted = true
	event_queue = 0
	event_time = 1
	failed = 0
	health_score = 0
	intent_queue = 0
	left = 0
	member_time = 1610
	members = 3
	query_queue = 0
	query_time = 1

Operating system and Environment details

OS, Architecture, and any other information you can provide about the environment.

Log Fragments

[2021-08-10 23:41:31.819][1][info][upstream] [source/common/upstream/cds_api_impl.cc:79] cds: add/update cluster 'service-b-query.default.stg-datacenter.query.b4963d21-eae8-5de6-3979-74315c41c4ca.consul'

"cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "5c479ab96bca0484194ec875465f4eeaf3269c05",
     "subject_alt_names": [
      {
       "dns": "service-b.svc.default.b4963d21.consul"
      },
      {
       "uri": "spiffe://b4963d21-eae8-5de6-3979-74315c41c4ca.consul/ns/default/dc/stg-datacenter/svc/service-b"
      }
     ],
     "days_until_expiration": "2",
     "valid_from": "2021-08-10T23:40:24Z",
     "expiration_time": "2021-08-13T23:40:54Z"
    }
   ]