Upgrading 1.11 -> 1.12 breaking auto-encrypt

[mr-miles]

When filing a bug, please include the following headings if possible. Any example text in this template can be deleted.

Overview of the Issue

Upgrading consul 1.11.8 to 1.12.4 is causing agents set up with auto-encrypt to be left presenting certificates without the intermediate certificate. This then causes trust issues for anything connecting to the client which is bad.

Reproduction Steps

Happens consistently if I up/down grade the version in our test setup.

Consul info for both Client and Server

Consul servers on ubuntu 20.04 v1.11.8
Agents are 1.12.4
Vault as the CA
All running on AWS
TLS enabled with auto-encrypt

Log Fragments

Enabled trace logging

Client logs seem to show AutoEncrypt.Sign RPC call being made successfully

2022-09-07T23:07:48.630Z [DEBUG] agent.auto_config: making AutoEncrypt.Sign RPC: addr=xx.xx.xx.xx:8300
2022-09-07T23:07:48.658Z [TRACE] agent.tlsutil: OutgoingRPCWrapper: version=1
2022-09-07T23:07:48.658Z [TRACE] agent.tlsutil: OutgoingRPCConfig: version=1
2022-09-07T23:07:48.725Z [TRACE] agent.tlsutil: UpdateAutoTLS: version=2
2022-09-07T23:07:48.725Z [INFO]  agent.auto_config: automatically upgraded to TLS
2022-09-07T23:07:48.725Z [TRACE] agent.tlsutil: Update: version=3
2022-09-07T23:07:48.725Z [TRACE] agent.tlsutil: IncomingGRPConfig: version=3
2022-09-07T23:07:48.727Z [INFO]  agent.client.serf.lan: serf: EventMemberJoin: ip-xx.xx.xx.xx.eu-west-1.compute.internal xx.xx.xx.xx
2022-09-07T23:07:48.727Z [INFO]  agent.router: Initializing LAN area manager
2022-09-07T23:07:48.727Z [INFO]  agent.auto_config: auto-config started

Server logs don't seem to be showing errors

Sep 07 23:07:48 ip-10-142-25-158 consul[1795]: 2022-09-07T23:07:48.559Z [TRACE] agent.server: rpc_server_call: method=Status.Leader errored=false request_type=read rpc_type=net/rpc leader=false allow_stale=fal>
Sep 07 23:07:48 ip-10-142-25-158 consul[1795]: 2022-09-07T23:07:48.662Z [TRACE] agent.tlsutil: IncomingInsecureRPCConfig: version=8
Sep 07 23:07:48 ip-10-142-25-158 consul[1795]: 2022-09-07T23:07:48.662Z [TRACE] agent.tlsutil: IncomingInsecureRPCConfig: version=8
Sep 07 23:07:48 ip-10-142-25-158 consul[1795]: 2022-09-07T23:07:48.668Z [TRACE] agent.tlsutil: OutgoingRPCWrapper: version=8
Sep 07 23:07:48 ip-10-142-25-158 consul[1795]: 2022-09-07T23:07:48.669Z [TRACE] agent.tlsutil: OutgoingRPCConfig: version=8
Sep 07 23:07:48 ip-10-142-25-158 consul[1795]: 2022-09-07T23:07:48.675Z [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=false
Sep 07 23:07:48 ip-10-142-25-158 consul[1795]: 2022-09-07T23:07:48.727Z [TRACE] agent.server: rpc_server_call: method=AutoEncrypt.Sign errored=false request_type=write rpc_type=net/rpc leader=false target_data>
Sep 07 23:07:48 ip-10-142-25-158 consul[1795]: 2022-09-07T23:07:48.796Z [DEBUG] agent.server.memberlist.lan: memberlist: Stream connection from=xx.xx.xx.xx:51960
Sep 07 23:07:49 ip-10-142-25-158 consul[1795]: 2022-09-07T23:07:49.011Z [INFO]  agent.server.serf.lan: serf: EventMemberJoin: ip-xx.xx.xx.xx.eu-west-1.compute.internal xx.xx.xx.xx
Sep 07 23:07:49 ip-10-142-25-158 consul[1795]: 2022-09-07T23:07:49.263Z [DEBUG] agent.server.serf.lan: serf: messageJoinType: ip-xx.xx.xx.xx.eu-west-1.compute.internal
Sep 07 23:07:49 ip-10-142-25-158 consul[1795]: 2022-09-07T23:07:49.296Z [DEBUG] agent.server.serf.lan: serf: messageJoinType: ip-xx.xx.xx.xx.eu-west-1.compute.internal

Certificate presented by the clients after auto-config are issued by the intermediate cert, but appear to omit the intermediate cert in the handshake.

Was suspecting the change to the new-style tls config stanza as that seems to be the biggest change, but the lack of error messages make it puzzling. Also have tried various combinations of new-style parameters but have not got it to run.

I am not sure where to look to diagnose what is going on further: seeking some guidance. Is it possible for consul to have lost the intermediate cert or is this a symptom of something else?

[mr-miles]

If I look at /agent/connect/ca/roots then I can see the intermediate certificate as expected, so agent and server definitely know about it. Totally perplexed and basically out of ideas as to where to look for further investigations, so any help much appreciated

[mr-miles]

Closing - bug and fix captured in #14564