Controller certificate watch fails in secondary datacenter

[nathancoleman]

Overview of the Issue

This issue is a contributing factor to #300.

The controller fails to initialize a watch on root certificates when starting up in a secondary datacenter. This is because the primary datacenter needs to be included when creating the watch but is not today.

consul-api-gateway/internal/consul/certmanager.go

Lines 215 to 217 in c35994b

	rootWatch, err := watch.Parse(map[string]interface{}{
	"type": "connect_roots",
	})

Reproduction Steps

1. Create a federated setup by following the guide Federation Between Kubernetes Clusters. When installing, ensure API Gateway is enabled in both the primary and secondary clusters (apiGateway.enabled: true in values.yaml).
2. View logs of the API gateway controller in the secondary datacenter (kubectl logs ...)

Logs

Logs

2022-09-06T19:58:26.971Z [INFO]  k8s/logger.go:30: consul-api-gateway-server.controller-runtime: Starting workers: controller=gateway controllerGroup=gateway.networking.k8s.io controllerKind=Gateway info="Starting workers" worker count=1
2022-09-06T19:58:26.971Z [INFO]  k8s/logger.go:30: consul-api-gateway-server.controller-runtime: Starting workers: controller=tcproute controllerGroup=gateway.networking.k8s.io controllerKind=TCPRoute info="Starting workers" worker count=1
2022-09-06T20:08:30.888Z [ERROR] watch/plan.go:95: consul-api-gateway-server.cert-manager.watch: Watch errored: type=connect_roots error="Unexpected response code: 500 (rpc error making call: i/o deadline reached)" retry=5s
2022-09-06T20:18:51.442Z [ERROR] watch/plan.go:95: consul-api-gateway-server.cert-manager.watch: Watch errored: type=connect_roots error="Unexpected response code: 500 (rpc error making call: rpc error making call: i/o deadline reached)" retry=5s
2022-09-06T20:39:13.030Z [ERROR] watch/plan.go:95: consul-api-gateway-server.cert-manager.watch: Watch errored: type=connect_roots error="Unexpected response code: 500 (rpc error making call: rpc error making call: i/o deadline reached)" retry=5s
2022-09-06T20:49:28.020Z [ERROR] watch/plan.go:95: consul-api-gateway-server.cert-manager.watch: Watch errored: type=connect_roots error="Unexpected response code: 500 (rpc error making call: i/o deadline reached)" retry=5s
2022-09-06T20:59:41.606Z [ERROR] watch/plan.go:95: consul-api-gateway-server.cert-manager.watch: Watch errored: type=connect_roots error="Unexpected response code: 500 (rpc error making call: rpc error making call: i/o deadline reached)" retry=5s
2022-09-06T21:40:44.984Z [ERROR] watch/plan.go:95: consul-api-gateway-server.cert-manager.watch: Watch errored: type=connect_roots error="Unexpected response code: 500 (rpc error making call: i/o deadline reached)" retry=5s
2022-09-06T21:51:03.780Z [ERROR] watch/plan.go:95: consul-api-gateway-server.cert-manager.watch: Watch errored: type=connect_roots error="Unexpected response code: 500 (rpc error making call: rpc error making call: i/o deadline reached)" retry=5s

Expected behavior

Controller successfully starts up (including cert watch initiated as part of startup)

Environment details

If not already included, please provide the following:

• consul-api-gateway version: v0.4.0
• configuration used to deploy the gateway controller:

apiGateway:
  enabled: true
  image: "hashicorp/consul-api-gateway:0.4.0"

Additionally, please provide details regarding the Kubernetes Infrastructure, as shown below:

• Consul Server version: v1.13.1
• Consul-K8s version: 0.47.1
• Cloud Provider (If self-hosted, the Kubernetes provider utilized): GKE

Additional Context