Fix consul-telemetry-collector deployment when enableConsulNamespaces #3192

jjti · 2023-11-08T17:28:59Z

Issues

@david-yu reported multiple issues with the Consul Telemetry Collector deployment. A different PR fixed a couple of consul-dataplane args: #3184

This attempts to fix some of the connect-inject issues with namespaces:

Once I create the namespace manually, the connect-init container is not able find the service because there are a couple of things in the deployment template that I think is not right: ... should be the namespace where the OTEL is deployed, that by default is consul. It is taking the namespace used the the SyncCatalog which is wrong because I am using just the connectInject.consulNamespaces.mirroringK8s=true, so the SyncCatalog is false and that value is default then

My guess at what happened when this was written was, we were trying to replicate the namespace selection logic of the Consul Dataplane Sidecar webhook:

consul-k8s/control-plane/connect-inject/webhook/consul_dataplane_sidecar.go

Line 237 in b843390

if w.EnableNamespaces {

We run into issues tho, doing that. Partly because that's using syncCatalog. But also because, if CTC is the first service in a namespace, we fail to inject it because the namespace does not exist.

Changes

This PR is an attempt at fixing some of those namespace/partition issues.

adds a new pod label. If set, we ignore all other namespace logic and just deploy the Consul Telemetry Collector to the default namespace. This is the same as what mesh-gateway does: https://github.com/hashicorp/consul-k8s/blob/e11143561551f86e0e7897a3f2830daa67ef5d74/control-plane/connect-inject/controllers/endpoints/endpoints_controller.go#L692
adds -service-partition arg to the consul-dataplane container

How I've tested this PR:

bats
the scenarios below

Non-default namespaces

I set connectInject.consulNamespaces.mirroringK8s=true and deployed the consul-telemetry-collector to a namespace: test-namespace after creating a deployment from this template change: https://github.com/jjti/consul-experiments/blob/ec80caaeded716a1dfbc34343d0da5db6142ad19/telemetry-collector/resources/consul-telemetry-collector.yaml#L33

$ k get pods --namespace test-namespace
NAME                                           READY   STATUS    RESTARTS   AGE
consul-connect-injector-74c6b4b7dc-5twg9       1/1     Running   0          14m
...
consul-telemetry-collector-68b7b446fc-nzwgb    2/2     Running   0          2m10s

Non-default partitions

I followed this guide shared by @david-yu: https://developer.hashicorp.com/consul/docs/enterprise/admin-partitions#install-on-the-non-default-partition-clusters-running-workloads
I deployed the consul-telemetry-collector into a non-default partition (in addition to the default partition) using a template created with the changes here: https://github.com/jjti/consul-experiments/blob/main/telemetry-collector/resources/consul-telemetry-collector-test.yaml

How I expect reviewers to test this PR:

fact check my assumptions + approaches about consul-k8s-auth-method
fact check whether I missed some combination of logic in connect-inject so this will fail with some combo of namespace/partition settings that I'm ignorant of

Checklist:

[] Tests added
CHANGELOG entry added

jjti · 2023-11-08T17:37:56Z

charts/consul/templates/telemetry-collector-deployment.yaml

          {{- if .Values.global.enableConsulNamespaces }}
-          - name: CONSUL_NAMESPACE
-            value: {{ .Values.syncCatalog.consulNamespaces.consulDestinationNamespace }}


this removes all ref to .Values.syncCatalog.consulNamespaces.consulDestinationNamespace. AFAICT, .Values.syncCatalog.enabled might be false but we'll still insert it here.

My guess is that this is a bug that was from copying this setting from the Sync Catalog Deployment:

consul-k8s/charts/consul/templates/sync-catalog-deployment.yaml

Line 177 in 768e29d

-consul-destination-namespace={{ .Values.syncCatalog.consulNamespaces.consulDestinationNamespace }} \

charts/consul/templates/telemetry-collector-deployment.yaml

Add test for v2 Fix bats tests Fix CONSUL_NAMESPACE to be default Fix consul telemetry collector login namespace Finish namespace/partition logic changes Update v2 too Remove more $roots

jjti · 2023-11-09T17:44:26Z

charts/consul/templates/telemetry-collector-deployment.yaml

-          - -login-namespace={{ .Values.syncCatalog.consulNamespaces.consulDestinationNamespace }}
+          - -login-namespace={{ .Values.connectInject.consulNamespaces.consulDestinationNamespace }}


this is one of the fixes: login-name is from connectInject destination namespace not syncCatalog

david-yu · 2023-11-09T17:45:13Z

@jjti When you say you failed to test with non-default partitions is this because non-default partition support is not available in the telemetry collector? Or is it because you would need to help with partitions on K8s?

jjti · 2023-11-09T17:47:02Z

@jjti When you say you failed to test with non-default partitions is this because non-default partition support is not available in the telemetry collector? Or is it because you would need to help with partitions on K8s?

At this point it's a me problem. If you have a ref values.yaml for that that'd be appreciated. It think it'll work, but the changes here are copied from adjacent deployments like ingress-gateways

jjti · 2023-11-09T17:49:08Z

charts/consul/templates/telemetry-collector-deployment.yaml

          {{- if .Values.global.enableConsulNamespaces }}
-          {{- if .Values.syncCatalog.consulNamespaces.mirroringK8S }}
+          {{- if .Values.connectInject.consulNamespaces.mirroringK8S }}
+          - -service-namespace={{ .Values.connectInject.consulNamespaces.mirroringK8SPrefix }}{{ .Release.Namespace }}


change here to use k8s namespace prefixing like we do in the webhook

david-yu · 2023-11-09T17:49:56Z

@jjti Does this help? https://developer.hashicorp.com/consul/docs/enterprise/admin-partitions#install-on-the-non-default-partition-clusters-running-workloads or do you need a writeup specifically written for HCP?

jjti · 2023-11-09T18:01:52Z

@jjti Does this help? https://developer.hashicorp.com/consul/docs/enterprise/admin-partitions#install-on-the-non-default-partition-clusters-running-workloads or do you need a writeup specifically written for HCP?

It does and thanks. I've been trying to hack helm templates together but lack the background knowledge to be efficient. Thanks! I'll draft this until that's done

Add 1.3.0 notes to changelog

…the latest Consul submodules (#3194) * Add replace directive in Go mod for Control Plane so that we pull in the latest Consul submodules * Add a comment to explain why we need a replace directive

Add MeshGateway to `mesh.consul.hashicorp.com/v2beta1`

add validation to account for type change

* checkpoint * checkpoint, passing test * kitchen sink, NET-4992 * lint issue * clean up unneeded calls --------- Co-authored-by: Sarah Alsmiller <sarah.alsmiller@sarah.alsmiller-RQQ26PQ2L5>

…eway-reso… (#3200) * Adds GatewayClassConfig and MeshGateway resources to the gateway-resources-configmap.yaml in the Helm chart * Updates the configmap to include more fields for the gatewayClassConfig for mesh gateways

* chore: skaffold build experiment * feedback: add experiment comments

…PU request (#3209)

* NET-6401 Stub MeshGateway controller * Add MeshGateways resource to connect-inject-clusterrole * Setup v2controller for MeshGateway * Add bats test assertion for connect-inject-clusterrole * Regenerate control-plane/config/rbac/role.yaml

jjti commented Nov 8, 2023

View reviewed changes

jjti changed the title ~~Jjtimmons/fix ctc namespace default~~ Fix consul-telemetry-collector deployment's setting of CONSUL_LOGIN_NAMESPACE Nov 8, 2023

jjti commented Nov 8, 2023

View reviewed changes

charts/consul/templates/telemetry-collector-deployment.yaml Show resolved Hide resolved

Base automatically changed from jjtimmons/fix-ctc-with-acls to main November 8, 2023 21:35

jjti force-pushed the jjtimmons/fix-ctc-namespace-default branch from 26240ad to e2fc3e9 Compare November 8, 2023 23:30

Add tests

b2f7f00

Add test for v2 Fix bats tests Fix CONSUL_NAMESPACE to be default Fix consul telemetry collector login namespace Finish namespace/partition logic changes Update v2 too Remove more $roots

jjti force-pushed the jjtimmons/fix-ctc-namespace-default branch from e2fc3e9 to b2f7f00 Compare November 8, 2023 23:31

Add changelog

29430f4

jjti changed the title ~~Fix consul-telemetry-collector deployment's setting of CONSUL_LOGIN_NAMESPACE~~ Fix consul-telemetry-collector deployment namespace/partition when non-default Nov 9, 2023

jjti added 3 commits November 8, 2023 21:15

Fix component name

f7d62aa

Partition is set for us already

a20bcf0

Fix auth method

7b71216

jjti added backport/1.1.x Backport to release/1.1.x branch backport/1.2.x This release branch is no longer active. backport/1.3.x labels Nov 9, 2023

Fix tests

85b3f45

jjti requested review from DanStough and david-yu November 9, 2023 17:41

jjti marked this pull request as ready for review November 9, 2023 17:41

jjti commented Nov 9, 2023

View reviewed changes

jjti marked this pull request as draft November 9, 2023 18:01

jjti changed the title ~~Fix consul-telemetry-collector deployment namespace/partition when non-default~~ Fix consul-telemetry-collector deployment in non-default partitions/namespaces Nov 14, 2023

jjti and others added 3 commits November 14, 2023 12:59

Set namespace to default for consul telemetry collector

12ba216

NET-6303 Add 1.3.0 notes to changelog (#3193)

e73ef2b

Add 1.3.0 notes to changelog

Add replace directive in Go mod for Control Plane so that we pull in …

cb9db90

…the latest Consul submodules (#3194) * Add replace directive in Go mod for Control Plane so that we pull in the latest Consul submodules * Add a comment to explain why we need a replace directive

Thomas Eckert and others added 8 commits November 14, 2023 12:59

NET-6331 Create MeshGateway CRD (#3195)

81ed4fe

Add MeshGateway to `mesh.consul.hashicorp.com/v2beta1`

Add validation to account for type change (#3202)

1bf8378

add validation to account for type change

NET-4992- Kitchen sink gateway test (#3196)

4218a8d

* checkpoint * checkpoint, passing test * kitchen sink, NET-4992 * lint issue * clean up unneeded calls --------- Co-authored-by: Sarah Alsmiller <sarah.alsmiller@sarah.alsmiller-RQQ26PQ2L5>

chore: skaffold build experiment (#3179)

499072a

* chore: skaffold build experiment * feedback: add experiment comments

fix(control-plane): sidecar CPU limit incorrectly validated against C…

273785b

…PU request (#3209)

NET-6401 Stub MeshGateway controller (#3204)

a4104b8

* NET-6401 Stub MeshGateway controller * Add MeshGateways resource to connect-inject-clusterrole * Setup v2controller for MeshGateway * Add bats test assertion for connect-inject-clusterrole * Regenerate control-plane/config/rbac/role.yaml

Fix deployment label comma

83ca25c

jjti force-pushed the jjtimmons/fix-ctc-namespace-default branch from 42e19e4 to 83ca25c Compare November 14, 2023 17:59

jjti added 3 commits November 14, 2023 12:59

Merge branch 'main' into jjtimmons/fix-ctc-namespace-default

40cf350

Fix a comment

0ba55ce

Switch to hardcoding further into endpoint controller

990cd7a

jjti changed the title ~~Fix consul-telemetry-collector deployment in non-default partitions/namespaces~~ Fix consul-telemetry-collector deployment when enableConsulNamespaces Nov 14, 2023

jjti added 3 commits November 14, 2023 15:28

Add telemetry-collector flag

4d0d40a

Fix flag

29f57ea

Use default explicitly

0c566cc

jjti closed this Nov 15, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix consul-telemetry-collector deployment when enableConsulNamespaces #3192

Fix consul-telemetry-collector deployment when enableConsulNamespaces #3192

jjti commented Nov 8, 2023 •

edited

Loading

jjti Nov 8, 2023 •

edited

Loading

jjti Nov 9, 2023

david-yu commented Nov 9, 2023

jjti commented Nov 9, 2023

jjti Nov 9, 2023

david-yu commented Nov 9, 2023

jjti commented Nov 9, 2023

		- -login-namespace={{ .Values.syncCatalog.consulNamespaces.consulDestinationNamespace }}
		- -login-namespace={{ .Values.connectInject.consulNamespaces.consulDestinationNamespace }}

Fix consul-telemetry-collector deployment when enableConsulNamespaces #3192

Fix consul-telemetry-collector deployment when enableConsulNamespaces #3192

Conversation

jjti commented Nov 8, 2023 • edited Loading

Issues

Changes

How I've tested this PR:

Non-default namespaces

Non-default partitions

How I expect reviewers to test this PR:

jjti Nov 8, 2023 • edited Loading

Choose a reason for hiding this comment

jjti Nov 9, 2023

Choose a reason for hiding this comment

david-yu commented Nov 9, 2023

jjti commented Nov 9, 2023

jjti Nov 9, 2023

Choose a reason for hiding this comment

david-yu commented Nov 9, 2023

jjti commented Nov 9, 2023

jjti commented Nov 8, 2023 •

edited

Loading

jjti Nov 8, 2023 •

edited

Loading