NET-581 - Added vault namespace in helm

[absolutelightning]

Changes proposed in this PR:

• Adds vaultNamespace in secretsBackend.vault in values.yaml
• This namespace is used for Vault namespace, this introduces one more way to specify namespace other than
  "{"connect": [{ "ca_config": [{ "namespace": "value"}]}]}" in connectCA.additionalConfig
• If vaultNamespace is present, it automatically adds annotation below to the templates.

    vault:
      agentAnnotations: |
        vault.hashicorp.com/namespace: vaultNamespace

How I've tested this PR:
a. CI
b. Updated test TestVault_VaultNamespace
Test Steps -

1. kind create cluster --name=dc1
2. kind create cluster --name=dc2
3. cd acceptance/test/vault
4. go test ./... -p 1 -timeout 2h -failfast -no-cleanup-on-failure -debug-directory=/tmp/debug -use-kind -enable-multi-cluster -kube-contexts=kind-dc1,kind-dc2  -run ^TestVault

Output -

asheshvidyut@absolutelightning-H2GX766V9T acceptance/tests (NET-581-Configure-Vault-namespaces-for-Connect-CA-via-Helm-Stanza) » go test ./... -p 1 -timeout 2h -failfast -no-cleanup-on-failure -debug-directory=/tmp/debug -use-kind -enable-multi-cluster -kube-contexts=kind-dc1,kind-dc2  -run ^TestVault
ok  	github.com/hashicorp/consul-k8s/acceptance/tests/api-gateway	0.492s [no tests to run]
ok  	github.com/hashicorp/consul-k8s/acceptance/tests/basic	0.547s [no tests to run]
ok  	github.com/hashicorp/consul-k8s/acceptance/tests/cli	0.529s [no tests to run]
ok  	github.com/hashicorp/consul-k8s/acceptance/tests/cloud	0.565s [no tests to run]
ok  	github.com/hashicorp/consul-k8s/acceptance/tests/config-entries	0.524s [no tests to run]
ok  	github.com/hashicorp/consul-k8s/acceptance/tests/connect	0.556s [no tests to run]
ok  	github.com/hashicorp/consul-k8s/acceptance/tests/consul-dns	0.543s [no tests to run]
ok  	github.com/hashicorp/consul-k8s/acceptance/tests/example	0.540s
ok  	github.com/hashicorp/consul-k8s/acceptance/tests/ingress-gateway	0.553s [no tests to run]
ok  	github.com/hashicorp/consul-k8s/acceptance/tests/metrics	0.555s [no tests to run]
ok  	github.com/hashicorp/consul-k8s/acceptance/tests/partitions	0.561s [no tests to run]
ok  	github.com/hashicorp/consul-k8s/acceptance/tests/peering	0.568s [no tests to run]
ok  	github.com/hashicorp/consul-k8s/acceptance/tests/sameness	0.552s
ok  	github.com/hashicorp/consul-k8s/acceptance/tests/snapshot-agent	0.572s [no tests to run]
ok  	github.com/hashicorp/consul-k8s/acceptance/tests/sync	0.554s [no tests to run]
ok  	github.com/hashicorp/consul-k8s/acceptance/tests/terminating-gateway	0.553s [no tests to run]
ok  	github.com/hashicorp/consul-k8s/acceptance/tests/vault	916.339s
ok  	github.com/hashicorp/consul-k8s/acceptance/tests/wan-federation	0.587s [no tests to run]

How I expect reviewers to test this PR:

Checklist:

• Tests added
• CHANGELOG entry added

[zalimeni]

Looks good to me overall! Added some minor feedback inline.

One clarifying question about requirements: I think @david-yu 's example in the ticket had namespace under the higher-level secretsBackend.vault key. I'm not opposed to keeping it under connectCA because that aligns w/ the existing agent config, but I could see a higher-level key being used for other namespaced Vault config, if we expect that one Vault namespace would always be used across Vault->Consul integrations (I'm not 100% certain whether that's the case).

1. Do we think it's best to move the namespace KV up, or keep it under connectCA?
2. If we move it, a suggestion in one of the linked Slack 🧵s was to call it vaultNamespace to disambiguate from Consul. That would align w/ other names such as consulServerRole.

[david-yu]

@curtbushko Is there a reason why you only added backport 1.2.x instead of 1.0.x, 1.1.x and 1.2.x?

[curtbushko]

@david-yu No reason. I was just doing a passby label adding. I didn't look too deep to see where this should go.

[david-yu]

-yu 's example in the ticket had namespace under the higher-level secretsBackend.vault key. I'm not opposed to keeping it under connectCA because that aligns w/ the existing agent config, but I could see a higher-level key being used for other namespaced Vault config, if we expect that one Vault namespace would always be used across Vault->Consul integrations (I'm not 100% certain whether that's the case).

1. Do we think it's best to move the namespace KV up, or keep it under connectCA?
2. If we move it, a suggestion in one of the linked Slack 🧵s was to call it vaultNamespace to disambiguate from Consul. That would align w/ other names such as consulServerRole.

I would prefer it be under the secretsBackend.vault stanza. If we do that we should automatically set agent annotations to the associated vault namespace as well as that is also required to get the vault namespace working. Below is an example of full desired secrets backend config to set the vault namespace to admin:

vault:
      enabled: true
      vaultNamespace: admin
      consulServerRole: consul-server
      consulClientRole: consul-client
      consulCARole: consul-ca
      manageSystemACLsRole: consul-server-acl-init
      controllerRole: consul-controller
      connectInjectRole: consul-connect-inject
      controller:
        caCert:
          secretName: "consul_controller_int/cert/ca"
        tlsCert:
          secretName: "consul_controller_int/issue/consul-controller"
      connectInject:
        caCert:
          secretName: "consul_connect_inject_int/cert/ca"
        tlsCert:
          secretName: "consul_connect_inject_int/issue/consul-connect-inject"
      connectCA:
        address: https://hcp-vault-cluster:8200/
        rootPKIPath: consul_pki/
        intermediatePKIPath: consul_connect_int/

Behind the scenes it should also set the following annotation to admin:

    vault:
      agentAnnotations: |
        vault.hashicorp.com/namespace: admin

It would be best to call it vaultNamespace as well since it makes it clear this a vault namespace versus Consul.

[absolutelightning]

@david-yu I am not super familiar with vault namespace. but can this situation arise where user would want different vault namespace for agentAnnotation and different for connectCA?

[david-yu]

I don't think it's very likely that the Connect CA will live in a different place than the Vault KV store but you can have it separate if they want to do that. I would think we set the namespace to the value provided by vaultNamespace by default and have it overridden by the user if need be.

[absolutelightning]

Thanks for reply. Also can user give some other annotations as well for vault? we are going to remove agentAnnotations field right.
Now it can only have one value. @david-yu

[absolutelightning]

@david-yu Removing agent annotations is probably incorrect as I see there are other annotations as well which users can add - https://developer.hashicorp.com/vault/docs/platform/k8s/injector/annotations. Let me know you thoughts.

[david-yu]

Added consul-docs for review of values.yaml

[zalimeni]

@absolutelightning good catch, my example was off - but I think this is still possible, see this updated version.

From the link above, if you use:

{{- if (and (.vaultNamespace) (not (hasKey (default "" .connectCA.additionalConfig | fromJson).connect.ca_config "namespace"))) }}
  "namespace": "{{ .vaultNamespace }}",
{{- end }}

does that work?

[zalimeni]

LGTM pending any input from docs team or @david-yu. Thank you again for working through my feedback and questions, @absolutelightning !

[absolutelightning]

LGTM pending any input from docs team or @david-yu. Thank you again for working through my feedback and questions, @absolutelightning !

Thanks @zalimeni for review.