Add support to external Vault running with tls

[juananinca]

Is your feature request related to a problem? Please describe.
I have an external Vault running outside the kubernetes cluster.
The vault service is running with the following tls settings:

listener "tcp" {
  address = "0.0.0.0:8200"
  cluster_address  = "X.X.X.X:8201"
  tls_cert_file = "/opt/vault/ssl/server-aeavaultdes01.pem"
  tls_key_file = "/opt/vault/ssl/server-aeavaultdes01-key.pem"
  tls_require_and_verify_client_cert = "true"
  tls_client_ca_file ="/opt/vault/ssl/vault-ca.pem"
}

But can't find any cert settings in the values.yaml file regarding the external vault service. I have just set the global.externalVaultAddr in the values.yaml and here is the logs of the init container vault-agent-init injected into an pod:

==> Vault Agent started! Log data will stream in below:

==> Vault Agent configuration:

           Api Address 1: http://bufconn
                     Cgo: disabled
               Log Level: info
                 Version: Vault v1.16.1, built 2024-04-03T12:35:53Z
             Version Sha: 6b5986790d7748100de77f7f127119c4a0f78946

2024-04-15T10:02:45.083Z [INFO]  agent.sink.file: creating file sink
2024-04-15T10:02:45.083Z [INFO]  agent.sink.file: file sink configured: path=/home/vault/.vault-token mode=-rw-r-----
2024-04-15T10:02:45.084Z [INFO]  agent.exec.server: starting exec server
2024-04-15T10:02:45.084Z [INFO]  agent.exec.server: no env templates or exec config, exiting
2024-04-15T10:02:45.084Z [INFO]  agent.auth.handler: starting auth handler
2024-04-15T10:02:45.084Z [INFO]  agent.sink.server: starting sink server
2024-04-15T10:02:45.084Z [INFO]  agent.auth.handler: authenticating
2024-04-15T10:02:45.084Z [INFO]  agent.template.server: starting template server
2024-04-15T10:02:45.084Z [INFO]  agent: (runner) creating new runner (dry: false, once: false)
2024-04-15T10:02:45.085Z [INFO]  agent: (runner) creating watcher
2024-04-15T10:02:45.094Z [ERROR] agent.auth.handler: error authenticating: error="Put \"https://myexternalvault:8200/v1/auth/kubernetes/login\": tls: failed to verify certificate: x509: certificate signed by unknown authority" backoff=920ms
2024-04-15T10:02:46.015Z [INFO]  agent.auth.handler: authenticating
2024-04-15T10:02:46.025Z [ERROR] agent.auth.handler: error authenticating: error="Put \"https://myexternalvault:8200/v1/auth/kubernetes/login\": tls: failed to verify certificate: x509: certificate signed by unknown authority" backoff=920ms
2024-04-15T10:02:47.476Z [INFO]  agent.auth.handler: authenticating
2024-04-15T10:02:47.486Z [ERROR] agent.auth.handler: error authenticating: error="Put \"https://myexternalvault:8200/v1/auth/kubernetes/login\": tls: failed to verify certificate: x509: certificate signed by unknown authority" backoff=1.45s
2024-04-15T10:02:49.692Z [INFO]  agent.auth.handler: authenticating
2024-04-15T10:02:49.702Z [ERROR] agent.auth.handler: error authenticating: error="Put \"https://myexternalvault:8200/v1/auth/kubernetes/login\": tls: failed to verify certificate: x509: certificate signed by unknown authority" backoff=2.2s
2024-04-15T10:02:53.390Z [INFO]  agent.auth.handler: authenticating
2024-04-15T10:02:53.401Z [ERROR] agent.auth.handler: error authenticating: error="Put \"https://myexternalvault:8200/v1/auth/kubernetes/login\": tls: failed to verify certificate: x509: certificate signed by unknown authority" backoff=3.68s
2024-04-15T10:02:59.834Z [INFO]  agent.auth.handler: authenticating
2024-04-15T10:02:59.845Z [ERROR] agent.auth.handler: error authenticating: error="Put \"https://myexternalvault:8200/v1/auth/kubernetes/login\": tls: failed to verify certificate: x509: certificate signed by unknown authority" backoff=6.43s
2024-04-15T10:03:10.306Z [INFO]  agent.auth.handler: authenticating
2024-04-15T10:03:10.317Z [ERROR] agent.auth.handler: error authenticating: error="Put \"https://myexternalvault:8200/v1/auth/kubernetes/login\": tls: failed to verify certificate: x509: certificate signed by unknown authority" backoff=10.46s
2024-04-15T10:03:26.719Z [INFO]  agent.auth.handler: authenticating
2024-04-15T10:03:26.732Z [ERROR] agent.auth.handler: error authenticating: error="Put \"https://myexternalvault:8200/v1/auth/kubernetes/login\": tls: failed to verify certificate: x509: certificate signed by unknown authority" backoff=16.4s
2024-04-15T10:03:54.753Z [INFO]  agent.auth.handler: authenticating
2024-04-15T10:03:54.763Z [ERROR] agent.auth.handler: error authenticating: error="Put \"https://myexternalvault:8200/v1/auth/kubernetes/login\": tls: failed to verify certificate: x509: certificate signed by unknown authority" backoff=28.02s
2024-04-15T10:04:40.098Z [INFO]  agent.auth.handler: authenticating
2024-04-15T10:04:40.110Z [ERROR] agent.auth.handler: error authenticating: error="Put \"https://myexternalvault:8200/v1/auth/kubernetes/login\": tls: failed to verify certificate: x509: certificate signed by unknown authority" backoff=45.33s
2024-04-15T10:05:58.840Z [INFO]  agent.auth.handler: authenticating
2024-04-15T10:05:58.855Z [ERROR] agent.auth.handler: error authenticating: error="Put \"https://myexternalvault:8200/v1/auth/kubernetes/login\": tls: failed to verify certificate: x509: certificate signed by unknown authority" backoff=1m18.72s
2024-04-15T10:08:32.377Z [INFO]  agent.auth.handler: authenticating
2024-04-15T10:08:32.390Z [ERROR] agent.auth.handler: error authenticating: error="Put \"https://myexternalvault:8200/v1/auth/kubernetes/login\": tls: failed to verify certificate: x509: certificate signed by unknown authority" backoff=2m33.52s
2024-04-15T10:12:37.113Z [INFO]  agent.auth.handler: authenticating
2024-04-15T10:12:37.125Z [ERROR] agent.auth.handler: error authenticating: error="Put \"https://myexternalvault:8200/v1/auth/kubernetes/login\": tls: failed to verify certificate: x509: certificate signed by unknown authority" backoff=4m4.72s
2024-04-15T10:17:31.374Z [INFO]  agent.auth.handler: authenticating
2024-04-15T10:17:31.387Z [ERROR] agent.auth.handler: error authenticating: error="Put \"https://myexternalvault:8200/v1/auth/kubernetes/login\": tls: failed to verify certificate: x509: certificate signed by unknown authority" backoff=4m54.24s
2024-04-15T10:21:39.304Z [INFO]  agent.auth.handler: authenticating
2024-04-15T10:21:39.317Z [ERROR] agent.auth.handler: error authenticating: error="Put \"https://myexternalvault:8200/v1/auth/kubernetes/login\": tls: failed to verify certificate: x509: certificate signed by unknown authority" backoff=4m7.91s
2024-04-15T10:26:10.346Z [INFO]  agent.auth.handler: authenticating
2024-04-15T10:26:10.372Z [ERROR] agent.auth.handler: error authenticating: error="Put \"https://myexternalvault:8200/v1/auth/kubernetes/login\": tls: failed to verify certificate: x509: certificate signed by unknown authority" backoff=4m31.02s
2024-04-15T10:30:36.484Z [INFO]  agent.auth.handler: authenticating
2024-04-15T10:30:38.498Z [ERROR] agent.auth.handler: error authenticating: error="Put \"https://myexternalvault:8200/v1/auth/kubernetes/login\": tls: failed to verify certificate: x509: certificate signed by unknown authority" backoff=4m26.11s
2024-04-15T10:35:08.123Z [INFO]  agent.auth.handler: authenticating
2024-04-15T10:35:08.135Z [ERROR] agent.auth.handler: error authenticating: error="Put \"https://myexternalvault:8200/v1/auth/kubernetes/login\": tls: failed to verify certificate: x509: certificate signed by unknown authority" backoff=4m29.62s
2024-04-15T10:39:12.290Z [INFO]  agent.auth.handler: authenticating
2024-04-15T10:39:12.302Z [ERROR] agent.auth.handler: error authenticating: error="Put \"https://myexternalvault:8200/v1/auth/kubernetes/login\": tls: failed to verify certificate: x509: certificate signed by unknown authority" backoff=4m4.15s

Sorry if I missed the specific settings, but wasn't able to find it.

Describe the solution you'd like

A tls settings for the external vault service would solve my problem.

Thank you!!

[tvoran]

Hi @juananinca, I think the setting you're looking for is the ca-cert annotation, which can also be set for all injected agents by setting AGENT_INJECT_VAULT_CACERT_BYTES on the injector deployment in the chart values:

injector:
  extraEnvironmentVars:
    AGENT_INJECT_VAULT_CACERT_BYTES: <PEM-encoded certificate or bundle contents>

That can also be base64 encoded IIRC.

[juananinca]

I set the AGENT_INJECT_VAULT_CACERT_BYTES and it looks that error log message went from certificate signed by unknown authority to bad certificate, so it seems settings the var took effect but the error makes sense to me since I am not settings the cert and key mentioned in the vault config /opt/vault/ssl/server-aeavaultdes01.pem and /opt/vault/ssl/server-aeavaultdes01-key.pem.

==> Vault Agent started! Log data will stream in below:

==> Vault Agent configuration:

           Api Address 1: http://bufconn
                     Cgo: disabled
2024-04-16T10:58:27.268Z [INFO]  agent.sink.file: creating file sink
2024-04-16T10:58:27.268Z [INFO]  agent.sink.file: file sink configured: path=/home/vault/.vault-token mode=-rw-r-----
               Log Level: info
                 Version: Vault v1.16.1, built 2024-04-03T12:35:53Z
             Version Sha: 6b5986790d7748100de77f7f127119c4a0f78946

2024-04-16T10:58:27.269Z [INFO]  agent.auth.handler: starting auth handler
2024-04-16T10:58:27.269Z [INFO]  agent.exec.server: starting exec server
2024-04-16T10:58:27.269Z [INFO]  agent.exec.server: no env templates or exec config, exiting
2024-04-16T10:58:27.269Z [INFO]  agent.template.server: starting template server
2024-04-16T10:58:27.269Z [INFO]  agent: (runner) creating new runner (dry: false, once: false)
2024-04-16T10:58:27.269Z [INFO]  agent.auth.handler: authenticating
2024-04-16T10:58:27.269Z [INFO]  agent.sink.server: starting sink server
2024-04-16T10:58:27.270Z [INFO]  agent: (runner) creating watcher
2024-04-16T10:58:27.280Z [ERROR] agent.auth.handler: error authenticating: error="Put \"https://myexternalvault:8200/v1/auth/kubernetes/login\": remote error: tls: bad certificate" backoff=840ms
2024-04-16T10:58:28.123Z [INFO]  agent.auth.handler: authenticating
2024-04-16T10:58:28.134Z [ERROR] agent.auth.handler: error authenticating: error="Put \"https://myexternalvault:8200/v1/auth/kubernetes/login\": remote error: tls: bad certificate" backoff=840ms
2024-04-16T10:58:29.607Z [INFO]  agent.auth.handler: authenticating
2024-04-16T10:58:29.619Z [ERROR] agent.auth.handler: error authenticating: error="Put \"https://myexternalvault:8200/v1/auth/kubernetes/login\": remote error: tls: bad certificate" backoff=1.47s
2024-04-16T10:58:31.861Z [INFO]  agent.auth.handler: authenticating
2024-04-16T10:58:31.874Z [ERROR] agent.auth.handler: error authenticating: error="Put \"https://myexternalvault:8200/v1/auth/kubernetes/login\": remote error: tls: bad certificate" backoff=2.24s
2024-04-16T10:58:35.510Z [INFO]  agent.auth.handler: authenticating
2024-04-16T10:58:35.522Z [ERROR] agent.auth.handler: error authenticating: error="Put \"https://myexternalvault:8200/v1/auth/kubernetes/login\": remote error: tls: bad certificate" backoff=3.63s
2024-04-16T10:58:41.308Z [INFO]  agent.auth.handler: authenticating
2024-04-16T10:58:41.324Z [ERROR] agent.auth.handler: error authenticating: error="Put \"https://myexternalvault:8200/v1/auth/kubernetes/login\": remote error: tls: bad certificate" backoff=5.78s

I took a look to the injector commad https://pkg.go.dev/github.com/hashicorp/vault-k8s/subcommand/injector, but I didn't find something like AGENT_INJECT_VAULT_CERT_BYTES or AGENT_INJECT_VAULT_KEY_BYTES which would suit perfectly to my case.