make kubernetes_ca_cert optional #238

thyton · 2024-03-13T06:05:30Z

Overview

Users want to make kubernetes_ca_cert optional. Since the CA cert is used only for establishing a TLS connection Kubernetes api, we can default to use the system's trust store with no harm as mentioned in #62.

Design of Change

The config write handler removes non-nil kubernetes_ca_cert enforcement.

When kubernetes_ca_cert is not given:

If disable_local_default_ca_jwt is false, set the local CA to caCertBytes.
If caCertBytes is empty, assign the default TLS config to transport.TLSClientConfig and return early.

Related Issues/Pull Requests

Issue Do not make ca cert or pem keys required #62
Closes PR Remove cert and pem enforcement #150

Contributor Checklist

vault docs PR #25963
manual tests

The self-signed kubernetes host CA is in the host’s trust store

2024-03-19T15:35:11.286Z [TRACE] auth.vault-plugin-auth-kubernetes.auth_vault-plugin-auth-kubernetes_2996fdb6.vault-plugin-auth-kubernetes: handle request: transport=gRPC path=login status=started
2024-03-19T15:35:11.311Z [TRACE] auth.vault-plugin-auth-kubernetes.auth_vault-plugin-auth-kubernetes_2996fdb6.vault-plugin-auth-kubernetes: handle request: transport=gRPC path=login status=finished err=<nil> took=24.570041ms
2024-03-19T15:35:11.311Z [DEBUG] identity: creating a new entity: alias="id:\"153b12e0-7d31-9c7d-a022-c2ad17a0d5b3\"  canonical_id:\"de0043a8-5f9b-9d19-ee52-0183d2110157\"  mount_type:\"vault-plugin-auth-kubernetes\"  mount_accessor:\"auth_vault-plugin-auth-kubernetes_2996fdb6\"  mount_path:\"auth/kubernetes/\"  metadata:{key:\"service_account_name\"  value:\"default\"}  metadata:{key:\"service_account_namespace\"  value:\"default\"}  metadata:{key:\"service_account_secret_name\"  value:\"\"}  metadata:{key:\"service_account_uid\"  value:\"6ac6361a-722a-4b11-8c4f-d1e4c5fdfaae\"}  name:\"6ac6361a-722a-4b11-8c4f-d1e4c5fdfaae\"  creation_time:{seconds:1710862511  nanos:311625256}  last_update_time:{seconds:1710862511  nanos:311625256}  namespace_id:\"root\"  local_bucket_key:\"packer/local-aliases/buckets/248\""
2024-03-19T15:35:11.313Z [TRACE] auth.vault-plugin-auth-kubernetes.auth_vault-plugin-auth-kubernetes_2996fdb6.vault-plugin-auth-kubernetes: type: transport=gRPC status=started
2024-03-19T15:35:11.314Z [TRACE] auth.vault-plugin-auth-kubernetes.auth_vault-plugin-auth-kubernetes_2996fdb6.vault-plugin-auth-kubernetes: type: transport=gRPC status=finished took=1.367708ms

The self-signed kubernetes host CA is not in the host’s trust store

2024-03-19T15:47:33.196Z [TRACE] auth.vault-plugin-auth-kubernetes.auth_vault-plugin-auth-kubernetes_2e239732.vault-plugin-auth-kubernetes: handle request: transport=gRPC path=login status=started
2024-03-19T15:47:33.211Z [DEBUG] auth.vault-plugin-auth-kubernetes.auth_vault-plugin-auth-kubernetes_2e239732.vault-plugin-auth-kubernetes.vault-plugin-auth-kubernetes: login unauthorized: err="Post \"https://192.168.1.95:60815/apis/authentication.k8s.io/v1/tokenreviews\": tls: failed to verify certificate: x509: certificate signed by unknown authority" timestamp=2024-03-19T15:47:33.211Z
2024-03-19T15:47:33.212Z [TRACE] auth.vault-plugin-auth-kubernetes.auth_vault-plugin-auth-kubernetes_2e239732.vault-plugin-auth-kubernetes: handle request: transport=gRPC path=login status=finished err="permission denied" took=15.917208ms

Invalid kubernetes_ca_cert

$ cat scratch/invalid-ca-bundble.pem 
bad
$ scr
atch/test-ca-bundle.sh scratch/invalid-ca-bundble.pem
Error writing data to auth/kubernetes/config: Error making API request.

URL: PUT http://0.0.0.0:8200/v1/auth/kubernetes/config
Code: 400. Errors:

* Configured CA PEM data contains no valid certificates, TLS verification will fail

Correctly formatted kubernetes_ca_cert but not the kubernetes host's CA

$ scratch/test-ca-bundle.sh scratch/non-k8s-valid-ca-bundle.pem
Success! Data written to: auth/kubernetes/config
Success! Data written to: auth/kubernetes/role/demo
Error writing data to auth/kubernetes/login: Error making API request.

URL: PUT http://0.0.0.0:8200/v1/auth/kubernetes/login
Code: 403. Errors:

* permission denied

2024-03-20T16:17:15.288Z [DEBUG] auth.vault-plugin-auth-kubernetes.auth_vault-plugin-auth-kubernetes_a119c08e.vault-plugin-auth-kubernetes.vault-plugin-auth-kubernetes: login unauthorized: err="Post \"https://192.168.1.95:60815/apis/authentication.k8s.io/v1/tokenreviews\": tls: failed to verify certificate: x509: certificate signed by unknown authority" timestamp=2024-03-20T16:17:15.288Z
2024-03-20T16:17:15.288Z [TRACE] auth.vault-plugin-auth-kubernetes.auth_vault-plugin-auth-kubernetes_a119c08e.vault-plugin-auth-kubernetes: handle request: transport=gRPC path=login status=finished err="permission denied" took=14.4695ms

Valid kubernetes_ca_cert

Success! Data written to: auth/kubernetes/config
Success! Data written to: auth/kubernetes/role/demo
Key                                       Value
---                                       -----
token                                     hvs.CAESIIFdnVdpvlFfXeR2uqyDgVCJaUInEcw_BC2ufWzOdnmaGh4KHGh2cy5CMGF2ZVp4RVJ1dWdlcDhDOFZpd1AzcXk
.....

Switching from system cert pool to kubernetes_ca_cert

[24/03/20 09:35:04] ~/go/src/github.com/hashicorp/vault-plugin-auth-kubernetes (VAULT-1729-make-kubernetes-ca-cert-optional) $ scratch/test-system.sh
Success! Data written to: auth/kubernetes/config
Success! Data written to: auth/kubernetes/role/demo
Key                                       Value
---                                       -----
token                                     hvs.CAESILLOEjFnCBbx6ZJ-Q-BLwWLQLXou9LwWp2n5KoT-7NkNGh4KHGh2cy5mMHpsVWVTQU91d1NsZUttZURKbW1BbHE
token_accessor                            NIELmHYd6Hv9vDSQay0sRllP
...
[24/03/20 09:35:23] ~/go/src/github.com/hashicorp/vault-plugin-auth-kubernetes (VAULT-1729-make-kubernetes-ca-cert-optional) $ scratch/test-ca-bundle.sh scratch/valid-ca-cert.pem                                                                
Success! Data written to: auth/kubernetes/config
Success! Data written to: auth/kubernetes/role/demo
Key                                       Value
---                                       -----
token                                     hvs.CAESINIGC0y-q0iJ_iBFcnaPJyQw0E-C7tijilW_K2TEO_30Gh4KHGh2cy4xV2hTbzU1bVlhamNFVGQ3T3VFN3FNelE
token_accessor                            GPE0XTcNZA9ZQnhn2THFs0P6
token_duration                            1h

2024-03-20T16:35:31.307Z [TRACE] auth.vault-plugin-auth-kubernetes.auth_vault-plugin-auth-kubernetes_f6d85e17.vault-plugin-auth-kubernetes: handle request: transport=gRPC path=config status=started
2024-03-20T16:35:31.310Z [TRACE] auth.vault-plugin-auth-kubernetes.auth_vault-plugin-auth-kubernetes_f6d85e17.vault-plugin-auth-kubernetes.vault-plugin-auth-kubernetes: Root CA certificate pool has changed, updating the client's transport: timestamp=2024-03-20T16:35:31.310Z
2024-03-20T16:35:31.312Z [TRACE] auth.vault-plugin-auth-kubernetes.auth_vault-plugin-auth-kubernetes_f6d85e17.vault-plugin-auth-kubernetes: handle request: transport=gRPC path=config status=finished err=<nil> took=5.18775ms

Switching back to system cert pool from kubernetes_ca_cert

[24/03/20 09:36:18] ~/go/src/github.com/hashicorp/vault-plugin-auth-kubernetes (VAULT-1729-make-kubernetes-ca-cert-optional) $ scratch/test-system.sh
Success! Data written to: auth/kubernetes/config
Success! Data written to: auth/kubernetes/role/demo
Key                                       Value
---                                       -----
token                                     hvs.CAESICgVKj0Kk_vppYBKtoJpyGVoOax90DJO4GJ-GckInnwbGh4KHGh2cy5tZUplNDVRRlF0TTJqMkNHTWVLMGszUlg
...

2024-03-20T16:39:07.523Z [TRACE] auth.vault-plugin-auth-kubernetes.auth_vault-plugin-auth-kubernetes_f6d85e17.vault-plugin-auth-kubernetes: handle request: transport=gRPC path=config status=started
2024-03-20T16:39:07.525Z [TRACE] auth.vault-plugin-auth-kubernetes.auth_vault-plugin-auth-kubernetes_f6d85e17.vault-plugin-auth-kubernetes.vault-plugin-auth-kubernetes: Root CA certificate pool has changed, updating the client's transport: timestamp=2024-03-20T16:39:07.525Z
2024-03-20T16:39:07.526Z [TRACE] auth.vault-plugin-auth-kubernetes.auth_vault-plugin-auth-kubernetes_f6d85e17.vault-plugin-auth-kubernetes: handle request: transport=gRPC path=config status=finished err=<nil> took=3.289667ms

Backwards compatible

…netes_ca_cert

…ack to the system's trust store

…RootCAs has become nil or at the initial update

- check that the CA cert chain contains at least one valid certificate.

benashz

Looking good. I had a few suggestions/nits for your consideration.

benashz · 2024-03-19T16:19:24Z

CHANGELOG.md

@@ -10,6 +10,10 @@
  * `k8s.io/api` v0.29.1 -> v0.29.2
  * `k8s.io/apimachinery` v0.29.1 -> v0.29.2

+### Improvements
+
+* Make kubernetes_ca_cert optional [GH-238](https://github.com/hashicorp/vault-plugin-auth-kubernetes/pull/238)


I think this is really about having "TLS uses the host's root CA set" if no CA chain is provided - which is a behavioural change.

We should also document the extra validation being done on the provided CA PEM bundle.

backend.go

path_config.go

path_config_test.go

Co-authored-by: Ben Ash <32777270+benashz@users.noreply.github.com>

thyton · 2024-03-26T17:25:29Z

Thank you for your feedback!

benashz

A few minor nits, otherwise 👍

benashz · 2024-03-27T18:20:21Z

path_config.go

+	}
+
+	if caCert != "" && !hasCerts(caCert) {
+		return logical.ErrorResponse("The provided CA PEM data contains no valid certificates"), nil


nit: the line is bit long, may bring the message down one line, as it was before.

benashz · 2024-03-27T18:21:13Z

CHANGELOG.md

@@ -10,6 +10,10 @@
  * `k8s.io/api` v0.29.1 -> v0.29.2
  * `k8s.io/apimachinery` v0.29.1 -> v0.29.2

+### Improvements
+
+* Allow TLS client to use the host's root CA set when no CA certificates are provided and `disable_local_ca_jwt` is true if running Vault in a Kubernetes pod. Additionally, validate the configuration's provided CA PEM bundle. [GH-238](https://github.com/hashicorp/vault-plugin-auth-kubernetes/pull/238)


nit: line is a bit long, mind folding it?

thyton added 2 commits March 12, 2024 18:28

make kubernetes-ca-cert optional and default to local CA cert

ee49a2e

update CHANGELOG

875494f

thyton linked an issue Mar 13, 2024 that may be closed by this pull request

Do not make ca cert or pem keys required #62

Closed

benashz self-requested a review March 14, 2024 15:54

thyton and others added 6 commits March 14, 2024 14:47

do nothing to TSL config when true disable_local_ca_jwt & unset kuber…

cac8247

…netes_ca_cert

skip updating root CAs

a3f2e86

set transport.TLSClientConfig to getDefaultTLSConfig() when falling b…

ef0454e

…ack to the system's trust store

forgot to stage transport.TLSClientConfig = getDefaultTLSConfig() change

e77f4e6

only update transport.TLSClientConfig when transport.TLSClientConfig.…

27b3933

…RootCAs has become nil or at the initial update

Use the SystemCertPool() when no CACert chain is provided

0af6e94

- check that the CA cert chain contains at least one valid certificate.

thyton requested a review from a team as a code owner March 19, 2024 19:42

benashz and others added 3 commits March 19, 2024 16:17

Minor fix ups

f033819

Guard against nil RootCAs

2a8f1d3

add test for invalid CA cert

e6e5bd8

benashz reviewed Mar 26, 2024

View reviewed changes

thyton and others added 5 commits March 26, 2024 09:34

Update path_config_test.go

82ec9fb

Co-authored-by: Ben Ash <32777270+benashz@users.noreply.github.com>

Update path_config.go

698a21b

Co-authored-by: Ben Ash <32777270+benashz@users.noreply.github.com>

Update path_config.go

6b84d97

Co-authored-by: Ben Ash <32777270+benashz@users.noreply.github.com>

update CHANGELOG entry

3a5f5d7

use truncated CA cert for invalid kubernetes_ca_cert test

5025f03

add new line

f1c5acf

thyton requested a review from benashz March 26, 2024 17:40

benashz approved these changes Mar 27, 2024

View reviewed changes

fold long lines

f916f7e

thyton merged commit da08a6a into main Mar 27, 2024
8 checks passed

thyton deleted the VAULT-1729-make-kubernetes-ca-cert-optional branch March 27, 2024 23:04

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

make kubernetes_ca_cert optional #238

make kubernetes_ca_cert optional #238

thyton commented Mar 13, 2024 •

edited

Loading

benashz left a comment

benashz Mar 19, 2024

thyton commented Mar 26, 2024

benashz left a comment

benashz Mar 27, 2024

benashz Mar 27, 2024

make kubernetes_ca_cert optional #238

make kubernetes_ca_cert optional #238

Conversation

thyton commented Mar 13, 2024 • edited Loading

Overview

Design of Change

Related Issues/Pull Requests

Contributor Checklist

benashz left a comment

Choose a reason for hiding this comment

benashz Mar 19, 2024

Choose a reason for hiding this comment

thyton commented Mar 26, 2024

benashz left a comment

Choose a reason for hiding this comment

benashz Mar 27, 2024

Choose a reason for hiding this comment

benashz Mar 27, 2024

Choose a reason for hiding this comment

thyton commented Mar 13, 2024 •

edited

Loading