How to configure my own CA for k3s ? #1868

parekhha · 2020-06-05T10:27:51Z

Hi,
I am trying to configure my own ca with k3s. My expectation is, I give my cacert and cakey to k3s and k3s automatically generate all required cert from that and also rotate those certificate if expired.

Is it possible ?

brandond · 2020-06-05T18:28:46Z

Can you discus your use case a bit? Other than it being fun to experiment with, I'm not sure what the point would be. As far as I know, even the commercial cloud providers just use unique self-signed CAs for each cluster.

k3s actually follows best practices and uses three separate CAs. There aren't any specific arguments or options to load existing CA certs, but you can pre-create the files and put them in the correct location and k3s will use them instead of creating new ones.

Here's an example:

mkdir -p /var/lib/rancher/k3s/server/tls
cd /var/lib/rancher/k3s/server/tls
openssl genrsa -out client-ca.key 2048
openssl genrsa -out server-ca.key 2048
openssl genrsa -out request-header-ca.key 2048
openssl req -x509 -new -nodes -key client-ca.key -sha256 -days 3560 -out client-ca.crt -addext keyUsage=critical,digitalSignature,keyEncipherment,keyCertSign -subj '/CN=k3s-client-ca'
openssl req -x509 -new -nodes -key server-ca.key -sha256 -days 3560 -out server-ca.crt -addext keyUsage=critical,digitalSignature,keyEncipherment,keyCertSign -subj '/CN=k3s-server-ca'
openssl req -x509 -new -nodes -key request-header-ca.key -sha256 -days 3560 -out request-header-ca.crt -addext keyUsage=critical,digitalSignature,keyEncipherment,keyCertSign -subj '/CN=k3s-request-header-ca'

After doing this you can install k3s as usual using install.sh and it should use the openssl-generated certs and keys.

parekhha · 2020-06-08T07:35:04Z

@brandond
Thanks. It worked. But when I rotate these ca certs all certificates are getting re-generated except dynamic-cert.json. Due to this, I am getting unknown cert error while using kubectl. Steps I followed

Stop K3s service
Remove all certs from /var/lib/rancher/k3s/server/tls
Recreate all three CAs.
Start k3s service.

I see all new certs are signed by new ca but dynamic-cert.json is having old certificate. Is it bug or I am missing something ?

brandond · 2020-06-08T07:46:17Z

The k3s-serving cert isn't handled the same as others. It doesn't get rotated when it expires, either - there's an open issue about it. I'm not sure what the best way is to get it updated on an existing cluster. Maybe try deleting it and the linked secret?

parekhha · 2020-06-08T08:53:25Z

@brandond
Yeah, Following steps worked

First delete secret and stop k3s before it recreate secret
Remove all certs from /var/lib/rancher/k3s/server/tls
Recreate all three CAs.
Start k3s service.
Thanks !!!

vntbbb · 2020-08-14T09:35:52Z

@brandond
Yeah, Following steps worked

First delete secret and stop k3s before it recreate secret

Remove all certs from /var/lib/rancher/k3s/server/tls

Recreate all three CAs.

Start k3s service.
Thanks !!!

Hi, parekhha, I do like you,but get error.
my shell:
#!/usr/bin/bash

kubectl -n kube-system delete secret k3s-serving && sleep 2
systemctl stop k3s && sleep 4

#default openssl.cnf: /etc/pki/tls/openssl.cnf
cd tls && rm ./* -rf
openssl genrsa -out client-ca.key 2048
openssl genrsa -out server-ca.key 2048
openssl genrsa -out request-header-ca.key 2048
#echo "gen client-ca.crt"
#openssl req -x509 -new -nodes -key client-ca.key -sha256 -days 3560 -out client-ca.crt -extensions v3_req -subj '/CN=k3s-client-ca'
#echo "gen server-ca.crt"
#openssl req -x509 -new -nodes -key server-ca.key -sha256 -days 3560 -out server-ca.crt -extensions v3_req -subj '/CN=k4s-server-ca'
#echo "gen request-header-ca.crt"
#openssl req -x509 -new -nodes -key request-header-ca.key -sha256 -days 3560 -out request-header-ca.crt -extensions v3_req -subj '/CN=k3s-request-header-ca'

echo "restart k3s ..."
systemctl start k3s

echo "over"
then get errors:
[root@localhost]# systemctl status k3s
● k3s.service - Lightweight Kubernetes
Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: disabled)
Active: active (running) since Fri 2020-08-14 05:22:01 EDT; 6min ago
...
Aug 14 05:28:00 localhost k3s[60167]: W0814 05:28:00.703863 60167 handler_proxy.go:97] no RequestInfo found in the context
Aug 14 05:28:00 localhost k3s[60167]: E0814 05:28:00.703923 60167 controller.go:114] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: faile...navailable
Aug 14 05:28:00 localhost k3s[60167]: , Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
Aug 14 05:28:00 localhost k3s[60167]: I0814 05:28:00.703932 60167 controller.go:127] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.i...d Requeue.
Aug 14 05:28:00 localhost k3s[60167]: I0814 05:28:00.743580 60167 log.go:172] http: TLS handshake error from 10.42.1.11:52766: remote error: tls: bad certificate
Aug 14 05:28:01 localhost k3s[60167]: I0814 05:28:01.638100 60167 log.go:172] http: TLS handshake error from 10.42.1.11:52770: remote error: tls: bad certificate
Aug 14 05:28:01 localhost k3s[60167]: I0814 05:28:01.643213 60167 log.go:172] http: TLS handshake error from 10.42.1.11:52772: remote error: tls: bad certificate
Aug 14 05:28:01 localhost k3s[60167]: I0814 05:28:01.644060 60167 log.go:172] http: TLS handshake error from 10.42.1.9:33172: remote error: tls: bad certificate
Aug 14 05:28:01 localhost k3s[60167]: I0814 05:28:01.646780 60167 log.go:172] http: TLS handshake error from 10.42.1.9:33174: remote error: tls: bad certificate
Aug 14 05:28:01 localhost k3s[60167]: I0814 05:28:01.744863 60167 log.go:172] http: TLS handshake error from 10.42.1.11:52780: remote error: tls: bad certificate
Hint: Some lines were ellipsized, use -l to show in full.

brandond · 2020-08-17T17:40:23Z

It sounds like you have some pods still running with old certificates. If you're going to try to use custom CA certs, you're best off doing it before starting k3s for the first time.

vntbbb · 2020-08-18T01:03:04Z

It sounds like you have some pods still running with old certificates. If you're going to try to use custom CA certs, you're best off doing it before starting k3s for the first time.

Yeah, I create my production cluster by k3s install.sh. Instead of restarting our cluster once a year, we want to generate a long time certificates(e.g., a 5-year serving-kube-apiserver.crt). Is there any tool to generate new certs and apply them to my cluster?

thanks.

brandond · 2020-08-18T02:01:26Z

There is not currently any automated way to do this. Most folks would tell you that you should probably be rebooting to apply kernel and package updates at least once a year. For k3s at least, restarting the service does not affect running pods, so you could just schedule a weekly restart of k3s and not worry about it.

vntbbb · 2020-08-18T02:03:46Z

There is not currently any automated way to do this. Most folks would tell you that you should probably be rebooting to apply kernel and package updates at least once a year. For k3s at least, restarting the service does not affect running pods, so you could just schedule a weekly restart of k3s and not worry about it.

ok, thank you!

ericis · 2021-07-23T16:06:12Z

Possibly related k3d discussion: k3d-io/k3d#687

TonyShanc · 2022-08-22T06:32:52Z

Hi, I use vntbbb 's script above to delete all certs after deleting secret and restart k3s, then recreate three CA certs and start k3s but old CA certs coverd my own CA certs. Is there any update has changed certs creation logic?
os: unbuntu 20.04
k3s version: v1.23.9+k3s1 (f45cf32)

ShylajaDevadiga · 2023-02-14T02:41:41Z

Re-opening for validation

ShylajaDevadiga · 2023-02-15T23:55:38Z

Validated on commit id `2156015` from master branch

Created a 3 node cluster with etcd backend
Validated user-provided CA certificate using certs.sh from the PR

$ curl -fL https://get.k3s.io | INSTALL_K3S_COMMIT=21560155217bdfc12c5105d7cb79fad61b2985aa INSTALL_K3S_SKIP_START=true sh -s - server --token <TOKEN> --cluster-init
$ wget https://raw.githubusercontent.com/brandond/k3s/00a76d82a2d64bc18f589bd9a3bec371c634ee6b/contrib/util/certs.sh


$ chmod u+x certs.sh 
$ sudo ./certs.sh 
$ sudo systemctl enable k3s
$ sudo systemctl start k3s

$ kubectl get nodes
NAME              STATUS   ROLES                       AGE    VERSION
ip-172-31-5-188   Ready    control-plane,etcd,master   15m    v1.26.1+k3s-21560155
ip-172-31-6-21    Ready    control-plane,etcd,master   3m6s   v1.26.1+k3s-21560155
ip-172-31-7-172   Ready    control-plane,etcd,master   12m    v1.26.1+k3s-21560155

$ kubectl get pods -A
NAMESPACE     NAME                                      READY   STATUS      RESTARTS   AGE
kube-system   coredns-5c6b6c5476-gqpd2                  1/1     Running     0          15m
kube-system   helm-install-traefik-crd-nwc9l            0/1     Completed   0          15m
kube-system   helm-install-traefik-lpstb                0/1     Completed   1          15m
kube-system   local-path-provisioner-5d56847996-649z2   1/1     Running     0          15m
kube-system   metrics-server-7b67f64457-m8x9l           1/1     Running     0          15m
kube-system   svclb-traefik-016ce390-22c9c              2/2     Running     0          12m
kube-system   svclb-traefik-016ce390-5px5h              2/2     Running     0          15m
kube-system   svclb-traefik-016ce390-djj42              2/2     Running     0          3m8s
kube-system   traefik-56b8c5fb5c-5n7zc                  1/1     Running     0          15m

dene14 · 2023-08-02T12:38:26Z

Just to save other people's time posting here full script that will generate both server side (3 CAs for k8s) and client side (kubectl, argocd, etc) certificates:

openssl genrsa -out client-ca.key 2048
openssl genrsa -out server-ca.key 2048
openssl genrsa -out request-header-ca.key 2048
openssl req -x509 -new -nodes -key client-ca.key -sha256 -days 3651 -out client-ca.crt -addext keyUsage=critical,digitalSignature,keyEncipherment,keyCertSign -subj '/CN=k3s-client-ca'
openssl req -x509 -new -nodes -key server-ca.key -sha256 -days 3651 -out server-ca.crt -addext keyUsage=critical,digitalSignature,keyEncipherment,keyCertSign -subj '/CN=k3s-server-ca'
openssl req -x509 -new -nodes -key request-header-ca.key -sha256 -days 3651 -out request-header-ca.crt -addext keyUsage=critical,digitalSignature,keyEncipherment,keyCertSign -subj '/CN=k3s-request-header-ca'

# Generating client key/cert
openssl req -newkey rsa:2048 -nodes -keyout client.key -out client.csr -subj "/CN=system:admin/O=system:masters" 
openssl x509 -req -in client.csr -CA client-ca.crt -CAkey client-ca.key -CAcreateserial -out client.crt -days 3650 -extensions v3_ext

echo """
# These keychains to be used on K8S control plane (3 independent CAs)
serverTlsSecret:
  clientCaCrt: `cat client-ca.crt | base64 -w0`
  clientCaKey: `cat client-ca.key | base64 -w0`
  requestHeaderCaCrt: `cat request-header-ca.crt | base64 -w0`
  requestHeaderCaKey: `cat request-header-ca.key | base64 -w0`
  serverCaCrt: `cat server-ca.crt | base64 -w0`
  serverCaKey: `cat server-ca.key | base64 -w0`

"""

echo """
# These keychains to be used on clients like kubectl or argocd
clientTlsSecret:
  certificate-authority-data: `cat server-ca.crt | base64 -w0`
  client-certificate-data: `cat client.crt client-ca.crt | base64 -w0`
  client-key-data: `cat client.key | base64 -w0`
"""
rm -f client-ca.key server-ca.key request-header-ca.key client-ca.crt server-ca.crt request-header-ca.crt client.csr client.key client.crt

brandond · 2023-08-02T14:35:12Z

@dene14 there is actually a page for this in the docs, along with a script to do it for you.

https://docs.k3s.io/cli/certificate#using-custom-ca-certificates

parekhha closed this as completed Jun 8, 2020

brandond mentioned this issue Jul 1, 2020

Start server with predetermined kubeconfig #1984

Closed

dweomer mentioned this issue Sep 25, 2020

How to install your own certificates for :6443 rancher/k3os#582

Closed

brandond mentioned this issue Feb 23, 2023

Need to document steps for manual CA certificate generation / rotation rancher/rke2-docs#59

Closed

gbarton mentioned this issue Nov 20, 2020

Document how to manage certificates in multi master HA setup #2560

Closed

TonyShanc mentioned this issue Aug 22, 2022

can not config my own CA certificates #6020

Closed

abe-winter mentioned this issue Sep 25, 2022

can you expand the docs for K3S_TOKEN security #6175

Closed

brandond mentioned this issue Dec 7, 2022

Add support for user-provided CA certificates #6615

Merged

caroline-suse-rancher added this to K3s Development Feb 9, 2023

caroline-suse-rancher added this to the v1.26.2+k3s1 milestone Feb 9, 2023

caroline-suse-rancher moved this to To Test in K3s Development Feb 9, 2023

bguzman-3pillar assigned ShylajaDevadiga Feb 10, 2023

ShylajaDevadiga reopened this Feb 14, 2023

ShylajaDevadiga closed this as completed Feb 15, 2023

github-project-automation bot moved this from To Test to Done Issue in K3s Development Feb 15, 2023

brandond mentioned this issue Mar 7, 2023

Improve support for rotating the default self-signed certs #7032

Merged

MohamedZ3kri mentioned this issue Apr 3, 2023

Add support for InitContainers in the charts loft-sh/vcluster#979

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

How to configure my own CA for k3s ? #1868

How to configure my own CA for k3s ? #1868

parekhha commented Jun 5, 2020 •

edited

Loading

brandond commented Jun 5, 2020 •

edited

Loading

parekhha commented Jun 8, 2020

brandond commented Jun 8, 2020 •

edited

Loading

parekhha commented Jun 8, 2020 •

edited

Loading

vntbbb commented Aug 14, 2020

brandond commented Aug 17, 2020

vntbbb commented Aug 18, 2020

brandond commented Aug 18, 2020 •

edited

Loading

vntbbb commented Aug 18, 2020

ericis commented Jul 23, 2021

TonyShanc commented Aug 22, 2022

ShylajaDevadiga commented Feb 14, 2023

ShylajaDevadiga commented Feb 15, 2023

dene14 commented Aug 2, 2023

brandond commented Aug 2, 2023

How to configure my own CA for k3s ? #1868

How to configure my own CA for k3s ? #1868

Comments

parekhha commented Jun 5, 2020 • edited Loading

brandond commented Jun 5, 2020 • edited Loading

parekhha commented Jun 8, 2020

brandond commented Jun 8, 2020 • edited Loading

parekhha commented Jun 8, 2020 • edited Loading

vntbbb commented Aug 14, 2020

brandond commented Aug 17, 2020

vntbbb commented Aug 18, 2020

brandond commented Aug 18, 2020 • edited Loading

vntbbb commented Aug 18, 2020

ericis commented Jul 23, 2021

TonyShanc commented Aug 22, 2022

ShylajaDevadiga commented Feb 14, 2023

ShylajaDevadiga commented Feb 15, 2023

Validated on commit id 2156015 from master branch

dene14 commented Aug 2, 2023

brandond commented Aug 2, 2023

parekhha commented Jun 5, 2020 •

edited

Loading

brandond commented Jun 5, 2020 •

edited

Loading

brandond commented Jun 8, 2020 •

edited

Loading

parekhha commented Jun 8, 2020 •

edited

Loading

brandond commented Aug 18, 2020 •

edited

Loading

Validated on commit id `2156015` from master branch