Trust the Kubernetes CA on nodes #1351

coreypobrien · 2017-07-11T12:35:05Z

Fixes #736

This is specifically useful for running a private Docker registry that uses the same signing chain as Kubernetes since Docker uses the system certificate store and needs to trust the CA for the registry.

Because Docker uses the system certificate store, installing the certificate during the ignition process avoids needing to restart Docker to pick up the changes later.

I've only manually tested the openstack/neutron and metal platforms. The aws, azure, and vmware changes were just a copy-paste from openstack/neutron, but if someone could check those more closely I'd appreciate it.

coreosbot · 2017-07-11T12:35:07Z

Can one of the admins verify this patch?

coreosbot · 2017-07-11T12:36:05Z

Can one of the admins verify this patch?

s-urbaniak · 2017-07-11T13:29:39Z

platforms/metal/variables.tf

@@ -185,3 +185,8 @@ SSH public key to use as an authorized key.
 Example: `ssh-rsa AAAB3N...`
 EOF
 }
+
+variable "tectonic_ssh_private_key" {


I believe this is unrelated to this PR.

Additionally I would like to avoid encoding private keys in the state. As you are using this currently only to transfer assets I would prefer not using the private key but rather rely on the host's SSH agent.

oop, good call. That bled over from the rebase I was doing for #625. I'll pull that out.

s-urbaniak

Thanks a lot for the PR! I have two questions/comments though.

s-urbaniak · 2017-07-11T13:32:05Z

modules/openstack/nodes/ignition.tf

+
+[Service]
+ExecStart=
+ExecStart=/usr/sbin/update-ca-certificates


That would execute update-ca-certificates at every boot, no? I would like to recheck with the OS team (/cc @crawford @euank) whether this is the canonical way of adding public CAs using ignition as I think https://github.com/coreos/coreos-overlay/blob/master/app-misc/ca-certificates/files/update-ca-certificates.service#L7 does this already at first boot.

Sounds good. As it currently is, the ConditionPathIsSymbolicLink and the --skip-rehash can cause it not to pick up the new certificate, so I removed those, but I'm open to a more canonical way.

@crawford @euank any thoughts to add here?

Apologies for not responding; this dropped off my radar.

Do you know of any specific cases where the existing ExecStart line actually does miss new certificates? It looks like it should generally do the right thing already with its timestamp comparison.

I'm not 100% clear on the reasoning, but it definitely requires both to get this to work. Here's what happens if I remove the drop-in, as well as with the drop-in changes:

# This is what we're looking for in the CA list core@terraform-master-0 ~ $ sudo cat /etc/ssl/certs/kube_ca.pem | openssl x509 -noout -subject subject= /C=/ST=/L=/postalCode=/O=bootkube/OU=/CN=kube-ca # update-ca-certificates didn't run because of the precondition (maybe already ran during the image build?) core@terraform-master-0 ~ $ systemctl status update-ca-certificates ● update-ca-certificates.service - Update CA bundle at /etc/ssl/certs/ca-certificates.crt Loaded: loaded (/usr/lib/systemd/system/update-ca-certificates.service; static; vendor preset: disabled) Active: inactive (dead) Condition: start condition failed at Wed 2017-07-26 19:33:02 UTC; 18min ago └─ ConditionPathIsSymbolicLink=!/etc/ssl/certs/ca-certificates.crt was not met core@terraform-master-0 ~ $ sudo mkdir /etc/systemd/system/update-ca-certificates.service.d/ # Just reset ConditionPathIsSymbolicLink in the drop-in core@terraform-master-0 ~ $ sudo vi /etc/systemd/system/update-ca-certificates.service.d/edits.conf core@terraform-master-0 ~ $ sudo systemctl daemon-reload core@terraform-master-0 ~ $ sudo systemctl start update-ca-certificates # update-ca-certificates runs now but doesn't do much core@terraform-master-0 ~ $ systemctl status update-ca-certificates ● update-ca-certificates.service - Update CA bundle at /etc/ssl/certs/ca-certificates.crt Loaded: loaded (/usr/lib/systemd/system/update-ca-certificates.service; static; vendor preset: disabled) Drop-In: /etc/systemd/system/update-ca-certificates.service.d └─edits.conf Active: inactive (dead) since Wed 2017-07-26 20:06:42 UTC; 6s ago Process: 6327 ExecStart=/usr/sbin/update-ca-certificates --skip-rehash (code=exited, status=0/SUCCESS) Main PID: 6327 (code=exited, status=0/SUCCESS) CPU: 34ms Jul 26 20:06:42 terraform-master-0 systemd[1]: Starting Update CA bundle at /etc/ssl/certs/ca-certificates.crt... Jul 26 20:06:42 terraform-master-0 update-ca-certificates[6327]: Recreating certificate bundle /etc/ssl/certs/ca-certificates.crt Jul 26 20:06:42 terraform-master-0 systemd[1]: Started Update CA bundle at /etc/ssl/certs/ca-certificates.crt. # Still no certificate in the list core@terraform-master-0 ~ $ awk -v cmd='openssl x509 -noout -subject' '/BEGIN/{close(cmd)};{print | cmd}' < /etc/ssl/certs/ca-certificates.crt | grep kube # Change the ExecStart too core@terraform-master-0 ~ $ sudo vi /etc/systemd/system/update-ca-certificates.service.d/edits.conf core@terraform-master-0 ~ $ sudo systemctl daemon-reload core@terraform-master-0 ~ $ sudo systemctl start update-ca-certificates core@terraform-master-0 ~ $ systemctl status update-ca-certificates ● update-ca-certificates.service - Update CA bundle at /etc/ssl/certs/ca-certificates.crt Loaded: loaded (/usr/lib/systemd/system/update-ca-certificates.service; static; vendor preset: disabled) Drop-In: /etc/systemd/system/update-ca-certificates.service.d └─edits.conf Active: inactive (dead) since Wed 2017-07-26 20:07:43 UTC; 2min 52s ago Process: 6691 ExecStart=/usr/sbin/update-ca-certificates (code=exited, status=0/SUCCESS) Main PID: 6691 (code=exited, status=0/SUCCESS) CPU: 5.597s Jul 26 20:07:43 terraform-master-0 update-ca-certificates[6691]: WoSign_China.pem => 5d63b0ae.0 Jul 26 20:07:43 terraform-master-0 update-ca-certificates[6691]: XRamp_Global_CA_Root.pem => 706f604c.0 Jul 26 20:07:43 terraform-master-0 update-ca-certificates[6691]: certSIGN_ROOT_CA.pem => 8d86cdd1.0 Jul 26 20:07:43 terraform-master-0 update-ca-certificates[6691]: ePKI_Root_Certification_Authority.pem => ca6e4ad9.0 Jul 26 20:07:43 terraform-master-0 update-ca-certificates[6691]: kube_ca.pem => dc214808.0 Jul 26 20:07:43 terraform-master-0 update-ca-certificates[6691]: thawte_Primary_Root_CA.pem => 2e4eed3c.0 Jul 26 20:07:43 terraform-master-0 update-ca-certificates[6691]: thawte_Primary_Root_CA_-_G2.pem => c089bbbd.0 Jul 26 20:07:43 terraform-master-0 update-ca-certificates[6691]: thawte_Primary_Root_CA_-_G3.pem => ba89ed3b.0 Jul 26 20:07:43 terraform-master-0 update-ca-certificates[6691]: Recreating certificate bundle /etc/ssl/certs/ca-certificates.crt Jul 26 20:07:43 terraform-master-0 systemd[1]: Started Update CA bundle at /etc/ssl/certs/ca-certificates.crt. # There it is core@terraform-master-0 ~ $ awk -v cmd='openssl x509 -noout -subject' '/BEGIN/{close(cmd)};{print | cmd}' < /etc/ssl/certs/ca-certificates.crt | grep kube subject= /C=/ST=/L=/postalCode=/O=bootkube/OU=/CN=kube-ca

Thanks for checking, sorry for the bit of a runaround there.

This still seems a bit sloppy, but since I don't have a better suggestion, I'm okay with it.

s-urbaniak · 2017-07-11T13:33:35Z

platforms/metal/variables.tf

@@ -185,3 +185,8 @@ SSH public key to use as an authorized key.
 Example: `ssh-rsa AAAB3N...`
 EOF
 }
+
+variable "tectonic_ssh_private_key" {


Additionally I would like to avoid encoding private keys in the state. As you are using this currently only to transfer assets I would prefer not using the private key but rather rely on the host's SSH agent.

euank · 2017-07-26T20:45:41Z

platforms/metal/cl/bootkube-worker.yaml.tmpl

@@ -101,6 +101,23 @@ storage:
 contents:
 inline: |
 fs.inotify.max_user_watches=16184
+ - path: /etc/systemd/system/update-ca-certificates.service.d/10-alwaysrun.conf


This should be part of the systemd: section, not the storage: section

Oop, good catch. Moved in the controller too.

euank · 2017-07-26T20:48:08Z

modules/openstack/nodes/ignition.tf

+
+[Service]
+ExecStart=
+ExecStart=/usr/sbin/update-ca-certificates


Thanks for checking, sorry for the bit of a runaround there.

This still seems a bit sloppy, but since I don't have a better suggestion, I'm okay with it.

coreos#1351 From 013-install-kube-ca.patch

lblackstone · 2017-08-16T15:22:42Z

@coreypobrien @s-urbaniak Is this good to go after a rebase?

coreosbot · 2017-08-16T15:22:44Z

Can one of the admins verify this patch?

coreosbot · 2017-08-16T15:22:44Z

Can one of the admins verify this patch?

coreypobrien · 2017-08-17T12:32:37Z

@lblackstone Rebased and should be good to go. Thanks!

lblackstone · 2017-08-17T15:42:48Z

This still seems a bit sloppy, but since I don't have a better suggestion, I'm okay with it.

@s-urbaniak Any further objections?

lblackstone · 2017-08-28T21:24:52Z

Ping @s-urbaniak

coreosbot · 2017-08-28T21:24:55Z

Can one of the admins verify this patch?

coreosbot · 2017-08-28T21:24:55Z

Can one of the admins verify this patch?

s-urbaniak · 2017-08-29T08:03:43Z

@lblackstone thanks a lot for the patience. This generally LGTM to me. Note though, that we did a major refactoring which unifies ignition across all platforms in #1743. I would therefore advocate to put all the systemd units in the shiny new modules/ignition module. I am sorry that this means a bit of rework for this PR, but we needed to do the factoring to reduce skew between platforms.

lblackstone · 2017-08-29T15:55:44Z

@s-urbaniak No problem, I understand things are moving fast! :)

@coreypobrien Given the extra effort required at this point, want me to take over this PR?

s-urbaniak · 2017-08-29T15:56:58Z

@lblackstone thanks a lot! 👍

coreypobrien · 2017-08-30T01:21:00Z

@lblackstone I think I've got it rebased, but I haven't had a chance to test any of the platforms yet. Let's see how CI looks and if you wouldn't mind testing OpenStack that'd be great.

s-urbaniak · 2017-09-01T10:09:13Z

modules/ignition/assets.tf

@@ -28,6 +40,22 @@ data "ignition_systemd_unit" "docker_dropin" {
 ]
 }

+data "template_file" "update_ca_certificates_dropin" {
+ template = "${file("${path.module}/resources/dropins/10-dockeropts.conf")}"


I believe this should be 10-always-update-ca-certificates.conf, no?

s-urbaniak · 2017-09-01T11:04:42Z

just fyi: I am actively working on this. My plan is to extend the functionality of this PR and not only inject the kubelet CA, but also any number of additional CA files.

lblackstone · 2017-09-01T14:32:24Z

@s-urbaniak Thanks for the info. I haven't had a chance to test this out anyway, so I'll just let you run with your changes.

lblackstone · 2017-10-02T16:20:16Z

@s-urbaniak Is this change in place now? Looks like d38ebc9 may have included it?

coreosbot · 2017-10-11T16:00:10Z

Can one of the admins verify this patch?

coreosbot · 2017-10-11T16:00:12Z

Can one of the admins verify this patch?

lblackstone · 2017-11-27T16:35:36Z

@s-urbaniak #2362 fixes this, correct?

s-urbaniak · 2017-11-27T21:55:03Z

@lblackstone correct, closing in favor of #2362

s-urbaniak reviewed Jul 11, 2017

View reviewed changes

s-urbaniak suggested changes Jul 11, 2017

View reviewed changes

coreypobrien force-pushed the kubeca branch from fde86df to 77b0e45 Compare July 11, 2017 13:47

cpanato added the waiting-response label Jul 20, 2017

coreypobrien force-pushed the kubeca branch from 77b0e45 to ad94a20 Compare July 26, 2017 15:42

euank reviewed Jul 26, 2017

View reviewed changes

coreypobrien force-pushed the kubeca branch from ad94a20 to 9a2eb6f Compare July 27, 2017 00:42

lblackstone pushed a commit to rcbops/tectonic-installer that referenced this pull request Aug 7, 2017

Install kube CA

06414c5

coreos#1351 From 013-install-kube-ca.patch

coreypobrien force-pushed the kubeca branch from 9a2eb6f to ecdbe21 Compare August 17, 2017 12:28

cpanato removed the waiting-response label Aug 28, 2017

Trust the Kubernetes CA on nodes

bb4f308

coreypobrien force-pushed the kubeca branch from ecdbe21 to bb4f308 Compare August 30, 2017 01:19

s-urbaniak reviewed Sep 1, 2017

View reviewed changes

cpanato added the waiting-response label Sep 7, 2017

s-urbaniak closed this Nov 27, 2017

Trust the Kubernetes CA on nodes #1351

Trust the Kubernetes CA on nodes #1351

Conversation

coreypobrien commented Jul 11, 2017

coreosbot commented Jul 11, 2017

coreosbot commented Jul 11, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

s-urbaniak left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lblackstone commented Aug 16, 2017

coreosbot commented Aug 16, 2017

coreosbot commented Aug 16, 2017

coreypobrien commented Aug 17, 2017

lblackstone commented Aug 17, 2017

lblackstone commented Aug 28, 2017

coreosbot commented Aug 28, 2017

coreosbot commented Aug 28, 2017

s-urbaniak commented Aug 29, 2017

lblackstone commented Aug 29, 2017

s-urbaniak commented Aug 29, 2017

coreypobrien commented Aug 30, 2017

Choose a reason for hiding this comment

s-urbaniak commented Sep 1, 2017

lblackstone commented Sep 1, 2017

lblackstone commented Oct 2, 2017

coreosbot commented Oct 11, 2017

coreosbot commented Oct 11, 2017

lblackstone commented Nov 27, 2017

s-urbaniak commented Nov 27, 2017