[OSDOCS#10055]: Document manual rotation of etcd signer certificates

[tmalove]

OSDOCS-10055

Note: This PR has been reviewed, however this version includes updates from the initial peer view. Thanks!

Version(s):
4.16+

Link to docs preview: etcd certificates (Updated 6/21/2024)

QE review:

• [ x] QE has approved this change.

[ocpdocs-previewbot]

🤖 Fri Jun 21 21:00:55 - Prow CI generated the docs preview:

https://77406--ocpdocs-pr.netlify.app/openshift-enterprise/latest/security/certificate_types_descriptions/etcd-certificates.html

[tmalove]

/retest

[tjungblu]

static pod

[tjungblu]

the CA doesn't rotate (yet), etcd restarts :)

[tmalove]

:-) Thanks Thomas!

[tjungblu]

looks good as a whole! we need to also mention the metrics-signer-ca rotation, which is equally important for metrics and alerting

[lahinson]

@tmalove Try adding a + here and see if that helps.

[tmalove]

@lahinson the problem was the file suffix...it is corrected now, thanks!

[tmalove]

/ok-to-test

[tmalove]

/retest

[geliu2016]

/lgtm
and I tried doc steps with ocp 4.16, and etcd restarted failure after removed etcd-signer, I will try more to make sure root cause.

[tjungblu]

/lgtm

@geliu2016 ensure that you don't delete the signer in the openshift-config namespace, that would indeed create failures

[tmalove]

/label peer-review-needed

[jeana-redhat]

Left some suggestions but overall great addition!

/remove-label peer-review-in-progress
/remove-label peer-review-needed
/label peer-review-done

[jeana-redhat]

Suggested change

	* xref:../../security/certificate_types_descriptions/etcd-certificates.adoc#rotating-certificate-authority_cert-types-etcd-certificates[Rotating the etcd certificate].
	* xref:../../security/certificate_types_descriptions/etcd-certificates.adoc#rotating-certificate-authority_cert-types-etcd-certificates[Rotating the etcd certificate]

[jeana-redhat]

Suggested change

	. Verify the remaining lifetime of the new signer certificate:
	. Verify the remaining lifetime of the new signer certificate by running the following command:

[jeana-redhat]

Suggested change

	. Re-create the signer, if the remaining lifetime is close to the current date, by deleting the signer and wait for the static pod rollout:
	+
	[source,terminal]
	----
	$ oc delete secret -n openshift-etcd etcd-signer
	----
	+
	[source,terminal]
	----
	$ oc wait --for=condition=Progressing=False --timeout=15m clusteroperator/etcd
	----
	. If the remaining lifetime is close to the current date, re-create the signer by running the following commands:
	.. Delete the signer:
	+
	[source,terminal]
	----
	$ oc delete secret -n openshift-etcd etcd-signer
	----
	.. Wait for the static pod rollout
	+
	[source,terminal]
	----
	$ oc wait --for=condition=Progressing=False --timeout=15m clusteroperator/etcd
	----

[jeana-redhat]

Suggested change

	. After `etcd` restarts, you can switch the original certificate authority(CA) in the `openshift-config` namespace with the new, rotated one in `openshift-etcd`.
	. After `etcd` restarts, switch the original CA in the `openshift-config` namespace with the new, rotated one in `openshift-etcd`.

[jeana-redhat]

We should document a single way to do this, unless there is a specific use case for a second method. In this instance, if the two methods are equivalent, I would lean towards the single-command step for simplicity.

Suggested change

	. After `etcd` restarts, you can switch the original certificate authority(CA) in the `openshift-config` namespace with the new, rotated one in `openshift-etcd`.
	+
	[source,terminal]
	----
	$ oc get secret etcd-signer -n openshift-etcd -ojson | jq 'del(.metadata["namespace","creationTimestamp","resourceVersion","selfLink","uid"])' | oc apply -n openshift-config -f -
	----
	+
	[source,terminal]
	----
	$ oc wait --for=condition=Progressing=False --timeout=25m clusteroperator/etcd
	----
	+
	[source,terminal]
	----
	$ oc wait --for=condition=Progressing=False --timeout=25m clusteroperator/kube-apiserver
	----
	You can also use this single command to switch the CA:
	[source,terminal]
	----
	$ oc adm wait-for-stable-cluster
	----
	. After `etcd` restarts, switch the original CA in the `openshift-config` namespace with the new, rotated one in `openshift-etcd` by running the following command:
	+
	[source,terminal]
	----
	$ oc adm wait-for-stable-cluster
	----

If you need to use the multi-command version, it should be split into substeps (similar to my suggestion for step 2 above) so we have a single command per step.

[hasbro17]

The oc adm wait-for-stable-cluster cmd does the equivalent check as the previous two cmds (and more) but will wait until that required conditions are true for a period of 5 mins (--minimum-stable-period 5m) so in practice it might take longer to wait on that step than the individual cmds.

$ oc adm wait-for-stable-cluster -h
Wait for all OCP v4 clusteroperators to report Available=true, Progressing=false, Degraded=false.

Examples:
  # Wait for all clusteroperators to become stable
  oc adm clusteroperator wait-for-stable-cluster

  # Consider operators to be stable if they report as such for 5 minutes straight
  oc adm clusteroperator wait-for-stable-cluster --minimum-stable-period 5m

Options:
    --minimum-stable-period=5m0s:
	minimum duration to consider a cluster stable. Defaults to 5 minutes.

    --timeout=1h0m0s:
	duration before the command times out. Defaults to 1 hour.

That may be a long time from the user's perspective so maybe a note here that the oc adm wait-for-stable-cluster will wait for a minimum of 5 mins by default.

@tjungblu Do you think we need to wait that long in practice or if we can shorten it e.g oc adm wait-for-stable-cluster --minimum-stable-period 2m

[tjungblu]

2m is enough for sure

[hasbro17]

Sounds good, thanks for confirming. Then we can combine those into the following. Note we still need the step to switch out the CA first and then wait with the singular cmd for 2m.

Suggested change

	. After `etcd` restarts, you can switch the original certificate authority(CA) in the `openshift-config` namespace with the new, rotated one in `openshift-etcd`.
	+
	[source,terminal]
	----
	$ oc get secret etcd-signer -n openshift-etcd -ojson | jq 'del(.metadata["namespace","creationTimestamp","resourceVersion","selfLink","uid"])' | oc apply -n openshift-config -f -
	----
	+
	[source,terminal]
	----
	$ oc wait --for=condition=Progressing=False --timeout=25m clusteroperator/etcd
	----
	+
	[source,terminal]
	----
	$ oc wait --for=condition=Progressing=False --timeout=25m clusteroperator/kube-apiserver
	----
	You can also use this single command to switch the CA:
	[source,terminal]
	----
	$ oc adm wait-for-stable-cluster
	----
	. After `etcd` restarts, you can switch the original certificate authority(CA) in the `openshift-config` namespace with the new, rotated one in `openshift-etcd`.
	+
	[source,terminal]
	----
	$ oc get secret etcd-signer -n openshift-etcd -ojson | jq 'del(.metadata["namespace","creationTimestamp","resourceVersion","selfLink","uid"])' | oc apply -n openshift-config -f -
	----
	+
	You can then wait for the cluster operators to rollout and become stable:
	[source,terminal]
	----
	$ oc adm wait-for-stable-cluster --minimum-stable-period 2m
	----

[jeana-redhat]

Do you know for sure if our tooling handles this well (looks good on the preview, but I haven't seen it before so wonder about the Portal 😅)

[tmalove]

@jeana-redhat I did a quick look at it, but couldn't find an example online. I decided that because it was documented in our doc references, that it would not be an issue. :-) I will make a note to verify this list after it hits the portal! Thanks for that input!

[mburke5678]

Oooh. I do not like that look at all! 😀

[kalexand-rh]

Please validate that this looks ok on docs.redhat after GA.

[jeana-redhat]

Suggested change

	You can rotate the certificate:
	* When you receive an expiration alert
	* When the private key is leaked
	You can rotate the certificate for the following reasons:
	* You receive an expiration alert
	* The private key is leaked

[jeana-redhat]

Honestly wonder if this is more important than a note 🤔

Suggested change

	NOTE: When a private key is leaked, you must rotate all of the certificates.
	[NOTE]
	====
	When a private key is leaked, you must rotate all of the certificates.
	====

[tmalove]

@jeana-redhat I can't find it now where I saw this construct for an admonition documented (but I will!). However, I will make this change and also change it to 'IMPORTANT', because I agree that it is more impacting than having it as a note.

[jeana-redhat]

two tiny nits otherwise LGTM

[jeana-redhat]

Missing a + here to attach the command to the step

Suggested change

	+

[jeana-redhat]

Suggested change

	. Wait for the cluster Operators to rollout and stabilize by running the following command:
	. Wait for the cluster Operators to roll out and stabilize by running the following command:

[tmalove]

/remove-label peer-review-done
/label peer-review-needed

[openshift-ci]

New changes are detected. LGTM label has been removed.

[mburke5678]

Per ISG: If the list items comprise only complete sentences, include a period after each sentence.

Suggested change

	* You receive an expiration alert
	* The private key is leaked
	* You receive an expiration alert.
	* The private key is leaked.

[mburke5678]

I assume a standard user would know when a key is leaked and what that means....

[mburke5678]

I think that putting the lifetime phrase first ties in better with the previous step.

Suggested change

	. Re-create the signer, if the remaining lifetime is close to the current date, by deleting the signer and wait for the static pod rollout by running the following commands:
	. If the remaining lifetime is close to the current date, re-create the signer by deleting the signer and wait for the static pod rollout by running the following commands:

What should the user do if the remaining lifetime is not close to the current date?

[mburke5678]

@tmalove A few nits. Otherwise LGTM. Don't forget to squash!

[tmalove]

/label merge-review-needed

[kalexand-rh]

Suggested change

	== etcd certificate rotation alerts and metrics signer certificates
	= etcd certificate rotation alerts and metrics signer certificates

The first heading in a module is always an H1. Adjust the leveloffset in the assembly, if necessary.

[kalexand-rh]

Suggested change

	== Rotating the etcd certificate
	= Rotating the etcd certificate

The first heading in a module is always an H1. Adjust the leveloffset in the assembly, if necessary.

[kalexand-rh]

Suggested change

	etcd certificates are used for encrypted communication between etcd member peers, and encrypted client traffic. The following certificates are generated and used by etcd and other processes that communicate with etcd:
	etcd certificates are used for encrypted communication between etcd member peers and encrypted client traffic. The following certificates are generated and used by etcd and other processes that communicate with etcd:

[kalexand-rh]

Please validate that this looks ok on docs.redhat after GA.

[openshift-ci]

@tmalove: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[kalexand-rh]

/cherrypick enterprise-4.16

[openshift-cherrypick-robot]

@kalexand-rh: new pull request created: #77935

In response to this:

/cherrypick enterprise-4.16

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.