Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use wildcard certificates for MinIO Pods #161

Merged
merged 1 commit into from
Jul 2, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 1 addition & 8 deletions docs/kes.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,11 +47,4 @@ KES uses CSR for self signed certificate generation. KES requires three certific
- X.509 certificate for the MinIO server and the corresponding private key.
- X.509 certificate for the KES client (MinIO is the KES client in this case) and the corresponding private key.

Accordingly, you'll need to approve three CSR requests, using below approach

```
kubectl get csr
kubectl certificate approve <csr-name>
```

Once all the CSRs are approved, MinIO Operator will deploy KES Pods and start MinIO Server with KES integration.
If `requestAutoCert` is enabled, Operator automatically creates the relevant CSRs and Certificates.
18 changes: 2 additions & 16 deletions docs/tls.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ This document explains how to enable TLS on MinIOInstance pods. These are the ap

## Automatic TLS

This approach creates TLS certificates automatically using the Kubernetes cluster root Certificate Authority (CA) to establish trust. In this approach, MinIO Operator creates a private key, and a certificate signing request (CSR) which is submitted via the `certificates.k8s.io` API for signing. Approving the CSR is done on a one off basis by a Kubernetes cluster administrator. Automatic TLS approach creates other certificates required for KES as well as explained in [KES document](./kes.md).
This approach creates TLS certificates automatically using the Kubernetes cluster root Certificate Authority (CA) to establish trust. In this approach, MinIO Operator creates a private key, and a certificate signing request (CSR) which is submitted via the `certificates.k8s.io` API for signing. Automatic TLS approach creates other certificates required for KES as well as explained in [KES document](./kes.md).

To enable automatic CSR generation on MinIOInstance, set `requestAutoCert` field in the config file to `true`. Optionally you can also pass additional configuration parameters to be used under `certConfig` section. The `certConfig` section currently supports below fields:

Expand All @@ -17,21 +17,7 @@ To enable automatic CSR generation on MinIOInstance, set `requestAutoCert` field

- DNSNames: By default set to list of all pod DNS names that are part of current MinIOInstance cluster. Any value added under this section will be appended to the list of existing pod DNS names.

Once you enable `requestAutoCert` field and create the MinIOInstance, MinIO Operator creates a CSR for this instance and sends to the Kubernetes API server. MinIO Operator will then wait for the CSR to be approved (wait timeout is 20 minutes). CSR can be approved manually via `kubectl` using below steps

- Get the CSR

```bash
kubectl get csr
```

- Approve the CSR

```bash
kubectl certificate approve <CSR_Name>
```

Once the CSR is approved and Certificate available, MinIO operator downloads the certificate and then mounts the Private Key and Certificate within the MinIOInstance pod.
Once you enable `requestAutoCert` field and create the MinIOInstance, MinIO Operator creates a CSR for this instance and sends to the Kubernetes API server. MinIO Operator will then approve the CSR. After the CSR is approved and Certificate available, MinIO operator downloads the certificate and then mounts the Private Key and Certificate within the MinIOInstance pod.

## Pass Certificate Secret to MinIOInstance

Expand Down
10 changes: 1 addition & 9 deletions pkg/apis/operator.min.io/v1/helper.go
Original file line number Diff line number Diff line change
Expand Up @@ -274,16 +274,8 @@ func (mi *MinIOInstance) TemplatedMinIOHosts(hostsTemplate string) []string {
// AllMinIOHosts returns the all the individual domain names relevant for current MinIOInstance
func (mi *MinIOInstance) AllMinIOHosts() []string {
hosts := make([]string, 0)
var max, index int32
for _, z := range mi.Spec.Zones {
max = max + z.Servers
for index < max {
hosts = append(hosts, fmt.Sprintf("%s-"+strconv.Itoa(int(index))+".%s.%s.svc.%s", mi.MinIOStatefulSetName(), mi.MinIOHLServiceName(), mi.Namespace, ClusterDomain))
index++
}
}
hosts = append(hosts, mi.MinIOCIServiceHost())
hosts = append(hosts, mi.MinIOHeadlessServiceHost())
hosts = append(hosts, "*."+mi.MinIOHeadlessServiceHost())
return hosts
}

Expand Down
28 changes: 23 additions & 5 deletions pkg/controller/cluster/csr.go
Original file line number Diff line number Diff line change
Expand Up @@ -122,7 +122,7 @@ func (c *Controller) createCSR(ctx context.Context, mi *miniov1.MinIOInstance) e
return err
}

err = c.submitCSR(ctx, mi.MinIOPodLabels(), mi.MinIOCSRName(), mi.Namespace, csrBytes, mi)
err = c.createCertificate(ctx, mi.MinIOPodLabels(), mi.MinIOCSRName(), mi.Namespace, csrBytes, mi)
if err != nil {
klog.Errorf("Unexpected error during the creation of the csr/%s: %v", mi.MinIOCSRName(), err)
return err
Expand All @@ -148,8 +148,8 @@ func (c *Controller) createCSR(ctx context.Context, mi *miniov1.MinIOInstance) e
return nil
}

// SubmitCSR is equivalent to kubectl create ${CSR}, if the override is configured, it becomes kubectl apply ${CSR}
func (c *Controller) submitCSR(ctx context.Context, labels map[string]string, name, namespace string, csrBytes []byte, mi *miniov1.MinIOInstance) error {
// createCertificate is equivalent to kubectl create <csr-name> and kubectl approve csr <csr-name>
func (c *Controller) createCertificate(ctx context.Context, labels map[string]string, name, namespace string, csrBytes []byte, mi *miniov1.MinIOInstance) error {
encodedBytes := pem.EncodeToMemory(&pem.Block{Type: csrType, Bytes: csrBytes})

kubeCSR := &certificates.CertificateSigningRequest{
Expand Down Expand Up @@ -179,11 +179,29 @@ func (c *Controller) submitCSR(ctx context.Context, labels map[string]string, na
},
},
}
cOpts := metav1.CreateOptions{}
_, err := c.certClient.CertificateSigningRequests().Create(ctx, kubeCSR, cOpts)

ks, err := c.certClient.CertificateSigningRequests().Create(ctx, kubeCSR, metav1.CreateOptions{})
if err != nil {
return err
}

// Update the CSR to be approved automatically
ks.Status = certificates.CertificateSigningRequestStatus{
Conditions: []certificates.CertificateSigningRequestCondition{
{
Type: certificates.CertificateApproved,
Reason: "MinIOOperatorAutoApproval",
Message: "Automatically approved by MinIO Operator",
LastUpdateTime: metav1.NewTime(time.Now()),
},
},
}

_, err = c.certClient.CertificateSigningRequests().UpdateApproval(ctx, ks, metav1.UpdateOptions{})
if err != nil {
return err
}

return nil
}

Expand Down
4 changes: 2 additions & 2 deletions pkg/controller/cluster/kes-csr.go
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ func (c *Controller) createKESTLSCSR(ctx context.Context, mi *miniov1.MinIOInsta
return err
}

err = c.submitCSR(ctx, mi.KESPodLabels(), mi.KESCSRName(), mi.Namespace, csrBytes, mi)
err = c.createCertificate(ctx, mi.KESPodLabels(), mi.KESCSRName(), mi.Namespace, csrBytes, mi)
if err != nil {
klog.Errorf("Unexpected error during the creation of the csr/%s: %v", mi.KESCSRName(), err)
return err
Expand Down Expand Up @@ -107,7 +107,7 @@ func (c *Controller) createMinIOClientTLSCSR(ctx context.Context, mi *miniov1.Mi
return err
}

err = c.submitCSR(ctx, mi.MinIOPodLabels(), mi.MinIOClientCSRName(), mi.Namespace, csrBytes, mi)
err = c.createCertificate(ctx, mi.MinIOPodLabels(), mi.MinIOClientCSRName(), mi.Namespace, csrBytes, mi)
if err != nil {
klog.Errorf("Unexpected error during the creation of the csr/%s: %v", mi.MinIOClientCSRName(), err)
return err
Expand Down
45 changes: 1 addition & 44 deletions pkg/controller/cluster/main-controller.go
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,6 @@ const (
waitingKESCert = "Waiting for KES TLS Certificate"
updatingMinIOVersion = "Updating MinIO Version"
updatingMCSVersion = "Updating MCS Version"
updatingMinIOStatefulSet = "Adding New Pods to MinIO Statefulset"
notOwned = "Statefulset not controlled by operator"
)

Expand Down Expand Up @@ -472,24 +471,9 @@ func (c *Controller) syncHandler(key string) error {
if err != nil {
return err
}
// If this is a TLS enabled Setup, we create new CSR because change in number of replicas means the new endpoints
// need to be added in the CSR
if mi.AutoCert() {
klog.V(2).Infof("Removing the existing MinIO CSRs and related secrets")
if err := c.removeMinIOCSRAndSecrets(ctx, mi); err != nil {
return err
}
klog.V(2).Infof("Creating required MinIO CSRs and related secrets")
if err := c.checkAndCreateMinIOCSR(ctx, nsName, mi, mi.HasKESEnabled()); err != nil {
return err
}
}
klog.V(2).Infof("Creating a new StatefulSet %s with replicas: %d", name, mi.MinIOReplicas())
mi, err = c.updateMinIOInstanceStatus(ctx, mi, updatingMinIOStatefulSet, 0)
if err != nil {
return err
}
// Create a new statefulset object and send an update request
// Even if this is an autoTLS enabled setup, the certs are wild card certs e.g. *.
ss = statefulsets.NewForMinIO(mi, hlSvc.Name, c.hostsTemplate)
if _, err := c.kubeClientSet.AppsV1().StatefulSets(mi.Namespace).Update(ctx, ss, uOpts); err != nil {
return err
Expand Down Expand Up @@ -658,33 +642,6 @@ func (c *Controller) syncHandler(key string) error {
return nil
}

func (c *Controller) removeMinIOCSRAndSecrets(ctx context.Context, mi *miniov1.MinIOInstance) error {
if err := c.certClient.CertificateSigningRequests().Delete(ctx, mi.MinIOCSRName(), metav1.DeleteOptions{}); err != nil {
if !apierrors.IsNotFound(err) {
return err
}
}
if err := c.kubeClientSet.CoreV1().Secrets(mi.Namespace).Delete(ctx, mi.MinIOTLSSecretName(), metav1.DeleteOptions{}); err != nil {
if !apierrors.IsNotFound(err) {
return err
}
}

if mi.HasKESEnabled() {
if err := c.certClient.CertificateSigningRequests().Delete(ctx, mi.MinIOClientCSRName(), metav1.DeleteOptions{}); err != nil {
if !apierrors.IsNotFound(err) {
return err
}
}
if err := c.kubeClientSet.CoreV1().Secrets(mi.Namespace).Delete(ctx, mi.MinIOClientTLSSecretName(), metav1.DeleteOptions{}); err != nil {
if !apierrors.IsNotFound(err) {
return err
}
}
}
return nil
}

func (c *Controller) checkAndCreateMinIOCSR(ctx context.Context, nsName types.NamespacedName, mi *miniov1.MinIOInstance, createClientCert bool) error {
if _, err := c.certClient.CertificateSigningRequests().Get(ctx, mi.MinIOCSRName(), metav1.GetOptions{}); err != nil {
if apierrors.IsNotFound(err) {
Expand Down