Skip to content

Commit

Permalink
Plugin command cleanup (#296)
Browse files Browse the repository at this point in the history
Add tenant generate, tenant expand, and delete

Generate command allows production users an opportunity to
optionally add/edit any additional fields in the yaml
before the tenant actually gets created.

Also rename operator create command to
operator init, volume add and volume list to
expand and info.
  • Loading branch information
nitisht authored Oct 7, 2020
1 parent 740e04c commit e91a441
Show file tree
Hide file tree
Showing 21 changed files with 341 additions and 239 deletions.
41 changes: 12 additions & 29 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,55 +5,38 @@ MinIO Operator brings native support for [MinIO](https://github.com/minio/minio)
## Prerequisites

- Kubernetes >= v1.17.0.
- Create PVs. We recommend [direct CSI driver](https://github.com/minio/operator/blob/master/docs/using-direct-csi.md) for PV creation.
- Install [`kubectl minio` plugin](https://github.com/minio/operator/tree/master/kubectl-minio#install-plugin).
- Create PVs.
- Install [`kubectl minio` plugin using `krew install minio`.

## Operator Setup

MinIO Operator offers MinIO Tenant creation, management, upgrade, zone addition and more. Operator is meant to control and manage multiple MinIO Tenants.

To get started, create the MinIO Operator deployment. This is a _one time_ process.
To get started, initialize the MinIO Operator deployment. This is a _one time_ process.

```sh
kubectl minio operator create
kubectl minio init
```

Once the MinIO Operator is created, proceed with Tenant creation.

## Tenant Setup

A Tenant is a MinIO cluster created and managed by the Operator.
A Tenant is a MinIO cluster created and managed by the Operator. Before creating tenant, please ensure you have requisite nodes and drives in place and relevant PVs are created.

### Step 1: Create Tenant Namespace
In below example, we ask MinIO Operator to create a Tenant yaml with 4 nodes, 16 volumes, and 16 Ti total raw capacity (4 volumes of 1 Ti per node). This means you need to have 4 PVs of 1Ti each per node, with a total of 4 nodes, before attempting to create the MinIO tenant.

Before creating a Tenant, please create the namespace where this Tenant will reside.

For logical isolation, Operator allows a single Tenant per Kubernetes Namespace.

```sh
kubectl create ns tenant1-ns
```

### Step 2: Create Secret for Tenant Credentials

Next, create the Kubernetes secret that encapsulates root credentials for MinIO Tenant. Please ensure to create secret object with literals `accesskey` and `secretkey`.

Remember to change `YOUR-ACCESS-KEY` and `YOUR-SECRET-KEY` to actual values.
We recommend [direct CSI driver](https://github.com/minio/operator/blob/master/docs/using-direct-csi.md) to create PVs.

```sh
kubectl create secret generic tenant1-secret --from-literal=accesskey=YOUR-ACCESS-KEY --from-literal=secretkey=YOUR-SECRET-KEY --namespace tenant1-ns
kubectl minio tenant create --name tenant1 --secret tenant1-secret --servers 4 --volumes 16 --capacity 16Ti
```

Note that the access key and secret key provided here is authorized to perform _all_ operations on the Tenant.

### Step 3: Create MinIO Tenant

We can create the Tenant now. Before that, please ensure you have requisite nodes and drives in place and relevant PVs are created. In below example, we ask MinIO Operator to create a Tenant with 4 nodes, 16 volumes, and 16 Ti total raw capacity (4 volumes of 1 Ti per node). This means you need to have 4 PVCs of 1 Ti each, per node, and total of 4 nodes, before attempting to create the MinIO Tenant.

We recommend [direct CSI driver](https://github.com/minio/operator/blob/master/docs/using-direct-csi.md) to create PVs.
Optionally, you can generate a yaml file with the `-o` flag in above command and modify the yaml file as per your specific requirements. Once you verify and optionally add any other relevant fields to the file, create the tenant

```sh
kubectl minio tenant create --name tenant1 --secret tenant1-secret --servers 4 --volumes 16 --capacity 16Ti --namespace tenant1-ns --storage-class direct.csi.min.io
kubectl minio tenant create --name tenant1 --secret tenant1-secret --servers 4 --volumes 16 --capacity 16Ti -o > tenant.yaml
kubectl apply -f tenant.yaml
```

## Post Tenant Creation
Expand All @@ -63,7 +46,7 @@ kubectl minio tenant create --name tenant1 --secret tenant1-secret --servers 4 -
You can add capacity to the tenant using `kubectl minio` plugin, like this

```
kubectl minio tenant volume add --name tenant1 --servers 8 --volumes 32 --capacity 32Ti --namespace tenant1-ns
kubectl minio tenant expand --name tenant1 --servers 8 --volumes 32 --capacity 32Ti
```

This will add 32 drives spread uniformly over 8 servers to the tenant `tenant1`, with additional capacity of 32Ti. Read more about [tenant expansion here](https://github.com/minio/operator/blob/master/docs/expansion.md).
Expand Down
2 changes: 1 addition & 1 deletion examples/tenant-encryption.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ stringData:
# retry: 15s # Duration until the server tries to re-authenticate after connection loss.
# tls: # The Vault client TLS configuration for mTLS authentication and certificate verification
# key: "" # Path to the TLS client private key for mTLS authentication to Vault
# cert: "" # Path to the TLS client certificate for mTLS authentication to Vailt
# cert: "" # Path to the TLS client certificate for mTLS authentication to Vault
# ca: "" # Path to one or multiple PEM root CA certificates
# status: # Vault status configuration. The server will periodically reach out to Vault to check its status.
# ping: 10s # Duration until the server checks Vault's status again.
Expand Down
41 changes: 21 additions & 20 deletions kubectl-minio/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,74 +3,76 @@
## Prerequisites

- Kubernetes >= v1.17.0.
- Create PVs using [direct CSI driver](https://github.com/minio/operator/blob/master/docs/using-direct-csi.md).
- kubectl installed on your local machine, configured to talk to the Kubernetes cluster.
- Create PVs.

## Install Plugin

Download the Kubectl plugin binary from [plugin release page](https://github.com/minio/operator/releases), and place the binary in executable path on your machine (e.g. `/usr/local/bin`).
Command: `kubectl krew install minio`

## Plugin Commands

### Operator Deployment

Command: `kubectl minio operator create`
Command: `kubectl minio init [options]`

Creates MinIO Operator Deployment along with MinIO Tenant CRD, Service account, Cluster Role and Cluster Role Binding.

Options:

- `--image=minio/k8s-operator:3.0.5`
- `--namespace=minio-operator`
- `--service-account=minio-operator`
- `--cluster-domain=cluster.local`
- `--namespace-to-watch=default`
- `--image-pull-secret=`
- `--output`

### Operator Deletion

Command: `kubectl minio delete [options]`

Deletes MinIO Operator Deployment along with MinIO Tenant CRD, Service account, Cluster Role and Cluster Role Binding. It also removes all the Tenant instances.

Options:

- `--namespace=minio-operator`

### Tenant

#### MinIO Tenant Creation

Command: `kubectl minio tenant create --name TENANT_NAME --secret SECRET_NAME --servers SERVERS --volumes TOTAL_VOLUMES --capacity TOTAL_RAW_CAPACITY [options]`
Command: `kubectl minio tenant create --name TENANT_NAME --servers SERVERS --volumes TOTAL_VOLUMES --capacity TOTAL_RAW_CAPACITY [options]`

Creates a MinIO Tenant based on the passed values.

example: `kubectl minio tenant create --name tenant1 --secret cred-secret --servers 4 --volumes 16 --capacity 16Ti`
example: `kubectl minio tenant create --name tenant1 --servers 4 --volumes 16 --capacity 16Ti`

Options:

- `--namespace=minio`
- `--image=minio/minio:RELEASE.2020-09-26T03-44-56Z`
- `--storageClass=local`
- `--kms-secret=secret-name`
- `--console-secret=secret-name`
- `--cert-secret=secret-name`
- `--image-pull-secret=`
- `--kes-config=kes-secret`
- `--output`

#### Add Tenant Zones

Command: `kubectl minio tenant volume add --name TENANT_NAME --servers SERVERS --volumes TOTAL_VOLUMES --capacity TOTAL_RAW_CAPACITY [options]`
Command: `kubectl minio tenant expand --name TENANT_NAME --servers SERVERS --volumes TOTAL_VOLUMES --capacity TOTAL_RAW_CAPACITY [options]`

Add new volumes (and nodes) to existing MinIO Tenant.

example: `kubectl minio tenant volume add --name cluster1 --servers 4 --volumes 16 --capacity 16Ti`
example: `kubectl minio tenant expand --name tenant1 --servers 4 --volumes 16 --capacity 16Ti`

Options:

- `--namespace=minio`
- `--storageClass=local`
- `--image-pull-secret=`
- `--output`

#### List Tenant Zones

Command: `kubectl minio tenant volume list --name TENANT_NAME [options]`
Command: `kubectl minio tenant info --name TENANT_NAME [options]`

List all existing MinIO Zones in the given MinIO Tenant.

example: `kubectl minio tenant volume list --name cluster1`
example: `kubectl minio tenant info --name tenant1`

Options:

Expand All @@ -82,12 +84,11 @@ Command: `kubectl minio tenant upgrade --name TENANT_NAME --image IMAGE_TAG [opt

Upgrade MinIO Docker image for the given MinIO Tenant.

example: `kubectl minio tenant upgrade --name cluster1 --image minio/minio:edge`
example: `kubectl minio tenant upgrade --name tenant1 --image minio/minio:RELEASE.2020-09-26T03-44-56Z`

Options:

- `--namespace=minio`
- `--image-pull-secret=`
- `--output`

#### Remove Tenant
Expand Down
124 changes: 124 additions & 0 deletions kubectl-minio/cmd/delete.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,124 @@
/*
* This file is part of MinIO Operator
* Copyright (C) 2020, MinIO, Inc.
*
* This code is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License, version 3,
* as published by the Free Software Foundation.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License, version 3,
* along with this program. If not, see <http://www.gnu.org/licenses/>
*
*/

package cmd

import (
"bufio"
"context"
"errors"
"fmt"
"io"
"os"
"strings"

"github.com/minio/kubectl-minio/cmd/helpers"
"github.com/minio/kubectl-minio/cmd/resources"
"github.com/spf13/cobra"

v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/runtime"
)

const (
deleteDesc = `
'delete' command delete MinIO Operator deployment along with all the tenants.`
deleteExample = ` kubectl minio delete`
)

type deleteCmd struct {
out io.Writer
errOut io.Writer
output bool
operatorOpts resources.OperatorOptions
steps []runtime.Object
}

func newDeleteCmd(out io.Writer, errOut io.Writer) *cobra.Command {
o := &deleteCmd{out: out, errOut: errOut}

cmd := &cobra.Command{
Use: "delete",
Short: "Delete MinIO Operator deployment",
Long: deleteDesc,
Example: deleteExample,
PreRunE: func(cmd *cobra.Command, args []string) error {
fmt.Printf("Are you sure you want to delete ALL the MinIO Tenants and MinIO Operator? [Y/N]: ")
if !o.ask() {
return errors.New("Aborting Operator deletion")
}
return nil
},
RunE: func(cmd *cobra.Command, args []string) error {
if len(args) != 0 {
return errors.New("this command does not accept arguments")
}
return o.run()
},
}

f := cmd.Flags()
f.StringVarP(&o.operatorOpts.NS, "namespace", "n", helpers.DefaultNamespace, "namespace scope for this request")

return cmd
}

func (o *deleteCmd) ask() bool {
reader := bufio.NewReader(os.Stdin)
for {
s, _ := reader.ReadString('\n')
s = strings.TrimSuffix(s, "\n")
s = strings.ToLower(s)
if len(s) > 1 {
fmt.Fprintln(os.Stderr, "Please enter Y or N")
continue
}
if strings.Compare(s, "n") == 0 {
return false
} else if strings.Compare(s, "y") == 0 {
break
} else {
continue
}
}
return true
}

func (o *deleteCmd) run() error {
client, err := helpers.GetKubeClient()
if err != nil {
return err
}
extclient, err := helpers.GetKubeExtensionClient()
if err != nil {
return err
}
if err := extclient.ApiextensionsV1().CustomResourceDefinitions().Delete(context.Background(), crdObj.Name, v1.DeleteOptions{}); err != nil {
return err
}
if err := client.RbacV1().ClusterRoles().Delete(context.Background(), crObj.Name, v1.DeleteOptions{}); err != nil {
return err
}
if err := client.CoreV1().ServiceAccounts(o.operatorOpts.NS).Delete(context.Background(), helpers.DefaultServiceAccount, v1.DeleteOptions{}); err != nil {
return err
}
if err := client.RbacV1().ClusterRoleBindings().Delete(context.Background(), helpers.ClusterRoleBindingName, v1.DeleteOptions{}); err != nil {
return err
}
return client.AppsV1().Deployments(o.operatorOpts.NS).Delete(context.Background(), helpers.DeploymentName, v1.DeleteOptions{})
}
2 changes: 1 addition & 1 deletion kubectl-minio/cmd/helpers/helpers.go
Original file line number Diff line number Diff line change
Expand Up @@ -25,10 +25,10 @@ import (
"os/exec"

operatorv1 "github.com/minio/operator/pkg/client/clientset/versioned"
"gopkg.in/yaml.v2"
apiextension "k8s.io/apiextensions-apiserver/pkg/client/clientset/clientset"
"k8s.io/apimachinery/pkg/api/resource"
"k8s.io/apimachinery/pkg/runtime"
"sigs.k8s.io/yaml"

"github.com/pkg/errors"
"k8s.io/client-go/kubernetes"
Expand Down
27 changes: 13 additions & 14 deletions kubectl-minio/cmd/operator-create.go → kubectl-minio/cmd/init.go
Original file line number Diff line number Diff line change
Expand Up @@ -41,27 +41,27 @@ import (
)

const (
operatorCreateDesc = `
'create' command creates MinIO Operator deployment along with all the dependencies.`
operatorCreateExample = ` kubectl minio operator create`
operatorInitDesc = `
'init' command creates MinIO Operator deployment along with all the dependencies.`
operatorInitExample = ` kubectl minio operator init`
)

type operatorCreateCmd struct {
type operatorInitCmd struct {
out io.Writer
errOut io.Writer
output bool
operatorOpts resources.OperatorOptions
steps []runtime.Object
}

func newOperatorCreateCmd(out io.Writer, errOut io.Writer) *cobra.Command {
o := &operatorCreateCmd{out: out, errOut: errOut}
func newInitCmd(out io.Writer, errOut io.Writer) *cobra.Command {
o := &operatorInitCmd{out: out, errOut: errOut}

cmd := &cobra.Command{
Use: "create",
Short: "Create MinIO Operator deployment",
Long: operatorCreateDesc,
Example: operatorCreateExample,
Use: "init",
Short: "Initialize MinIO Operator deployment",
Long: operatorInitDesc,
Example: operatorInitExample,
RunE: func(cmd *cobra.Command, args []string) error {
if len(args) != 0 {
return errors.New("this command does not accept arguments")
Expand All @@ -74,7 +74,6 @@ func newOperatorCreateCmd(out io.Writer, errOut io.Writer) *cobra.Command {
f.StringVarP(&o.operatorOpts.Image, "image", "i", helpers.DefaultOperatorImage, "operator image")
f.StringVarP(&o.operatorOpts.NS, "namespace", "n", helpers.DefaultNamespace, "namespace scope for this request")
f.StringVarP(&o.operatorOpts.ClusterDomain, "cluster-domain", "d", helpers.DefaultClusterDomain, "cluster domain of the Kubernetes cluster")
f.StringVar(&o.operatorOpts.ServiceAccount, "service-account", helpers.DefaultServiceAccount, "service account for operator")
f.StringVar(&o.operatorOpts.NSToWatch, "namespace-to-watch", "", "namespace where operator looks for MinIO tenants, leave empty for all namespaces")
f.StringVar(&o.operatorOpts.ImagePullSecret, "image-pull-secret", "", "image pull secret to be used for pulling operator image")
f.BoolVarP(&o.output, "output", "o", false, "dry run this command and generate requisite yaml")
Expand All @@ -83,9 +82,9 @@ func newOperatorCreateCmd(out io.Writer, errOut io.Writer) *cobra.Command {
}

// run initializes local config and installs MinIO Operator to Kubernetes cluster.
func (o *operatorCreateCmd) run() error {
sa := resources.NewServiceAccountForOperator(o.operatorOpts.ServiceAccount, o.operatorOpts.NS)
crb := resources.NewCluterRoleBindingForOperator(o.operatorOpts.NS, o.operatorOpts.ServiceAccount)
func (o *operatorInitCmd) run() error {
sa := resources.NewServiceAccountForOperator(helpers.DefaultServiceAccount, o.operatorOpts.NS)
crb := resources.NewCluterRoleBindingForOperator(helpers.DefaultServiceAccount, o.operatorOpts.NS)
d := resources.NewDeploymentForOperator(o.operatorOpts)

if !o.output {
Expand Down
3 changes: 2 additions & 1 deletion kubectl-minio/cmd/kubectl-minio.go
Original file line number Diff line number Diff line change
Expand Up @@ -101,7 +101,8 @@ func NewCmdMinIO(streams genericclioptions.IOStreams) *cobra.Command {
SilenceUsage: true,
}

cmd.AddCommand(newOperatorCmd(cmd.OutOrStdout(), cmd.ErrOrStderr()))
cmd.AddCommand(newInitCmd(cmd.OutOrStdout(), cmd.ErrOrStderr()))
cmd.AddCommand(newDeleteCmd(cmd.OutOrStdout(), cmd.ErrOrStderr()))
cmd.AddCommand(newTenantCmd(cmd.OutOrStdout(), cmd.ErrOrStderr()))

return cmd
Expand Down
Loading

0 comments on commit e91a441

Please sign in to comment.