Skip to content

Commit

Permalink
Rename Zones to Pools (#371)
Browse files Browse the repository at this point in the history
  • Loading branch information
nitisht authored Nov 27, 2020
1 parent 07159c5 commit 7538e5d
Show file tree
Hide file tree
Showing 56 changed files with 1,848 additions and 1,225 deletions.
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ MinIO Operator brings native support for [MinIO](https://github.com/minio/minio)

## Operator Setup

MinIO Operator offers MinIO Tenant creation, management, upgrade, zone addition and more. Operator is meant to control and manage multiple MinIO Tenants.
MinIO Operator offers MinIO Tenant creation, management, upgrade, pool addition and more. Operator is meant to control and manage multiple MinIO Tenants.

To get started, initialize the MinIO Operator deployment. This is a _one time_ process.

Expand Down
4 changes: 2 additions & 2 deletions docs/custom-name-templates.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ This document explains how to control the names used for host discovery. This al

## Getting Started

If MinIO Tenant is named `tenant1`, then the four servers will be called `tenant1-zone-0-0`, `tenant1-zone-0-1`, `tenant1-zone-0-2`, and `tenant1-zone-0-3`. If all of your hosts are available at the domain `example.com` then you can use the `--hosts-template` flag in [MinIO Operator Deployment yaml](https://github.com/minio/operator/blob/master/minio-operator.yaml) to update discovery. This will generate the discovery string `tenant1-zone-0-{0...3}.example.com`.
If MinIO Tenant is named `tenant1`, then the four servers will be called `tenant1-pool-0-0`, `tenant1-pool-0-1`, `tenant1-pool-0-2`, and `tenant1-pool-0-3`. If all of your hosts are available at the domain `example.com` then you can use the `--hosts-template` flag in [MinIO Operator Deployment yaml](https://github.com/minio/operator/blob/master/minio-operator.yaml) to update discovery. This will generate the discovery string `tenant1-pool-0-{0...3}.example.com`.

```yaml
containers:
Expand All @@ -21,5 +21,5 @@ The following fields can be configured:
| StatefulSet | The name of the tenant StatefulSet (e.g. `minio`). |
| CIService | The name of the service provided in `spec.serviceName`. |
| HLService | The name of the headless service that is generated (e.g. `minio-hl-service`) |
| Ellipsis | `{0...N-1}` the per-zone host numbers. |
| Ellipsis | `{0...N-1}` the per-pool host numbers. |
| Domain | The cluster domain, either `cluster.local` or the contents of the `CLUSTER_DOMAIN` environment variable. |
12 changes: 6 additions & 6 deletions docs/expansion.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

This document explains how to expand an existing MinIO Tenant with Operator. This is only applicable to a Tenant (MinIO Deployment) created by MinIO Operator.

MinIO expansion is done in terms of MinIO zones, read more about the design in [MinIO Docs](https://github.com/minio/minio/blob/master/docs/distributed).
MinIO expansion is done in terms of MinIO pools, read more about the design in [MinIO Docs](https://github.com/minio/minio/blob/master/docs/distributed).

## Getting Started

Expand All @@ -22,19 +22,19 @@ Remember to replace `TENANT_NAME` with tenant name where you want to add volumes

## Underlying Details in Tenant Expansion

### What are MinIO Zones
### What are MinIO pools

A MinIO zone is a self contained entity with same SLA's (read/write quorum) for each object. There are no limits on how many zones can be combined. After adding of a zone, MinIO simply uses the least used zone. All zones are for all purposes are invisible to an any application, and MinIO handles the zones internally.
A MinIO pool is a self contained entity with same SLA's (read/write quorum) for each object. There are no limits on how many pools can be combined. After adding of a pool, MinIO simply uses the least used pool. All pools are for all purposes are invisible to an any application, and MinIO handles the pools internally.

### Rules of Adding Zones
### Rules of Adding pools

There is only one requirement, i.e. based on initial zone's erasure set count (say `n`), new zones are expected to have a minimum of `n` drives to match the original Tenant SLA or it should be in multiples of `n`. For example if initial set count is 4, new zones should have at least 4 or multiple of 4 drives.
There is only one requirement, i.e. based on initial pool's erasure set count (say `n`), new pools are expected to have a minimum of `n` drives to match the original Tenant SLA or it should be in multiples of `n`. For example if initial set count is 4, new pools should have at least 4 or multiple of 4 drives.

### Effects on KES/TLS Enabled Instance

If your MinIO Operator configuration has [KES](https://github.com/minio/operator/blob/master/docs/kes.md) or [Automatic TLS](https://github.com/minio/operator/blob/master/docs/tls.md#automatic-csr-generation) enabled, there are additional considerations:

- When new zones are added, Operator invalidates older self signed TLS certificates and the related secrets. Operator then creates new certificate signing requests (CSR). This is because there are new MinIO nodes that must be added in certificate DNS names. The administrator must approve these CSRs for MinIO server to be deployed again. Unless the CSR are approved, Operator will not create MinIO StatefulSet pods.
- When new pools are added, Operator invalidates older self signed TLS certificates and the related secrets. Operator then creates new certificate signing requests (CSR). This is because there are new MinIO nodes that must be added in certificate DNS names. The administrator must approve these CSRs for MinIO server to be deployed again. Unless the CSR are approved, Operator will not create MinIO StatefulSet pods.

- If you're using your own certificates, as explained [here](https://github.com/minio/operator/blob/master/docs/tls.md#pass-certificate-secret-to-tenant), please ensure to use/update proper certificates that allow older and new MinIO nodes.

Expand Down
4 changes: 2 additions & 2 deletions docs/operator-fields.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,8 @@ MinIO Operator creates native Kubernetes resources within the cluster. If the Te
| scheduler | Set custom scheduler for pods created by MinIO Operator.|
| spec.metadata | Define the object metadata to be passed to all the members pods of this Tenant. This allows adding annotations and labels. For example,you can add Prometheus annotations here. Internally `metadata` is a struct type as [explained here](https://godoc.org/k8s.io/apimachinery/pkg/apis/meta/v1#ObjectMeta).|
| spec.image | Set the container registry and image tag for MinIO server to be used in the Tenant.|
| spec.zones | Set the number of servers per MinIO Zone. Add a new Zone field to expand the MinIO cluster. Read more on [MinIO zones here](https://github.com/minio/minio/blob/master/docs/distributed/DESIGN.md).|
| spec.volumesPerServer | Set the number of volume mounts per MinIO node. For example if you set `spec.zones[0].Servers = 4`, `spec.zones[1].Servers = 8` and `spec.volumesPerServer = 4`, then you'll have total 12 MinIO Pods, with 4 volume mounts on each Pod. Note that `volumesPerServer` is static per cluster, expanding a cluster will add new nodes. |
| spec.pools | Set the number of servers per MinIO Pool. Add a new Pool field to expand the MinIO cluster. Read more on [MinIO pools here](https://github.com/minio/minio/blob/master/docs/distributed/DESIGN.md).|
| spec.volumesPerServer | Set the number of volume mounts per MinIO node. For example if you set `spec.pools[0].Servers = 4`, `spec.pools[1].Servers = 8` and `spec.volumesPerServer = 4`, then you'll have total 12 MinIO Pods, with 4 volume mounts on each Pod. Note that `volumesPerServer` is static per cluster, expanding a cluster will add new nodes. |
| spec.imagePullSecret | Defines the secret to be used for pull image from a private Docker image. |
| spec.credsSecret | Use this secret to assign custom credentials (access key and secret key) to Tenant.|
| spec.replicas | Define the number of nodes to be created for current Tenant cluster.|
Expand Down
6 changes: 3 additions & 3 deletions examples/patch.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,10 @@ kind: Tenant
metadata:
name: minio
spec:
zones:
pools:
- volumesPerServer: 4
## Number of MinIO servers/pods in zone 0.
## Number of MinIO servers/pods in pool 0.
servers: 4
- volumesPerServer: 4
## Number of MinIO servers/pods in zone 1.
## Number of MinIO servers/pods in pool 1.
servers: 4
8 changes: 4 additions & 4 deletions examples/tenant-encryption.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -116,9 +116,9 @@ spec:
credsSecret:
name: minio-creds-secret

## Specification for MinIO Zone(s) in this Tenant.
zones:
## Servers specifies the number of MinIO Tenant Pods / Servers in this zone.
## Specification for MinIO Pool(s) in this Tenant.
pools:
## Servers specifies the number of MinIO Tenant Pods / Servers in this pool.
## For standalone mode, supply 1. For distributed mode, supply 4 or more.
## Note that the operator does not support upgrading from standalone to distributed mode.
- servers: 4
Expand All @@ -127,7 +127,7 @@ spec:
volumesPerServer: 4

## This VolumeClaimTemplate is used across all the volumes provisioned for MinIO Tenant in this
## Zone.
## Pool.
volumeClaimTemplate:
metadata:
name: data
Expand Down
4 changes: 2 additions & 2 deletions examples/tenant-pod-security-policy.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -79,9 +79,9 @@ spec:
image: minio/minio:RELEASE.2020-11-19T23-48-16Z
## Service account to be used for all the MinIO Pods
serviceAccountName: minio-pods
zones:
pools:
- volumesPerServer: 4
## Number of MinIO servers/pods in this zone.
## Number of MinIO servers/pods in this pool.
## For standalone mode, supply 1. For distributed mode, supply 4 or more.
## Note that the operator does not support upgrading from standalone to distributed mode.
servers: 4
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -109,9 +109,9 @@ spec:
credsSecret:
name: minio-autocert-custom-cert-encryption-minio-creds-secret

## Specification for MinIO Zone(s) in this Tenant.
zones:
## Servers specifies the number of MinIO Tenant Pods / Servers in this zone.
## Specification for MinIO Pool(s) in this Tenant.
pools:
## Servers specifies the number of MinIO Tenant Pods / Servers in this pool.
## For standalone mode, supply 1. For distributed mode, supply 4 or more.
## Note that the operator does not support upgrading from standalone to distributed mode.
- servers: 4
Expand All @@ -120,7 +120,7 @@ spec:
volumesPerServer: 4

## This VolumeClaimTemplate is used across all the volumes provisioned for MinIO Tenant in this
## Zone.
## Pool.
volumeClaimTemplate:
metadata:
name: data
Expand Down
4 changes: 2 additions & 2 deletions examples/tenant-with-autocert-and-ldap.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -155,8 +155,8 @@ spec:
credsSecret:
name: minio-autocert-ldap-minio-creds-secret

## Specification for MinIO Zone(s) in this Tenant.
zones:
## Specification for MinIO Pool(s) in this Tenant.
pools:
- servers: 4
volumesPerServer: 4
volumeClaimTemplate:
Expand Down
4 changes: 2 additions & 2 deletions examples/tenant-with-autocert-encryption-disabled.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -55,8 +55,8 @@ spec:
credsSecret:
name: minio-autocert-no-encryption-minio-creds-secret

## Specification for MinIO Zone(s) in this Tenant.
zones:
## Specification for MinIO Pool(s) in this Tenant.
pools:
- servers: 4
volumesPerServer: 4
volumeClaimTemplate:
Expand Down
4 changes: 2 additions & 2 deletions examples/tenant-with-autocert-encryption-enabled.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -98,8 +98,8 @@ spec:
credsSecret:
name: minio-autocert-encryption-minio-creds-secret

## Specification for MinIO Zone(s) in this Tenant.
zones:
## Specification for MinIO Pool(s) in this Tenant.
pools:
- servers: 4
volumesPerServer: 4
volumeClaimTemplate:
Expand Down
8 changes: 4 additions & 4 deletions examples/tenant-with-custom-ca-certs.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -94,9 +94,9 @@ spec:
credsSecret:
name: minio-custom-cert-no-encryption-minio-creds-secret

## Specification for MinIO Zone(s) in this Tenant.
zones:
## Servers specifies the number of MinIO Tenant Pods / Servers in this zone.
## Specification for MinIO Pool(s) in this Tenant.
pools:
## Servers specifies the number of MinIO Tenant Pods / Servers in this pool.
## For standalone mode, supply 1. For distributed mode, supply 4 or more.
## Note that the operator does not support upgrading from standalone to distributed mode.
- servers: 4
Expand All @@ -105,7 +105,7 @@ spec:
volumesPerServer: 4

## This VolumeClaimTemplate is used across all the volumes provisioned for MinIO Tenant in this
## Zone.
## Pool.
volumeClaimTemplate:
metadata:
name: data
Expand Down
8 changes: 4 additions & 4 deletions examples/tenant-with-custom-cert-encryption-disabled.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -86,9 +86,9 @@ spec:
credsSecret:
name: minio-custom-cert-no-encryption-minio-creds-secret

## Specification for MinIO Zone(s) in this Tenant.
zones:
## Servers specifies the number of MinIO Tenant Pods / Servers in this zone.
## Specification for MinIO Pool(s) in this Tenant.
pools:
## Servers specifies the number of MinIO Tenant Pods / Servers in this pool.
## For standalone mode, supply 1. For distributed mode, supply 4 or more.
## Note that the operator does not support upgrading from standalone to distributed mode.
- servers: 4
Expand All @@ -97,7 +97,7 @@ spec:
volumesPerServer: 4

## This VolumeClaimTemplate is used across all the volumes provisioned for MinIO Tenant in this
## Zone.
## Pool.
volumeClaimTemplate:
metadata:
name: data
Expand Down
8 changes: 4 additions & 4 deletions examples/tenant-with-custom-cert-encryption-enabled.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -149,9 +149,9 @@ spec:
credsSecret:
name: minio-custom-cert-encryption-minio-creds-secret

## Specification for MinIO Zone(s) in this Tenant.
zones:
## Servers specifies the number of MinIO Tenant Pods / Servers in this zone.
## Specification for MinIO Pool(s) in this Tenant.
pools:
## Servers specifies the number of MinIO Tenant Pods / Servers in this pool.
## For standalone mode, supply 1. For distributed mode, supply 4 or more.
## Note that the operator does not support upgrading from standalone to distributed mode.
- servers: 4
Expand All @@ -160,7 +160,7 @@ spec:
volumesPerServer: 4

## This VolumeClaimTemplate is used across all the volumes provisioned for MinIO Tenant in this
## Zone.
## Pool.
volumeClaimTemplate:
metadata:
name: data
Expand Down
8 changes: 4 additions & 4 deletions examples/tenant.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -55,9 +55,9 @@ spec:
credsSecret:
name: minio-creds-secret

## Specification for MinIO Zone(s) in this Tenant.
zones:
## Servers specifies the number of MinIO Tenant Pods / Servers in this zone.
## Specification for MinIO Pool(s) in this Tenant.
pools:
## Servers specifies the number of MinIO Tenant Pods / Servers in this pool.
## For standalone mode, supply 1. For distributed mode, supply 4 or more.
## Note that the operator does not support upgrading from standalone to distributed mode.
- servers: 4
Expand All @@ -66,7 +66,7 @@ spec:
volumesPerServer: 4

## This VolumeClaimTemplate is used across all the volumes provisioned for MinIO Tenant in this
## Zone.
## Pool.
volumeClaimTemplate:
metadata:
name: data
Expand Down
6 changes: 3 additions & 3 deletions kubectl-minio/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ Options:
- `--kes-config=kes-secret`
- `--output`

#### Add Tenant Zones
#### Add Tenant pools

Command: `kubectl minio tenant expand --name TENANT_NAME --servers SERVERS --volumes TOTAL_VOLUMES --capacity TOTAL_RAW_CAPACITY [options]`

Expand All @@ -66,11 +66,11 @@ Options:
- `--namespace=minio`
- `--output`

#### List Tenant Zones
#### List Tenant pools

Command: `kubectl minio tenant info --name TENANT_NAME [options]`

List all existing MinIO Zones in the given MinIO Tenant.
List all existing MinIO pools in the given MinIO Tenant.

example: `kubectl minio tenant info --name tenant1`

Expand Down
6 changes: 3 additions & 3 deletions kubectl-minio/cmd/resources/common.go
Original file line number Diff line number Diff line change
Expand Up @@ -32,9 +32,9 @@ func tenantStorage(q resource.Quantity) v1.ResourceList {
return m
}

// Zone returns a Zone object from given values
func Zone(servers, volumes int32, q resource.Quantity, sc string) miniov1.Zone {
return miniov1.Zone{
// Pool returns a Pool object from given values
func Pool(servers, volumes int32, q resource.Quantity, sc string) miniov1.Pool {
return miniov1.Pool{
Servers: servers,
VolumesPerServer: volumes,
VolumeClaimTemplate: &v1.PersistentVolumeClaim{
Expand Down
15 changes: 8 additions & 7 deletions kubectl-minio/cmd/resources/tenant.go
Original file line number Diff line number Diff line change
Expand Up @@ -132,13 +132,14 @@ func tenantConsoleConfig(tenant, secret string) *miniov1.ConsoleConfiguration {
return nil
}

func externalCertSecret(secret string) *miniov1.LocalCertificateReference {
func externalCertSecret(secret string) []*miniov1.LocalCertificateReference {
certs := make([]*miniov1.LocalCertificateReference, 1)
if secret != "" {
return &miniov1.LocalCertificateReference{
certs[0] = &miniov1.LocalCertificateReference{
Name: secret,
}
}
return nil
return certs
}

func storageClass(sc string) *string {
Expand All @@ -150,6 +151,7 @@ func storageClass(sc string) *string {

// NewTenant will return a new Tenant for a MinIO Operator
func NewTenant(opts *TenantOptions) (*miniov1.Tenant, error) {
autoCert := true
volumesPerServer := helpers.VolumesPerServer(opts.Volumes, opts.Servers)
capacityPerVolume, err := helpers.CapacityPerVolume(opts.Capacity, opts.Volumes)
if err != nil {
Expand All @@ -166,13 +168,12 @@ func NewTenant(opts *TenantOptions) (*miniov1.Tenant, error) {
Namespace: opts.NS,
},
Spec: miniov1.TenantSpec{
Image: opts.Image,
ServiceName: helpers.ServiceName(opts.Name),
Image: opts.Image,
CredsSecret: &v1.LocalObjectReference{
Name: opts.SecretName,
},
Zones: []miniov1.Zone{Zone(opts.Servers, volumesPerServer, *capacityPerVolume, opts.StorageClass)},
RequestAutoCert: true,
Pools: []miniov1.Pool{Pool(opts.Servers, volumesPerServer, *capacityPerVolume, opts.StorageClass)},
RequestAutoCert: &autoCert,
CertConfig: &miniov1.CertificateConfig{
CommonName: "",
OrganizationName: []string{},
Expand Down
2 changes: 1 addition & 1 deletion kubectl-minio/cmd/tenant-create.go
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ func newTenantCreateCmd(out io.Writer, errOut io.Writer) *cobra.Command {
f.StringVar(&c.tenantOpts.Name, "name", "", "name of the MinIO tenant to create")
f.Int32Var(&c.tenantOpts.Servers, "servers", 0, "total number of pods in MinIO tenant")
f.Int32Var(&c.tenantOpts.Volumes, "volumes", 0, "total number of volumes in the MinIO tenant")
f.StringVar(&c.tenantOpts.Capacity, "capacity", "", "total raw capacity of MinIO tenant in this zone, e.g. 16Ti")
f.StringVar(&c.tenantOpts.Capacity, "capacity", "", "total raw capacity of MinIO tenant in this pool, e.g. 16Ti")
f.StringVarP(&c.tenantOpts.NS, "namespace", "n", helpers.DefaultNamespace, "namespace scope for this request")
f.StringVarP(&c.tenantOpts.StorageClass, "storage-class", "s", helpers.DefaultStorageclass, "storage class for this MinIO tenant")
f.StringVar(&c.tenantOpts.KmsSecret, "kes-config", "", "name of secret with details for enabling encryption, refer example https://github.com/minio/operator/blob/master/examples/kes-secret.yaml")
Expand Down
6 changes: 3 additions & 3 deletions kubectl-minio/cmd/tenant-expand.go
Original file line number Diff line number Diff line change
Expand Up @@ -98,10 +98,10 @@ func (v *expandCmd) run() error {
return err
}

t.Spec.Zones = append(t.Spec.Zones, resources.Zone(v.tenantOpts.Servers, volumesPerServer, *capacityPerVolume, v.tenantOpts.StorageClass))
t.Spec.Pools = append(t.Spec.Pools, resources.Pool(v.tenantOpts.Servers, volumesPerServer, *capacityPerVolume, v.tenantOpts.StorageClass))

if !v.output {
return addZoneToTenant(client, t)
return addPoolToTenant(client, t)
}

o, err := yaml.Marshal(t)
Expand All @@ -112,7 +112,7 @@ func (v *expandCmd) run() error {
return nil
}

func addZoneToTenant(client *operatorv1.Clientset, t *miniov1.Tenant) error {
func addPoolToTenant(client *operatorv1.Clientset, t *miniov1.Tenant) error {
data, err := json.Marshal(t)
if err != nil {
return err
Expand Down
Loading

0 comments on commit 7538e5d

Please sign in to comment.