Skip to content

Commit

Permalink
Support drive anti-affinity for volumes
Browse files Browse the repository at this point in the history
Some optimal setups would require volumes to be allocated on unique disks.
ie, not more than one volume per disk.

By default, the volume scheduling algorithm choses the drive based on most
free capacity. This will end up allocating more than one volumes per disk.

This PR provides a way for such optimal setups, by using the storage class
parameters. Using a storage class with `directpv.min.io/unique-alloc-id: XXX`
parameter enables unique allocation for PVCs. This unique allocation id enables
the one-to-one cardinality for the drives and volumes.
  • Loading branch information
Praveenrajmani committed Sep 13, 2023
1 parent c26a1c1 commit 5cb3558
Show file tree
Hide file tree
Showing 9 changed files with 241 additions and 41 deletions.
16 changes: 10 additions & 6 deletions docs/tools/create-storage-class.sh
Original file line number Diff line number Diff line change
Expand Up @@ -18,27 +18,31 @@

set -e -C -o pipefail

declare NAME DRIVE_LABEL
declare NAME
declare DRIVE_LABELS=()

function init() {
if [[ $# -ne 2 ]]; then
if [[ $# -lt 2 ]]; then
cat <<EOF
USAGE:
create-storage-class.sh <NAME> <DRIVE-LABEL>
create-storage-class.sh <NAME> <DRIVE-LABELS> ...
ARGUMENTS:
NAME new storage class name.
DRIVE-LABEL drive labels to be attached.
DRIVE-LABELS drive labels to be attached.
EXAMPLE:
# Create new storage class 'fast-tier-storage' with drive labels 'directpv.min.io/tier: fast'
$ create-storage-class.sh fast-tier-storage 'directpv.min.io/tier: fast'
# Create new storage class with more than one drive label
$ create-storage-class.sh fast-tier-unique 'directpv.min.io/tier: fast' 'directpv.min.io/volume-claim-id: bcea279a-df70-4d23-be41-9490f9933004'
EOF
exit 255
fi

NAME="$1"
DRIVE_LABEL="$2"
DRIVE_LABELS=( "${@: 2}" )

if ! which kubectl >/dev/null 2>&1; then
echo "kubectl not found; please install"
Expand Down Expand Up @@ -67,7 +71,7 @@ metadata:
name: ${NAME}
parameters:
fstype: xfs
${DRIVE_LABEL}
$(printf "%s\n " "${DRIVE_LABELS[@]}")
provisioner: directpv-min-io
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
Expand Down
116 changes: 101 additions & 15 deletions docs/volume-scheduling.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ DirectPV CSI controller selects suitable drive for `CreateVolume` request like b
a. By requested capacity
b. By access-tier if requested
c. By topology constraints if requested
d. By volume claim ID if requested
4. In the process of step (3), if more than one drive is selected, the maximum free capacity drive is picked.
5. If step (4) picks up more than one drive, a drive is randomly selected.
6. Finally the selected drive is updated with requested volume information.
Expand Down Expand Up @@ -42,23 +43,26 @@ DirectPV CSI controller selects suitable drive for `CreateVolume` request like b
│ drive │ │ matched? │ | No │ Match by │
└─────^─────┘ ╰╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╯ |<----│ access-tier │
| | Yes | │ if requested? │
| ┌───────V───────┐ | ╰╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╯
| ┌───────V───────┐ | ╰╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╯
| │ Filter drives │ | | Yes
| │ by maximum │ | ╭╌╌╌╌╌╌╌V╌╌╌╌╌╌╌╮
| │ free capacity │ | Match by
| └───────────────┘ | No │ topology
| | |<----│ constraints
| ╭╌╌╌╌╌╌╌V╌╌╌╌╌╌╌╮ | if requested?
| No │ Is more than │ | ╰╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╯
| │ by maximum │ | ╭╌╌╌╌╌╌╌V╌╌╌╌╌╌╌
| │ free capacity │ | No | Match by |
| └───────────────┘ |<----| topology |
| | | | constraints |
| ╭╌╌╌╌╌╌╌V╌╌╌╌╌╌╌╮ | | if requested? |
| No │ Is more than │ | ╰╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌
+-----------│ one drive │ | | Yes
│ matched? │ | ┌───────V───────┐
╰╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╯ | │ Append to │
| Yes +<----│ matched drives│
┌───────V───────┐ └───────────────┘
│ Return │
│ Randomly │
│ selected drive│
└───────────────┘
| matched? │ | ╭╌╌╌╌╌╌╌V╌╌╌╌╌╌╌╮
╰╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╯ | Yes │ Match by │
| Yes |<----│ volume │
┌───────V───────┐ | | claim ID │
│ Return │ │ | if requested? │
│ Randomly │ | ╰╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╯
│ selected drive│ | | No
└───────────────┘ | ┌───────V───────┐
| │ Append to │
+<----│ matched drives│
└───────────────┘
```

## Customizing drive selection
Expand Down Expand Up @@ -92,3 +96,85 @@ spec:
storage: 8Mi
EOF
```

### Unique drive selection

The default free capacity based drive selection leads to allocate more than one volume in a single drive for StatefulSet deployments which lacks performance and high availability for application like MinIO object storage. To overcome this behavior, DirectPV provides a way to allocate one volume per drive. This feature needs to be set by having custom storage class with label 'directpv.min.io/volume-claim-id'. Below is an example to create custom storage class using [create-storage-class.sh script](../tools/create-storage-class.sh):

```sh
# NOTE: The claim ID should be a valid UUID based on [RFC 4122](https://www.rfc-editor.org/rfc/rfc4122.html)
create-storage-class.sh tenant-1-storage 'directpv.min.io/volume-claim-id: 555e99eb-e255-4407-83e3-fc443bf20f86'
```

This custom storage class has to be used in your StatefulSet deployment. Below is an example to deploy MinIO object storage

```yaml
kind: Service
apiVersion: v1
metadata:
name: minio
labels:
app: minio
spec:
selector:
app: minio
ports:
- name: minio
port: 9000

---

apiVersion: apps/v1
kind: StatefulSet
metadata:
name: minio
labels:
app: minio
spec:
serviceName: "minio"
replicas: 2
selector:
matchLabels:
app: minio
template:
metadata:
labels:
app: minio
directpv.min.io/organization: minio
directpv.min.io/app: minio-example
directpv.min.io/tenant: tenant-1
spec:
containers:
- name: minio
image: minio/minio
env:
- name: MINIO_ACCESS_KEY
value: minio
- name: MINIO_SECRET_KEY
value: minio123
volumeMounts:
- name: minio-data-1
mountPath: /data1
- name: minio-data-2
mountPath: /data2
args:
- "server"
- "http://minio-{0...1}.minio.default.svc.cluster.local:9000/data{1...2}"
volumeClaimTemplates:
- metadata:
name: minio-data-1
spec:
storageClassName: tenant-1-storage
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 16Mi
- metadata:
name: minio-data-2
spec:
storageClassName: tenant-1-storage
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 16Mi
```
9 changes: 9 additions & 0 deletions pkg/apis/directpv.min.io/types/label.go
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,15 @@ const (

// SuspendLabelKey denotes if the volume is suspended.
SuspendLabelKey LabelKey = consts.GroupName + "/suspend"

// VolumeClaimIDLabelKey label key to denote the unique allocation of drives for volumes
VolumeClaimIDLabelKey LabelKey = consts.GroupName + "/volume-claim-id"

// VolumeClaimIDLabelKeyPrefix label key prefix for volume claim id to be set on the drive
VolumeClaimIDLabelKeyPrefix = consts.GroupName + "/volume-claim-id-"

// ClaimIDLabelKey label key to denote the claim id of the volumes
ClaimIDLabelKey LabelKey = consts.GroupName + "/claim-id"
)

// LabelValue is a type definition for label value
Expand Down
25 changes: 25 additions & 0 deletions pkg/apis/directpv.min.io/v1beta1/drive.go
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@
package v1beta1

import (
"strconv"
"strings"

"github.com/minio/directpv/pkg/apis/directpv.min.io/types"
Expand Down Expand Up @@ -221,6 +222,30 @@ func (drive DirectPVDrive) GetNodeID() types.NodeID {
return types.NodeID(drive.getLabel(types.NodeLabelKey))
}

// HasVolumeClaimID checks if the provided volume claim id is set on the drive.
func (drive *DirectPVDrive) HasVolumeClaimID(claimID string) bool {
if claimID == "" {
return false
}
return drive.GetLabels()[types.VolumeClaimIDLabelKeyPrefix+claimID] == strconv.FormatBool(true)
}

// SetVolumeClaimID sets the provided claim id on the drive.
func (drive *DirectPVDrive) SetVolumeClaimID(claimID string) {
if claimID == "" {
return
}
drive.SetLabel(types.LabelKey(types.VolumeClaimIDLabelKeyPrefix+claimID), types.LabelValue(strconv.FormatBool(true)))
}

// RemoveVolumeClaimID removes the volume claim id label.
func (drive *DirectPVDrive) RemoveVolumeClaimID(claimID string) {
if claimID == "" {
return
}
drive.RemoveLabel(types.LabelKey(types.VolumeClaimIDLabelKeyPrefix + claimID))
}

// SetLabel sets label to this drive.
func (drive *DirectPVDrive) SetLabel(key types.LabelKey, value types.LabelValue) bool {
values := drive.GetLabels()
Expand Down
13 changes: 13 additions & 0 deletions pkg/apis/directpv.min.io/v1beta1/volume.go
Original file line number Diff line number Diff line change
Expand Up @@ -293,6 +293,19 @@ func (volume DirectPVVolume) IsSuspended() bool {
return string(volume.getLabel(types.SuspendLabelKey)) == strconv.FormatBool(true)
}

// SetClaimID sets the provided claim id on the volume.
func (volume *DirectPVVolume) SetClaimID(claimID string) {
if claimID == "" {
return
}
volume.SetLabel(types.ClaimIDLabelKey, types.LabelValue(claimID))
}

// GetClaimID gets the claim id set on the volume.
func (volume *DirectPVVolume) GetClaimID() string {
return string(volume.getLabel(types.ClaimIDLabelKey))
}

// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object

// DirectPVVolumeList denotes list of volumes.
Expand Down
11 changes: 10 additions & 1 deletion pkg/csi/controller/server.go
Original file line number Diff line number Diff line change
Expand Up @@ -145,11 +145,18 @@ func (c *Server) CreateVolume(ctx context.Context, req *csi.CreateVolumeRequest)
return nil, status.Errorf(codes.InvalidArgument, "unsupported filesystem type %v for volume %v", req.GetVolumeCapabilities()[0].GetMount().GetFsType(), name)
}

var volumeClaimID string
for key, value := range req.GetParameters() {
if key == string(directpvtypes.AccessTierLabelKey) {
switch key {
case string(directpvtypes.AccessTierLabelKey):
if _, err := directpvtypes.StringsToAccessTiers(value); err != nil {
return nil, status.Errorf(codes.InvalidArgument, "unknown access-tier %v for volume %v; %v", value, name, err)
}
case string(directpvtypes.VolumeClaimIDLabelKey):
if err := validateVolumeClaimID(value); err != nil {
return nil, status.Errorf(codes.InvalidArgument, "invalid %v value; %v", directpvtypes.VolumeClaimIDLabelKey, err)
}
volumeClaimID = value
}
}

Expand Down Expand Up @@ -177,6 +184,7 @@ func (c *Server) CreateVolume(ctx context.Context, req *csi.CreateVolumeRequest)
drive.GetDriveName(),
size,
)
newVolume.SetClaimID(volumeClaimID)

if _, err := client.VolumeClient().Create(ctx, newVolume, metav1.CreateOptions{}); err != nil {
if !errors.IsAlreadyExists(err) {
Expand Down Expand Up @@ -206,6 +214,7 @@ func (c *Server) CreateVolume(ctx context.Context, req *csi.CreateVolumeRequest)
}

if drive.AddVolumeFinalizer(req.GetName()) {
drive.SetVolumeClaimID(volumeClaimID)
drive.Status.FreeCapacity -= size
drive.Status.AllocatedCapacity += size

Expand Down
18 changes: 18 additions & 0 deletions pkg/csi/controller/utils.go
Original file line number Diff line number Diff line change
Expand Up @@ -19,10 +19,12 @@ package controller
import (
"context"
"crypto/rand"
"fmt"
"math/big"
"strings"

"github.com/container-storage-interface/spec/lib/go/csi"
"github.com/google/uuid"
directpvtypes "github.com/minio/directpv/pkg/apis/directpv.min.io/types"
"github.com/minio/directpv/pkg/consts"
"github.com/minio/directpv/pkg/drive"
Expand Down Expand Up @@ -65,6 +67,11 @@ func matchDrive(drive *types.Drive, req *csi.CreateVolumeRequest) bool {
if len(accessTiers) > 0 && drive.GetAccessTier() != accessTiers[0] {
return false
}
case string(directpvtypes.VolumeClaimIDLabelKey):
if drive.HasVolumeClaimID(value) {
// Do not allocate another volume with this claim id
return false
}
default:
if labels[key] != value {
return false
Expand Down Expand Up @@ -159,3 +166,14 @@ func selectDrive(ctx context.Context, req *csi.CreateVolumeRequest) (*types.Driv

return &maxFreeCapacityDrives[n.Int64()], nil
}

func validateVolumeClaimID(claimID string) error {
if _, err := uuid.Parse(claimID); err != nil {
return fmt.Errorf("claimID is not an UUID; %v", err)
}
_, err := directpvtypes.NewLabelKey(directpvtypes.VolumeClaimIDLabelKeyPrefix + claimID)
if err == nil {
_, err = directpvtypes.NewLabelValue(claimID)
}
return err
}
Loading

0 comments on commit 5cb3558

Please sign in to comment.