From 44f2b14cf1d9425368483fcb1237216d8f0b3539 Mon Sep 17 00:00:00 2001 From: shay-berman Date: Sun, 25 Nov 2018 12:56:22 +0200 Subject: [PATCH 1/8] Update README.md Add Spectrum Scale support into the README --- README.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 39e17ac67..d675dfb29 100644 --- a/README.md +++ b/README.md @@ -7,14 +7,16 @@ This project includes components for managing [Kubernetes persistent storage](ht - Ubiquity Dynamic Provisioner for creating and deleting persistent volumes - Ubiquity FlexVolume Driver CLI for attaching and detaching persistent volumes +The IBM official solution for Kubernetes, based on the Ubiquity project, is referred to as IBM Storage Enabler for Containers. You can download the installation package and its documentation from [IBM Fix Central TODO get the new location](https://www.ibm.com/support/fixcentral/swg/selectFixes?parent=Software%2Bdefined%2Bstorage&product=ibm/StorageSoftware/IBM+Spectrum+Connect&release=All&platform=Linux&function=all). + Currently, the following storage systems use Ubiquity: * IBM block storage. - The IBM block storage is supported for Kubernetes via IBM Spectrum Connect (3.4.0), previously known as IBM Spectrum Control Base Edition. Ubiquity communicates with the IBM storage systems through Spectrum Connect. Spectrum Connect creates a storage profile (for example, gold, silver or bronze) and makes it available for Kubernetes. For details about supported storage systems, refer to the latest Spectrum Connect release notes. + The IBM block storage is supported for Kubernetes via IBM Spectrum Connect. Ubiquity communicates with the IBM storage systems through Spectrum Connect. Spectrum Connect creates a storage profile (for example, gold, silver or bronze) and makes it available for Kubernetes. - The IBM official solution for Kubernetes, based on the Ubiquity project, is referred to as IBM Storage Enabler for Containers. You can download the installation package and its documentation from [IBM Fix Central](https://www.ibm.com/support/fixcentral/swg/selectFixes?parent=Software%2Bdefined%2Bstorage&product=ibm/StorageSoftware/IBM+Spectrum+Connect&release=All&platform=Linux&function=all). For details on the IBM Storage Enabler for Containers, see the relevant sections in the Spectrum Connect user guide. +* IBM Spectrum Scale -* IBM Spectrum Scale, for testing only. + The IBM Spectrum Scale file storage is supported for Kubernetes. Ubiquity communicates with Spectrum Scale system directly via Spectrum Scale REST API v2. The code is provided as is, without warranty. Any issue will be handled on a best-effort basis. @@ -22,7 +24,7 @@ The code is provided as is, without warranty. Any issue will be handled on a bes ![Ubiquity Overview](images/ubiquity_architecture_draft_for_github.jpg) -Deployment description: +Main deployment description: * Ubiquity Kubernetes Dynamic Provisioner (ubiquity-k8s-provisioner) runs as a Kubernetes deployment with replica=1. * Ubiquity Kubernetes FlexVolume (ubiquity-k8s-flex) runs as a Kubernetes daemonset on all the worker and master nodes. * Ubiquity (ubiquity) runs as a Kubernetes deployment with replica=1. From 49778df28fc91b95fc985e8863c17cac705b7d0c Mon Sep 17 00:00:00 2001 From: shay-berman Date: Sun, 25 Nov 2018 13:08:29 +0200 Subject: [PATCH 2/8] Update ibm-block-storage-via-sc.md fix syntax issues --- ibm-block-storage-via-sc.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/ibm-block-storage-via-sc.md b/ibm-block-storage-via-sc.md index 380929c25..c4c304306 100644 --- a/ibm-block-storage-via-sc.md +++ b/ibm-block-storage-via-sc.md @@ -236,8 +236,8 @@ The creation of a Pod/Deployment causes the FlexVolume to: * Attach the volume to the host (This action triggered from the controller-manager on the master node.) * Rescan and discover the multipath device of the new volume * Create xfs or ext4 filesystem on the device (if filesystem does not exist on the volume) -* Mount the new multipath device on /ubiquity/[WWN of the volume] -* Create a symbolic link /var/lib/kubelet/pods/[POD-ID]/volumes/ibm~ubiquity-k8s-flex/[PVC-ID] -> /ubiquity/[WWN of the volume] +* Mount the new multipath device on `/ubiquity/[WWN of the volume]` +* Create a symbolic link `/var/lib/kubelet/pods/[POD-ID]/volumes/ibm~ubiquity-k8s-flex/[PVC-ID]` -> `/ubiquity/[WWN of the volume]` For example, to create a Pod `pod1` that uses the PVC `pvc1` that was already created: ```bash @@ -310,8 +310,8 @@ lrwxrwxrwx. 1 root root 42 Aug 13 22:41 pvc-254e4b5e-805d-11e7-a42b-005056a46c49 ### Deleting a Pod The Kuberenetes delete Pod command: -* Removes symbolic link /var/lib/kubelet/pods/[POD-ID]/volumes/ibm~ubiquity-k8s-flex/[PVC-ID] -> /ubiquity/[WWN of the volume] -* Unmounts the new multipath device on /ubiquity/[WWN of the volume] +* Removes symbolic link `/var/lib/kubelet/pods/[POD-ID]/volumes/ibm~ubiquity-k8s-flex/[PVC-ID]` -> `/ubiquity/[WWN of the volume]` +* Unmounts the new multipath device on `/ubiquity/[WWN of the volume]` * Removes the multipath device of the volume * Detaches (unmaps) the volume from the host * Rescans with cleanup mode to remove the physical device files of the detached volume @@ -411,7 +411,7 @@ Filesystem Size Used Available Use% Mounted on COOL ``` -- Delete the POD so Kubernetes will reschedule the POD on a diffrent node (node1) +- Delete the POD so Kubernetes will reschedule the POD on a different node (node1) ```bash #> kubectl delete pod $pod pod "sanity-deployment-75f959859f-dh979" deleted From d539d7fd272cc2e5139ebeca3c5683c5eab8f8b4 Mon Sep 17 00:00:00 2001 From: deeghuge Date: Mon, 26 Nov 2018 19:29:05 +0530 Subject: [PATCH 3/8] spectrumscale - fixed the spectrumscale documentation along with examples --- ibm-spectrum-scale.md | 397 +++++++++++++++++++++++++++++++++++++----- 1 file changed, 358 insertions(+), 39 deletions(-) diff --git a/ibm-spectrum-scale.md b/ibm-spectrum-scale.md index a4e49044d..18a026cfd 100644 --- a/ibm-spectrum-scale.md +++ b/ibm-spectrum-scale.md @@ -1,57 +1,376 @@ # IBM Spectrum Scale -* [Deployment Prerequisities](#deployment-prerequisities) -* [Configuration](#configuring-ubiquity-docker-plugin-with-ubiquity-and-spectrum-scale) -* [Volume Creation](#volume-creation-using-spectrum-scale-storage-system) - * [Fileset Volumes](#creating-fileset-volumes) - * [Independent Fileset Volumes](#creating-independent-fileset-volumes) - * [Lightweight Volumes](#creating-lightweight-volumes) - * [Fileset with Quota Volumes](#creating-fileset-with-quota-volumes) +IBM Spectrum Scale can be used as persistent storage for Kubernetes via Ubiquity service. Ubiquity communicates with the IBM Spectrum Scale through IBM Spectrum Scale management api version 2. Filesets are created on IBM Spectrum Scale and made available for Ubiquity FlexVolume and Ubiquity Dynamic Provisioner. -## Deployment Prerequisities - * Spectrum-Scale - Ensure the Spectrum Scale client (NSD client) is installed and part of a Spectrum Scale cluster. - * NFS - Ensure hosts support mounting NFS file systems. +# Usage examples for Ubiquity Dynamic Provisioner and FlexVolume +The IBM official solution for Kubernetes, based on the Ubiquity project, is referred to as IBM Storage Enabler for Containers. You can download the installation package and its documentation (including full usage examples) from [IBM Fix Central](https://www.ibm.com/support/fixcentral/swg/selectFixes?parent=Software%2Bdefined%2Bstorage&product=ibm/StorageSoftware/IBM+Spectrum+Connect&release=All&platform=Linux&function=all). -## Ubiquity Dynamic Provisioner +Usage examples index: +* [Example 1 : Basic flow for running a stateful container in a pod](#example-1--basic-flow-for-running-a-stateful-container-with-ubiquity-volume) +* [Example 2 : Basic flow breakdown](#example-2--basic-flow-breakdown) +* [Example 3 : Deployment fail over](#example-3--deployment-fail-over-example) - -## Volume Creation usage -Volumes are dynamically provisioned by the dynamic provisioner. In order to have this functionality we need to create a storageClass that refers to the provisioner. Afterwards, we need to create PersistentVolumeClaims (PVC) that will be handled by the provisioner. -The provisioner will create volumes and bind them to the PVC. -### Available Storage Classes -These storage classes are described in the YAML files in `deploy` folder: -* spectrum-scale-fileset - described in `deploy/storage_class_fileset.yml`, it allows the dynamic provisioner to create volumes out of Spectrum Scale filesets. -* spectrum-scale-fileset-lightweight - described in `deploy/storage_class_lightweight.yml`, it allows the dynamic provisioner to create volumes out of sub-directories of filesets. +## Example 1 : Basic flow for running a stateful container with Ubiquity volume +Flow overview: +1. Create a StorageClass `spectrumscale-primaryfs` that refers to Spectrum Scale filesysten `primaryfs`. +2. Create a PVC `pvc1` that uses the StorageClass `spectrumscale-primaryfs`. +3. Create a Pod `pod1` with container `container1` that uses PVC `pvc1`. +3. Start I/Os into `/data/myDATA` in `pod1\container1`. +4. Delete the `pod1` and then create a new `pod1` with the same PVC and verify that the file `/data/myDATA` still exists. +5. Delete the `pod1` `pvc1`, `pv` and storage class `spectrumscale-primaryfs`. -* spectrum-scale-fileset-nfs - described in `deploy/storage_class_fileset_nfs.yml`, it allows the dynamic provisioner to create volumes out of Spectrum Scale filesets based on NFS. +Relevant yml files (`storage-class-primaryfs.yml`, `pvc1.yml` and `pod1.yml`): +```bash +#> cat storage-class-primaryfs.yml pvc1.yml pod1.yml +kind: StorageClass +apiVersion: storage.k8s.io/v1beta1 +metadata: + name: "spectrumscale-primaryfs" + labels: + product: ibm-storage-enabler-for-containers +provisioner: "ubiquity/flex" +parameters: + backend: "spectrum-scale" + filesystem: "primaryfs" + type: fileset + + +kind: PersistentVolumeClaim +apiVersion: v1 +metadata: + name: "pvc1" + annotations: + volume.beta.kubernetes.io/storage-class: "spectrumscale-primaryfs" +spec: + accessModes: + - ReadWriteOnce # Ubiquity Spectrum Scale backend supports ReadWriteOnce and ReadWriteMany mode. + resources: + requests: + storage: 1Gi # Size in Gi unit only + + +kind: Pod +apiVersion: v1 +metadata: + name: pod1 # Pod name +spec: + containers: + - name: container1 # Container name + image: alpine:latest + command: [ "/bin/sh", "-c", "--" ] + args: [ "while true; do sleep 30; done;" ] + volumeMounts: + - name: vol1 + mountPath: "/data" # Where to mount the vol1(pvc1) + restartPolicy: "Never" + volumes: + - name: vol1 + persistentVolumeClaim: + claimName: pvc1 +``` + +Running the basic flow: +```bash +#> kubectl create -f storage-class-primaryfs.yml -f pvc1.yml -f pod1.yml +storageclass "spectrumscale-primaryfs" created +persistentvolumeclaim "pvc1" created +pod "pod1" created + +#### Wait for PV to be created and pod1 to be in the Running state... + +#> kubectl get storageclass spectrumscale-primaryfs +NAME PROVISIONER AGE +spectrumscale-primaryfs ubiquity/flex 1m + +#> kubectl get pvc pvc1 +NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE +pvc1 Bound pvc-2073e8fd-f0bd-11e8-a8f1-000c29e45a24 1Gi RWO spectrumscale-primaryfs 1m + +#> kubectl get pv pvc-2073e8fd-f0bd-11e8-a8f1-000c29e45a24 +NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE +pvc-2073e8fd-f0bd-11e8-a8f1-000c29e45a24 1Gi RWO Delete Bound default/pvc1 spectrumscale-primaryfs 2m + +#> kubectl get pod pod1 +NAME READY STATUS RESTARTS AGE +pod1 1/1 Running 0 2m + +#> kubectl exec pod1 -c container1 -- sh -c "dd if=/dev/zero of=/data/myDATA bs=10M count=1" + +#> kubectl exec pod1 -c container1 -- sh -c "ls -l /data/myDATA" +-rw-r--r-- 1 root root 10485760 Nov 25 14:24 /data/myDATA + +#> kubectl delete -f pod1.yml +pod "pod1" deleted + +#### Wait for pod1 deletion... + +#> kubectl get pod pod1 +Error from server (NotFound): pods "pod1" not found + +#> kubectl create -f pod1.yml +pod "pod1" created + +#### Wait for pod1 to be in the Running state... + +#> kubectl get pod pod1 +NAME READY STATUS RESTARTS AGE +pod1 1/1 Running 0 46s + +#### Verify the /data/myDATA still exist + +#> kubectl exec pod1 -c container1 -- sh -c "ls -l /data/myDATA" +-rw-r--r-- 1 root root 10485760 Nov 25 14:24 /data/myDATA + +### Delete pod1, pvc1, pv and the gold storage class +#> kubectl delete -f pod1.yml -f pvc1.yml -f storage-class-primaryfs.yml +pod "pod1" deleted +persistentvolumeclaim "pvc1" deleted +storageclass "spectrumscale-primaryfs" deleted +``` + + +## Example 2 : Basic flow breakdown +This section describes separate steps of the generic flow in greater detail. + + +### Creating a Storage Class +For example, to create a Storage Class named `spectrumscale-primaryfs` that refers to a Spectrum Scale filesystem `primaryfs` with fileset quota enabled. As a result, every volume from this storage class will be provisioned on the Spectrum Scale and each volume is a fileset in a Spectrum Scale filesystem. +```bash +#> cat storage-class-primaryfs.yml +kind: StorageClass +apiVersion: storage.k8s.io/v1beta1 +metadata: + name: "spectrumscale-primaryfs" + labels: + product: ibm-storage-enabler-for-containers +provisioner: "ubiquity/flex" +parameters: + backend: "spectrum-scale" + filesystem: "primaryfs" + type: fileset + +#> kubectl create -f storage-class-primaryfs.yml +storageclass "spectrumscale-primaryfs" created +``` + +List the newly created Storage Class: +```bash +#> kubectl get storageclass spectrumscale-primaryfs +NAME PROVISIONER AGE +spectrumscale-primaryfs ubiquity/flex 1m +``` + +### Creating a PersistentVolumeClaim +To create a PVC `pvc1` with size `1Gi` that uses the `spectrumscale-primaryfs` Storage Class: +```bash +#> cat pvc1.yml +kind: PersistentVolumeClaim +apiVersion: v1 +metadata: + name: "pvc1" + annotations: + volume.beta.kubernetes.io/storage-class: "spectrumscale-primaryfs" +spec: + accessModes: + - ReadWriteOnce # Ubiquity Spectrum Scale backend supports ReadWriteOnce and ReadWriteMany mode. + resources: + requests: + storage: 1Gi # Size in Gi unit only + +#> kubectl create -f pvc1.yml +persistentvolumeclaim "pvc1 created +``` + +Ubiquity Dynamic Provisioner automatically creates a PersistentVolume (PV) and binds it to the PVC. The PV name will be PVC-ID. The volume name on the storage will be `[PVC-ID]`. + +List a PersistentVolumeClaim and PersistentVolume +```bash +#> kubectl get pvc pvc1 +NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE +pvc1 Bound pvc-2073e8fd-f0bd-11e8-a8f1-000c29e45a24 1Gi RWO spectrumscale-primaryfs 1m -### Usage example: -In order to test the dynamic provisioner create a storage class: +#> kubectl get pv pvc-2073e8fd-f0bd-11e8-a8f1-000c29e45a24 +NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE +pvc-2073e8fd-f0bd-11e8-a8f1-000c29e45a24 1Gi RWO Delete Bound default/pvc1 spectrumscale-primaryfs 2m +``` + +### Create a Pod with an Ubiquity volume +The creation of a Pod/Deployment causes the FlexVolume to: +* Create a symbolic link /var/lib/kubelet/pods/[POD-ID]/volumes/ibm~ubiquity-k8s-flex/[PVC-ID] -> [fileset linkpath] + +For example, to create a Pod `pod1` that uses the PVC `pvc1` that was already created: +```bash +#> cat pod1.yml +kind: Pod +apiVersion: v1 +metadata: + name: pod1 # Pod name +spec: + containers: + - name: container1 # Container name + image: alpine:latest + command: [ "/bin/sh", "-c", "--" ] + args: [ "while true; do sleep 30; done;" ] + volumeMounts: + - name: vol1 + mountPath: "/data" # Where to mount the vol1(pvc1) + restartPolicy: "Never" + volumes: + - name: vol1 + persistentVolumeClaim: + claimName: pvc1 + +#> kubectl create -f pod1.yml +pod "pod1" created +``` + +To display the newly created `pod1` and write data to the persistent volume of `pod1`: +```bash +#> kubectl get pod pod1 +NAME READY STATUS RESTARTS AGE +pod1 1/1 Running 0 16m + +#> kubectl exec pod1 -c container1 -- sh -c "df -h /data" +Filesystem Size Used Available Use% Mounted on +primaryfs 100.0G 4.5G 95.5G 4% /data + +#> kubectl exec pod1 -c container1 -- sh -c "dd if=/dev/zero of=/data/myDATA bs=10M count=1" + +#> kubectl exec pod1 -c container1 -- sh -c "ls -l /data/myDATA" +-rw-r--r-- 1 root root 10485760 Nov 25 14:24 /data/myDATA +``` + +### Deleting a Pod +The Kuberenetes delete Pod command: +* Removes symbolic link /var/lib/kubelet/pods/[POD-ID]/volumes/ibm~ubiquity-k8s-flex/[PVC-ID] -> [fileset link path] + +For example: +```bash +#> kubectl delete pod pod1 +pod "pod1" deleted +``` + +### Removing a volume +Removing the PVC deletes the PVC and its PV. + +For example: +```bash +#> kubectl delete -f pvc1.yml +persistentvolumeclaim "pvc1" deleted +``` + +### Removing a Storage Class +For example: ```bash -kubectl create -f deploy/storage_class_fileset.yml +#> kubectl delete -f storage-class-primaryfs.yml +storageclass "spectrumscale-primaryfs" deleted ``` -The class is referring to `ubiquity/flex` as its provisioner. So this provisioner should be up and running in order to be able to dynamically create volumes. -`filesystem` parameter refers to the name of the filesystem to be used by the dynamic provisioner to create the volume. `backend` parameter is used to select the backend used by the system. -The `type` parameter is used to specify the type of volumes to be provisioned by spectrum-scale backend. -The following snippet shows a sample persistent volume claim for using dynamic provisioning: +## Example 3 : Deployment fail over example +This section describes how to run stateful Pod with k8s Deployment object, and then delete the Pod and see how kubernetes schedule the pod on different node and the PV follows. + + +- Prerequisits +1. Create the same storage class (as previous example) +```bash +#> kubectl create -f storage-class-primaryfs.yml +storageclass "spectrumscale-primaryfs" created +``` +2. Create the PVC (as previous example) ```bash -kubectl create -f deploy/pvc_fileset.yml +#> kubectl create -f pvc1.yml +persistentvolumeclaim "pvc1" created ``` -The claim is referring to `spectrum-scale-fileset` as the storage class to be used. -A persistent volume should be dynamically created and bound to the claim. +- Create Kubernetes Deployment with stateful POD (on node6) and write some data inside +```bash +#> cat deployment1.yml +apiVersion: "extensions/v1beta1" +kind: Deployment +metadata: + name: deployment1 +spec: + replicas: 1 + template: + metadata: + labels: + app: deployment1 + spec: + containers: + - name: container1 + image: alpine:latest + command: [ "/bin/sh", "-c", "--" ] + args: [ "while true; do sleep 30; done;" ] + volumeMounts: + - name: pvc + mountPath: "/data" + volumes: + - name: pvc + persistentVolumeClaim: + claimName: pvc1 + +#> kubectl create -f deployment1.yml +deployment "deployment1" created +#> kubectl get -o wide deploy,pod +NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR +deploy/deployment1 1 1 1 1 1m container1 alpine:latest app=deployment1 -## Ubiquity FlexVolume CLI +NAME READY STATUS RESTARTS AGE IP NODE +po/deployment1-df6dd77d4-6kb7k 1/1 Running 0 1m 10.244.3.48 node6 -### Configuring Ubiquity Docker Plugin with Ubiquity and Spectrum Scale - - In case you need Spectrum Scale with NFS connectivity add this 2 lines into the ubiquity-client.conf file. - -```toml -[SpectrumNfsRemoteConfig] -ClientConfig = "192.168.1.0/24(Access_Type=RW,Protocols=3:4,Transports=TCP:UDP)" +#> pod=`kubectl get pod | awk '/deployment1/{print $1}'` +#> echo $pod +deployment1-df6dd77d4-6kb7k + +#> kubectl exec $pod -- /bin/sh -c "echo COOL > /data/file" +#> kubectl exec $pod -- /bin/sh -c "cat /data/file" +COOL ``` -Where the ClientConfig contains the CIDR that the node where the volume will be mounted belongs to. + +- Delete the POD so Kubernetes will reschedule the POD on a diffrent node (node5) +```bash +#> kubectl delete pod $pod +pod "deployment1-df6dd77d4-6kb7k" deleted +#> kubectl get -o wide deploy,pod +NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR +deploy/deployment1 1 1 1 0 4m container1 alpine:latest app=deployment1 + +NAME READY STATUS RESTARTS AGE IP NODE +po/deployment1-df6dd77d4-6kb7k 1/1 Terminating 0 4m 10.244.3.48 node6 +po/deployment1-df6dd77d4-ngfdp 0/1 ContainerCreating 0 23s node5 + +############# +## Wait a few seconds +############# + +#> kubectl get -o wide deploy,pod +NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR +deploy/deployment1 1 1 1 1 5m container1 alpine:latest app=deployment1 + +NAME READY STATUS RESTARTS AGE IP NODE +po/deployment1-df6dd77d4-ngfdp 1/1 Running 0 1m 10.244.2.47 node5 + +############# +## Now check data remains +############# +#> pod=`kubectl get pod | awk '/deployment1/{print $1}'` +#> echo $pod +deployment1-df6dd77d4-ngfdp +#> kubectl exec $pod -- /bin/sh -c "cat /data/file" +COOL + +``` + +- Tier down the Deployment, PVC, PV and Storage Class +```bash +#> kubectl delete -f deployment1.yml -f pvc1.yml -f storage-class-primaryfs.yml +deployment "deployment1" deleted +persistentvolumeclaim "pvc1" deleted +storageclass "spectrumscale-primaryfs" deleted +``` + +
+ +**Note:** For detailed usage examples, refer to the IBM Storage Enabler for Containers [user guide](https://www-945.ibm.com/support/fixcentral/swg/selectFixes?parent=Software%2Bdefined%2Bstorage&product=ibm/StorageSoftware/IBM+Spectrum+Connect&release=All&platform=Linux&function=all). From 8dc6ed4604e8040d84e5727ee201b902a7bf84f2 Mon Sep 17 00:00:00 2001 From: deeghuge Date: Mon, 26 Nov 2018 20:43:47 +0530 Subject: [PATCH 4/8] spectrumscale - Formatting changes based on review comments from scale doc team --- ibm-spectrum-scale.md | 40 +++++++++++++++++++--------------------- 1 file changed, 19 insertions(+), 21 deletions(-) diff --git a/ibm-spectrum-scale.md b/ibm-spectrum-scale.md index 18a026cfd..2cf12ba51 100644 --- a/ibm-spectrum-scale.md +++ b/ibm-spectrum-scale.md @@ -12,12 +12,12 @@ Usage examples index: ## Example 1 : Basic flow for running a stateful container with Ubiquity volume Flow overview: -1. Create a StorageClass `spectrumscale-primaryfs` that refers to Spectrum Scale filesysten `primaryfs`. +1. Create a StorageClass `spectrumscale-primaryfs` that refers to IBM Spectrum Scale filesystem `primaryfs`. 2. Create a PVC `pvc1` that uses the StorageClass `spectrumscale-primaryfs`. 3. Create a Pod `pod1` with container `container1` that uses PVC `pvc1`. -3. Start I/Os into `/data/myDATA` in `pod1\container1`. +3. Write some data into file `/data/myDATA` in `pod1\container1`. 4. Delete the `pod1` and then create a new `pod1` with the same PVC and verify that the file `/data/myDATA` still exists. -5. Delete the `pod1` `pvc1`, `pv` and storage class `spectrumscale-primaryfs`. +5. Delete the `pod1` `pvc1`, `pv` and `spectrumscale-primaryfs`. Relevant yml files (`storage-class-primaryfs.yml`, `pvc1.yml` and `pod1.yml`): ```bash @@ -43,7 +43,7 @@ metadata: volume.beta.kubernetes.io/storage-class: "spectrumscale-primaryfs" spec: accessModes: - - ReadWriteOnce # Ubiquity Spectrum Scale backend supports ReadWriteOnce and ReadWriteMany mode. + - ReadWriteOnce # Ubiquity spectrum-scale backend supports ReadWriteOnce and ReadWriteMany mode. resources: requests: storage: 1Gi # Size in Gi unit only @@ -121,7 +121,7 @@ pod1 1/1 Running 0 46s #> kubectl exec pod1 -c container1 -- sh -c "ls -l /data/myDATA" -rw-r--r-- 1 root root 10485760 Nov 25 14:24 /data/myDATA -### Delete pod1, pvc1, pv and the gold storage class +### Delete pod1, pvc1, pv and the spectrumscale-primaryfs #> kubectl delete -f pod1.yml -f pvc1.yml -f storage-class-primaryfs.yml pod "pod1" deleted persistentvolumeclaim "pvc1" deleted @@ -133,8 +133,8 @@ storageclass "spectrumscale-primaryfs" deleted This section describes separate steps of the generic flow in greater detail. -### Creating a Storage Class -For example, to create a Storage Class named `spectrumscale-primaryfs` that refers to a Spectrum Scale filesystem `primaryfs` with fileset quota enabled. As a result, every volume from this storage class will be provisioned on the Spectrum Scale and each volume is a fileset in a Spectrum Scale filesystem. +### Creating a StorageClass +Create a StorageClass named `spectrumscale-primaryfs` that refers to a IBM Spectrum Scale filesystem `primaryfs` with fileset quota enabled. As a result, every volume from this StorageClass will be provisioned on the IBM Spectrum Scale and each volume is a fileset in a IBM Spectrum Scale filesystem. ```bash #> cat storage-class-primaryfs.yml kind: StorageClass @@ -153,7 +153,7 @@ parameters: storageclass "spectrumscale-primaryfs" created ``` -List the newly created Storage Class: +List the newly created StorageClass: ```bash #> kubectl get storageclass spectrumscale-primaryfs NAME PROVISIONER AGE @@ -161,7 +161,7 @@ spectrumscale-primaryfs ubiquity/flex 1m ``` ### Creating a PersistentVolumeClaim -To create a PVC `pvc1` with size `1Gi` that uses the `spectrumscale-primaryfs` Storage Class: +Create a PVC `pvc1` with size `1Gi` that uses the `spectrumscale-primaryfs` StorageClass: ```bash #> cat pvc1.yml kind: PersistentVolumeClaim @@ -172,7 +172,7 @@ metadata: volume.beta.kubernetes.io/storage-class: "spectrumscale-primaryfs" spec: accessModes: - - ReadWriteOnce # Ubiquity Spectrum Scale backend supports ReadWriteOnce and ReadWriteMany mode. + - ReadWriteOnce # Ubiquity IBM Spectrum Scale backend supports ReadWriteOnce and ReadWriteMany mode. resources: requests: storage: 1Gi # Size in Gi unit only @@ -195,8 +195,7 @@ pvc-2073e8fd-f0bd-11e8-a8f1-000c29e45a24 1Gi RWO Delete ``` ### Create a Pod with an Ubiquity volume -The creation of a Pod/Deployment causes the FlexVolume to: -* Create a symbolic link /var/lib/kubelet/pods/[POD-ID]/volumes/ibm~ubiquity-k8s-flex/[PVC-ID] -> [fileset linkpath] +The creation of a Pod/Deployment causes the FlexVolume to create a symbolic link /var/lib/kubelet/pods/[POD-ID]/volumes/ibm~ubiquity-k8s-flex/[PVC-ID] -> [fileset linkpath] For example, to create a Pod `pod1` that uses the PVC `pvc1` that was already created: ```bash @@ -224,7 +223,7 @@ spec: pod "pod1" created ``` -To display the newly created `pod1` and write data to the persistent volume of `pod1`: +Display the newly created `pod1` and write data to the persistent volume of `pod1`: ```bash #> kubectl get pod pod1 NAME READY STATUS RESTARTS AGE @@ -259,7 +258,7 @@ For example: persistentvolumeclaim "pvc1" deleted ``` -### Removing a Storage Class +### Removing a StorageClass For example: ```bash #> kubectl delete -f storage-class-primaryfs.yml @@ -268,22 +267,21 @@ storageclass "spectrumscale-primaryfs" deleted ## Example 3 : Deployment fail over example -This section describes how to run stateful Pod with k8s Deployment object, and then delete the Pod and see how kubernetes schedule the pod on different node and the PV follows. +This section describes how to run stateful Pod with k8s Deployment object and how the kubernetes schedule the pod on different node when the pod is deleted. -- Prerequisits -1. Create the same storage class (as previous example) +### Create the StorageClass (same as previous example). ```bash #> kubectl create -f storage-class-primaryfs.yml storageclass "spectrumscale-primaryfs" created ``` -2. Create the PVC (as previous example) +### Create the PVC (same as previous example) ```bash #> kubectl create -f pvc1.yml persistentvolumeclaim "pvc1" created ``` -- Create Kubernetes Deployment with stateful POD (on node6) and write some data inside +### Create Kubernetes Deployment with stateful POD (on node6) and write some data inside. ```bash #> cat deployment1.yml apiVersion: "extensions/v1beta1" @@ -329,7 +327,7 @@ deployment1-df6dd77d4-6kb7k COOL ``` -- Delete the POD so Kubernetes will reschedule the POD on a diffrent node (node5) +### Delete the POD so Kubernetes will reschedule the POD on a diffrent node(node5). ```bash #> kubectl delete pod $pod pod "deployment1-df6dd77d4-6kb7k" deleted @@ -363,7 +361,7 @@ COOL ``` -- Tier down the Deployment, PVC, PV and Storage Class +### Tier down the Deployment, PVC, PV and StorageClass. ```bash #> kubectl delete -f deployment1.yml -f pvc1.yml -f storage-class-primaryfs.yml deployment "deployment1" deleted From b2605fd4057bab8b542c10c2f7ac4e5ac36b9f06 Mon Sep 17 00:00:00 2001 From: deeghuge Date: Mon, 26 Nov 2018 20:58:14 +0530 Subject: [PATCH 5/8] spectrumscale - minor fixes as per doc person comments --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index d675dfb29..5afede276 100644 --- a/README.md +++ b/README.md @@ -16,7 +16,7 @@ Currently, the following storage systems use Ubiquity: * IBM Spectrum Scale - The IBM Spectrum Scale file storage is supported for Kubernetes. Ubiquity communicates with Spectrum Scale system directly via Spectrum Scale REST API v2. + The IBM Spectrum Scale file storage is supported for Kubernetes. Ubiquity communicates with IBM Spectrum Scale system directly via IBM Spectrum Scale management API v2. The code is provided as is, without warranty. Any issue will be handled on a best-effort basis. From 894a5ca5860ab8e1d7db91f8c6250f0c698cad96 Mon Sep 17 00:00:00 2001 From: deeghuge Date: Tue, 27 Nov 2018 11:14:00 +0530 Subject: [PATCH 6/8] Addressing review comments --- ibm-spectrum-scale.md | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/ibm-spectrum-scale.md b/ibm-spectrum-scale.md index 2cf12ba51..74b599dd2 100644 --- a/ibm-spectrum-scale.md +++ b/ibm-spectrum-scale.md @@ -23,7 +23,7 @@ Relevant yml files (`storage-class-primaryfs.yml`, `pvc1.yml` and `pod1.yml`): ```bash #> cat storage-class-primaryfs.yml pvc1.yml pod1.yml kind: StorageClass -apiVersion: storage.k8s.io/v1beta1 +apiVersion: storage.k8s.io/v1 metadata: name: "spectrumscale-primaryfs" labels: @@ -39,9 +39,8 @@ kind: PersistentVolumeClaim apiVersion: v1 metadata: name: "pvc1" - annotations: - volume.beta.kubernetes.io/storage-class: "spectrumscale-primaryfs" spec: + storageClassName: "spectrumscale-primaryfs" accessModes: - ReadWriteOnce # Ubiquity spectrum-scale backend supports ReadWriteOnce and ReadWriteMany mode. resources: @@ -138,7 +137,7 @@ Create a StorageClass named `spectrumscale-primaryfs` that refers to a IBM Spect ```bash #> cat storage-class-primaryfs.yml kind: StorageClass -apiVersion: storage.k8s.io/v1beta1 +apiVersion: storage.k8s.io/v1 metadata: name: "spectrumscale-primaryfs" labels: @@ -168,9 +167,8 @@ kind: PersistentVolumeClaim apiVersion: v1 metadata: name: "pvc1" - annotations: - volume.beta.kubernetes.io/storage-class: "spectrumscale-primaryfs" spec: + storageClassName: "spectrumscale-primaryfs" accessModes: - ReadWriteOnce # Ubiquity IBM Spectrum Scale backend supports ReadWriteOnce and ReadWriteMany mode. resources: @@ -284,7 +282,7 @@ persistentvolumeclaim "pvc1" created ### Create Kubernetes Deployment with stateful POD (on node6) and write some data inside. ```bash #> cat deployment1.yml -apiVersion: "extensions/v1beta1" +apiVersion: "extensions/v1" kind: Deployment metadata: name: deployment1 From f3a934c50c7ea37319ffa1f22cff113852738217 Mon Sep 17 00:00:00 2001 From: shay-berman Date: Wed, 5 Dec 2018 22:30:33 +0200 Subject: [PATCH 7/8] Update README.md Apply Samuel comments and new URL to fix central --- README.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 5afede276..995ef3536 100644 --- a/README.md +++ b/README.md @@ -7,16 +7,16 @@ This project includes components for managing [Kubernetes persistent storage](ht - Ubiquity Dynamic Provisioner for creating and deleting persistent volumes - Ubiquity FlexVolume Driver CLI for attaching and detaching persistent volumes -The IBM official solution for Kubernetes, based on the Ubiquity project, is referred to as IBM Storage Enabler for Containers. You can download the installation package and its documentation from [IBM Fix Central TODO get the new location](https://www.ibm.com/support/fixcentral/swg/selectFixes?parent=Software%2Bdefined%2Bstorage&product=ibm/StorageSoftware/IBM+Spectrum+Connect&release=All&platform=Linux&function=all). +The IBM official solution for Kubernetes, based on the Ubiquity project, is referred to as IBM Storage Enabler for Containers. You can download the installation package and its documentation from [IBM Fix Central](http://www.ibm.com/support/fixcentral/swg/quickorder?parent=Software%20defined%20storage&product=ibm/StorageSoftware/IBM+Storage+Enabler+for+Containers&release=All&platform=All&function=all&source=fc). -Currently, the following storage systems use Ubiquity: +Ubiquity supports the following storage systems: * IBM block storage. - The IBM block storage is supported for Kubernetes via IBM Spectrum Connect. Ubiquity communicates with the IBM storage systems through Spectrum Connect. Spectrum Connect creates a storage profile (for example, gold, silver or bronze) and makes it available for Kubernetes. + IBM block storage is supported for Kubernetes via IBM Spectrum Connect. Ubiquity communicates with the IBM storage systems through Spectrum Connect. Spectrum Connect creates a storage profile (for example, gold, silver or bronze) and makes it available for Kubernetes. * IBM Spectrum Scale - The IBM Spectrum Scale file storage is supported for Kubernetes. Ubiquity communicates with IBM Spectrum Scale system directly via IBM Spectrum Scale management API v2. + IBM Spectrum Scale file storage is supported for Kubernetes. Ubiquity communicates with IBM Spectrum Scale system directly via IBM Spectrum Scale management API v2. The code is provided as is, without warranty. Any issue will be handled on a best-effort basis. @@ -26,7 +26,7 @@ The code is provided as is, without warranty. Any issue will be handled on a bes Main deployment description: * Ubiquity Kubernetes Dynamic Provisioner (ubiquity-k8s-provisioner) runs as a Kubernetes deployment with replica=1. - * Ubiquity Kubernetes FlexVolume (ubiquity-k8s-flex) runs as a Kubernetes daemonset on all the worker and master nodes. + * Ubiquity Kubernetes FlexVolume (ubiquity-k8s-flex) runs as a Kubernetes daemonset in all the worker and master nodes. * Ubiquity (ubiquity) runs as a Kubernetes deployment with replica=1. * Ubiquity database (ubiquity-db) runs as a Kubernetes deployment with replica=1. From 485053b691926f4cdea718ffb305f1f69e1b8a4e Mon Sep 17 00:00:00 2001 From: shay-berman Date: Wed, 5 Dec 2018 22:33:36 +0200 Subject: [PATCH 8/8] Update glide.yaml --- glide.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/glide.yaml b/glide.yaml index 7cf95eb9f..12fcf9d1e 100644 --- a/glide.yaml +++ b/glide.yaml @@ -110,7 +110,7 @@ import: subpackages: - lib - package: github.com/IBM/ubiquity - version: b5bddd136a049339c92eb44fd5bb499215a22c44 + version: f203222e3190aca3de0bf851316a2de3b68efcad subpackages: - remote - resources