Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update local-volume-provisioner + doc #9463

Merged
merged 1 commit into from
Nov 7, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -170,7 +170,7 @@ Note: Upstart/SysV init based OS types are not supported.
- [cinder-csi-plugin](https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/cinder-csi-plugin/using-cinder-csi-plugin.md) v1.22.0
- [gcp-pd-csi-plugin](https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver) v1.4.0
- [local-path-provisioner](https://github.com/rancher/local-path-provisioner) v0.0.22
- [local-volume-provisioner](https://github.com/kubernetes-sigs/sig-storage-local-static-provisioner) v2.4.0
- [local-volume-provisioner](https://github.com/kubernetes-sigs/sig-storage-local-static-provisioner) v2.5.0

## Container Runtime Notes

Expand Down
125 changes: 67 additions & 58 deletions docs/kubernetes-apps/local_volume_provisioner.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,11 @@
# Local Storage Provisioner
# Local Static Storage Provisioner
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# Local Static Storage Provisioner
# Local Persistence Volume Static Provisioner

as the title of https://github.com/kubernetes-sigs/sig-storage-local-static-provisioner


The [local storage provisioner](https://github.com/kubernetes-incubator/external-storage/tree/master/local-volume)
The [local static storage provisioner](https://github.com/kubernetes-sigs/sig-storage-local-static-provisioner)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

is NOT a dynamic storage provisioner as you would
expect from a cloud provider. Instead, it simply creates PersistentVolumes for
all mounts under the host_dir of the specified storage class.
all mounts under the `host_dir` of the specified storage class.
These storage classes are specified in the `local_volume_provisioner_storage_classes` nested dictionary.

Example:

```yaml
Expand All @@ -16,15 +17,18 @@ local_volume_provisioner_storage_classes:
host_dir: /mnt/fast-disks
mount_dir: /mnt/fast-disks
block_cleaner_command:
- "/scripts/shred.sh"
- "2"
- "/scripts/shred.sh"
- "2"
volume_mode: Filesystem
fs_type: ext4
```

For each key in `local_volume_provisioner_storage_classes` a storageClass with the
same name is created. The subkeys of each storage class are converted to camelCase and added
as attributes to the storageClass.
For each key in `local_volume_provisioner_storage_classes` a "storage class" with
the same name is created in the entry `storageClassMap` of the ConfigMap `local-volume-provisioner`.
The subkeys of each storage class in `local_volume_provisioner_storage_classes`
are converted to camelCase and added as attributes to the storage class in the
ConfigMap.

The result of the above example is:

```yaml
Expand All @@ -43,80 +47,85 @@ data:
fsType: ext4
```

The default StorageClass is local-storage on /mnt/disks,
the rest of this doc will use that path as an example.
Additionally, a StorageClass object (`storageclasses.storage.k8s.io`) is also
created for each storage class:

```bash
$ kubectl get storageclasses.storage.k8s.io
NAME PROVISIONER RECLAIMPOLICY
fast-disks kubernetes.io/no-provisioner Delete
local-storage kubernetes.io/no-provisioner Delete
```

The default StorageClass is `local-storage` on `/mnt/disks`;
the rest of this documentation will use that path as an example.

## Examples to create local storage volumes

1. tmpfs method:
1. Using tmpfs

``` bash
for vol in vol1 vol2 vol3; do
mkdir /mnt/disks/$vol
mount -t tmpfs -o size=5G $vol /mnt/disks/$vol
done
```
```bash
for vol in vol1 vol2 vol3; do
mkdir /mnt/disks/$vol
mount -t tmpfs -o size=5G $vol /mnt/disks/$vol
done
```

The tmpfs method is not recommended for production because the mount is not
persistent and data will be deleted on reboot.
The tmpfs method is not recommended for production because the mounts are not
persistent and data will be deleted on reboot.

1. Mount physical disks

``` bash
mkdir /mnt/disks/ssd1
mount /dev/vdb1 /mnt/disks/ssd1
```
```bash
mkdir /mnt/disks/ssd1
mount /dev/vdb1 /mnt/disks/ssd1
```

Physical disks are recommended for production environments because it offers
complete isolation in terms of I/O and capacity.
Physical disks are recommended for production environments because it offers
complete isolation in terms of I/O and capacity.

1. Mount unpartitioned physical devices

``` bash
for disk in /dev/sdc /dev/sdd /dev/sde; do
ln -s $disk /mnt/disks
done
```
```bash
for disk in /dev/sdc /dev/sdd /dev/sde; do
ln -s $disk /mnt/disks
done
```

This saves time of precreating filesystems. Note that your storageclass must have
volume_mode set to "Filesystem" and fs_type defined. If either is not set, the
disk will be added as a raw block device.
This saves time of precreating filesystems. Note that your storageclass must have
`volume_mode` set to `"Filesystem"` and `fs_type` defined. If either is not set, the
disk will be added as a raw block device.

1. File-backed sparsefile method
1. PersistentVolumes with `volumeMode="Block"`

``` bash
truncate /mnt/disks/disk5 --size 2G
mkfs.ext4 /mnt/disks/disk5
mkdir /mnt/disks/vol5
mount /mnt/disks/disk5 /mnt/disks/vol5
```
Just like above, you can create PersistentVolumes with volumeMode `Block`
by creating a symbolic link under discovery directory to the block device on
the node, if you set `volume_mode` to `"Block"`. This will create a volume
presented into a Pod as a block device, without any filesystem on it.

If you have a development environment and only one disk, this is the best way
to limit the quota of persistent volumes.
1. File-backed sparsefile method

1. Simple directories
```bash
truncate /mnt/disks/disk5 --size 2G
mkfs.ext4 /mnt/disks/disk5
mkdir /mnt/disks/vol5
mount /mnt/disks/disk5 /mnt/disks/vol5
```

In a development environment using `mount --bind` works also, but there is no capacity
management.
If you have a development environment and only one disk, this is the best way
to limit the quota of persistent volumes.

1. Block volumeMode PVs
1. Simple directories

Create a symbolic link under discovery directory to the block device on the node. To use
raw block devices in pods, volume_type should be set to "Block".
In a development environment, using `mount --bind` works also, but there is no capacity
management.

## Usage notes

Beta PV.NodeAffinity field is used by default. If running against an older K8s
version, the useAlphaAPI flag must be set in the configMap.

The volume provisioner cannot calculate volume sizes correctly, so you should
delete the daemonset pod on the relevant host after creating volumes. The pod
will be recreated and read the size correctly.

Make sure to make any mounts persist via /etc/fstab or with systemd mounts (for
Flatcar Container Linux). Pods with persistent volume claims will not be
Make sure to make any mounts persist via `/etc/fstab` or with systemd mounts (for
Flatcar Container Linux or Fedora CoreOS). Pods with persistent volume claims will not be
able to start if the mounts become unavailable.

## Further reading

Refer to the upstream docs here: <https://github.com/kubernetes-incubator/external-storage/tree/master/local-volume>
Refer to the upstream docs here: <https://github.com/kubernetes-sigs/sig-storage-local-static-provisioner>
2 changes: 1 addition & 1 deletion roles/download/defaults/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -988,7 +988,7 @@ registry_image_tag: "{{ registry_version }}"
metrics_server_version: "v0.6.1"
metrics_server_image_repo: "{{ kube_image_repo }}/metrics-server/metrics-server"
metrics_server_image_tag: "{{ metrics_server_version }}"
local_volume_provisioner_version: "v2.4.0"
local_volume_provisioner_version: "v2.5.0"
local_volume_provisioner_image_repo: "{{ kube_image_repo }}/sig-storage/local-volume-provisioner"
local_volume_provisioner_image_tag: "{{ local_volume_provisioner_version }}"
cephfs_provisioner_version: "v2.1.0-k8s1.11"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,16 +6,15 @@ local_volume_provisioner_nodelabels: []
# - topology.kubernetes.io/region
# - topology.kubernetes.io/zone
local_volume_provisioner_tolerations: []
# Levarages Ansibles string to Python datatype casting. Otherwise the dict_key isn't substituted
# see https://github.com/ansible/ansible/issues/17324
local_volume_provisioner_use_node_name_only: false
# Leverages Ansible's string to Python datatype casting. Otherwise the dict_key isn't substituted.
# see https://github.com/ansible/ansible/issues/17324
local_volume_provisioner_storage_classes: |
{
"{{ local_volume_provisioner_storage_class | default('local-storage') }}": {
"host_dir": "{{ local_volume_provisioner_base_dir | default ('/mnt/disks') }}",
"mount_dir": "{{ local_volume_provisioner_mount_dir | default('/mnt/disks') }}",
"volume_mode": "Filesystem",
"fs_type": "ext4"

}
}
Original file line number Diff line number Diff line change
Expand Up @@ -24,17 +24,6 @@
- { name: local-volume-provisioner-cm, file: local-volume-provisioner-cm.yml, type: cm }
- { name: local-volume-provisioner-ds, file: local-volume-provisioner-ds.yml, type: ds }
- { name: local-volume-provisioner-sc, file: local-volume-provisioner-sc.yml, type: sc }
local_volume_provisioner_templates_for_psp_not_system_ns:
- { name: local-volume-provisioner-psp, file: local-volume-provisioner-psp.yml, type: psp }
- { name: local-volume-provisioner-psp-role, file: local-volume-provisioner-psp-role.yml, type: role }
- { name: local-volume-provisioner-psp-rb, file: local-volume-provisioner-psp-rb.yml, type: rolebinding }

- name: Local Volume Provisioner | Insert extra templates to Local Volume Provisioner templates list for PodSecurityPolicy
set_fact:
local_volume_provisioner_templates: "{{ local_volume_provisioner_templates[:2] + local_volume_provisioner_templates_for_psp_not_system_ns + local_volume_provisioner_templates[2:] }}"
when:
- podsecuritypolicy_enabled
- local_volume_provisioner_namespace != "kube-system"

- name: Local Volume Provisioner | Create manifests
template:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,18 @@ metadata:
name: local-volume-provisioner-node-clusterrole
namespace: {{ local_volume_provisioner_namespace }}
rules:
- apiGroups: [""]
resources: ["persistentvolumes"]
verbs: ["get", "list", "watch", "create", "delete"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["events"]
verbs: ["watch"]
- apiGroups: ["", "events.k8s.io"]
resources: ["events"]
verbs: ["create", "update", "patch"]
- apiGroups: [""]
resources: ["nodes"]
verbs: ["get"]
Original file line number Diff line number Diff line change
@@ -1,20 +1,6 @@
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: local-volume-provisioner-system-persistent-volume-provisioner
namespace: {{ local_volume_provisioner_namespace }}
subjects:
- kind: ServiceAccount
name: local-volume-provisioner
namespace: {{ local_volume_provisioner_namespace }}
roleRef:
kind: ClusterRole
name: system:persistent-volume-provisioner
apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: local-volume-provisioner-system-node
namespace: {{ local_volume_provisioner_namespace }}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@ spec:
spec:
priorityClassName: {% if local_volume_provisioner_namespace == 'kube-system' %}system-node-critical{% else %}k8s-cluster-critical{% endif %}{{''}}
serviceAccountName: local-volume-provisioner
nodeSelector:
kubernetes.io/os: linux
{% if local_volume_provisioner_tolerations %}
tolerations:
{{ local_volume_provisioner_tolerations | to_nice_yaml(indent=2) | indent(width=8) }}
Expand Down

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.