Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

doc(1.8.0, 1.7, 1.6): online volume expansion for encrypted volumes #1026

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,8 @@ For more information, see [cryptsetup(8)](https://man7.org/linux/man-pages/man8/
csi.storage.k8s.io/node-publish-secret-namespace: "longhorn-system"
csi.storage.k8s.io/node-stage-secret-name: "longhorn-crypto"
csi.storage.k8s.io/node-stage-secret-namespace: "longhorn-system"
csi.storage.k8s.io/node-expand-secret-name: "longhorn-crypto"
csi.storage.k8s.io/node-expand-secret-namespace: "longhorn-system"
```

- Example of a StorageClass with a volume-specific Secret:
Expand All @@ -85,6 +87,8 @@ For more information, see [cryptsetup(8)](https://man7.org/linux/man-pages/man8/
csi.storage.k8s.io/node-publish-secret-namespace: ${pvc.namespace}
csi.storage.k8s.io/node-stage-secret-name: ${pvc.name}
csi.storage.k8s.io/node-stage-secret-namespace: ${pvc.namespace}
csi.storage.k8s.io/node-expand-secret-name: ${pvc.name}
csi.storage.k8s.io/node-expand-secret-namespace: ${pvc.namespace}
```

# Using an Encrypted Volume
Expand All @@ -96,9 +100,14 @@ A newly-created PVC remains in the `Pending` state until the associated Secret i

# Filesystem Expansion

Longhorn supports [offline expansion](../../../nodes-and-volumes/volumes/expansion/#encrypted-volume) for encrypted volumes.
Longhorn supports [both online and offline expansion](../../../nodes-and-volumes/volumes/expansion/#encrypted-volume) for encrypted volumes.

StorageClass parameters are needed to enable online expansion:

- `csi.storage.k8s.io/node-expand-secret-name`
- `csi.storage.k8s.io/node-expand-secret-namespace`

# History
- Encryption of volumes in `Filesystem` mode available starting v1.2.0 ([#1859](https://github.com/longhorn/longhorn/issues/1859))

- Encryption of volumes in `Block` mode available starting v1.6.0 ([#4883](https://github.com/longhorn/longhorn/issues/4883))
- Encryption of volumes in `Block` mode available starting v1.6.0 ([#4883](https://github.com/longhorn/longhorn/issues/4883))
10 changes: 8 additions & 2 deletions content/docs/1.6.4/nodes-and-volumes/volumes/expansion.md
Original file line number Diff line number Diff line change
Expand Up @@ -116,7 +116,13 @@ If a volume is reverted to a snapshot with smaller size, the frontend of the vol

#### Encrypted volume

Due to [the upstream limitation](https://kubernetes.io/blog/2022/09/21/kubernetes-1-25-use-secrets-while-expanding-csi-volumes-on-node-alpha/), Longhorn cannot handle **online** expansion for encrypted volumes automatically unless you enable the feature gate `CSINodeExpandSecret`.
Longhorn support for online expansion depends on Kubernetes.
- Kubernetes natively supports [authenticated CSI storage resizing](https://kubernetes.io/blog/2023/12/15/csi-node-expand-secret-support-ga/) starting in v1.29.
- In [Kubernetes v1.25 to v1.28](https://kubernetes.io/blog/2022/09/21/kubernetes-1-25-use-secrets-while-expanding-csi-volumes-on-node-alpha/), the feature gate `CSINodeExpandSecret` is required.
You can enable online expansion for encrypted volumes by specifying the following [encryption parameters in the StorageClass](../../../advanced-resources/security/volume-encryption#setting-up-kubernetes-secrets-and-storageclasses):

- `csi.storage.k8s.io/node-expand-secret-name`
- `csi.storage.k8s.io/node-expand-secret-namespace`

If you cannot enable it but still prefer to do online expansion, you can:
1. Login the node host the encrypted volume is attached to.
Expand All @@ -134,7 +140,7 @@ Longhorn currently does not support fully automatic expansion of the filesystem
kubectl -n longhorn-system exec -it <the share manager pod> -- resize2fs /dev/longhorn/<volume name>
```

> **Important**:
> **Important**:
> Online expansion is possible only for `ext4` volumes. Attempts to manually expand `xfs` volumes with `xfs_growfs` may initially appear to be successful, but issues occur when the workload is scaled up and the volume is reattached. In particular, the pods become stuck in the `ContainerCreating` state, and the logs show an error message about attempts to mount the filesystem.

##### Offline
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,8 @@ For more information, see [cryptsetup(8)](https://man7.org/linux/man-pages/man8/
csi.storage.k8s.io/node-publish-secret-namespace: "longhorn-system"
csi.storage.k8s.io/node-stage-secret-name: "longhorn-crypto"
csi.storage.k8s.io/node-stage-secret-namespace: "longhorn-system"
csi.storage.k8s.io/node-expand-secret-name: "longhorn-crypto"
csi.storage.k8s.io/node-expand-secret-namespace: "longhorn-system"
```

- Example of a StorageClass with a volume-specific Secret:
Expand All @@ -85,6 +87,8 @@ For more information, see [cryptsetup(8)](https://man7.org/linux/man-pages/man8/
csi.storage.k8s.io/node-publish-secret-namespace: ${pvc.namespace}
csi.storage.k8s.io/node-stage-secret-name: ${pvc.name}
csi.storage.k8s.io/node-stage-secret-namespace: ${pvc.namespace}
csi.storage.k8s.io/node-expand-secret-name: ${pvc.name}
csi.storage.k8s.io/node-expand-secret-namespace: ${pvc.namespace}
```

# Using an Encrypted Volume
Expand All @@ -96,9 +100,14 @@ A newly-created PVC remains in the `Pending` state until the associated Secret i

# Filesystem Expansion

Longhorn supports [offline expansion](../../../nodes-and-volumes/volumes/expansion/#encrypted-volume) for encrypted volumes.
Longhorn supports [both online and offline expansion](../../../nodes-and-volumes/volumes/expansion/#encrypted-volume) for encrypted volumes.

StorageClass parameters are needed to enable online expansion:

- `csi.storage.k8s.io/node-expand-secret-name`
- `csi.storage.k8s.io/node-expand-secret-namespace`

# History
- Encryption of volumes in `Filesystem` mode available starting v1.2.0 ([#1859](https://github.com/longhorn/longhorn/issues/1859))

- Encryption of volumes in `Block` mode available starting v1.6.0 ([#4883](https://github.com/longhorn/longhorn/issues/4883))
- Encryption of volumes in `Block` mode available starting v1.6.0 ([#4883](https://github.com/longhorn/longhorn/issues/4883))
10 changes: 8 additions & 2 deletions content/docs/1.7.3/nodes-and-volumes/volumes/expansion.md
Original file line number Diff line number Diff line change
Expand Up @@ -116,7 +116,13 @@ If a volume is reverted to a snapshot with smaller size, the frontend of the vol

#### Encrypted volume

Due to [the upstream limitation](https://kubernetes.io/blog/2022/09/21/kubernetes-1-25-use-secrets-while-expanding-csi-volumes-on-node-alpha/), Longhorn cannot handle **online** expansion for encrypted volumes automatically unless you enable the feature gate `CSINodeExpandSecret`.
Longhorn support for online expansion depends on Kubernetes.
- Kubernetes natively supports [authenticated CSI storage resizing](https://kubernetes.io/blog/2023/12/15/csi-node-expand-secret-support-ga/) starting in v1.29.
- In [Kubernetes v1.25 to v1.28](https://kubernetes.io/blog/2022/09/21/kubernetes-1-25-use-secrets-while-expanding-csi-volumes-on-node-alpha/), the feature gate `CSINodeExpandSecret` is required.
You can enable online expansion for encrypted volumes by specifying the following [encryption parameters in the StorageClass](../../../advanced-resources/security/volume-encryption#setting-up-kubernetes-secrets-and-storageclasses):

- `csi.storage.k8s.io/node-expand-secret-name`
- `csi.storage.k8s.io/node-expand-secret-namespace`

If you cannot enable it but still prefer to do online expansion, you can:
1. Login the node host the encrypted volume is attached to.
Expand All @@ -134,7 +140,7 @@ Longhorn currently does not support fully automatic expansion of the filesystem
kubectl -n longhorn-system exec -it <the share manager pod> -- resize2fs /dev/longhorn/<volume name>
```

> **Important**:
> **Important**:
> Online expansion is possible only for `ext4` volumes. Attempts to manually expand `xfs` volumes with `xfs_growfs` may initially appear to be successful, but issues occur when the workload is scaled up and the volume is reattached. In particular, the pods become stuck in the `ContainerCreating` state, and the logs show an error message about attempts to mount the filesystem.

##### Offline
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,8 @@ For more information, see [cryptsetup(8)](https://man7.org/linux/man-pages/man8/
csi.storage.k8s.io/node-publish-secret-namespace: "longhorn-system"
csi.storage.k8s.io/node-stage-secret-name: "longhorn-crypto"
csi.storage.k8s.io/node-stage-secret-namespace: "longhorn-system"
csi.storage.k8s.io/node-expand-secret-name: "longhorn-crypto"
csi.storage.k8s.io/node-expand-secret-namespace: "longhorn-system"
```

- Example of a StorageClass with a volume-specific Secret:
Expand All @@ -85,6 +87,8 @@ For more information, see [cryptsetup(8)](https://man7.org/linux/man-pages/man8/
csi.storage.k8s.io/node-publish-secret-namespace: ${pvc.namespace}
csi.storage.k8s.io/node-stage-secret-name: ${pvc.name}
csi.storage.k8s.io/node-stage-secret-namespace: ${pvc.namespace}
csi.storage.k8s.io/node-expand-secret-name: ${pvc.name}
csi.storage.k8s.io/node-expand-secret-namespace: ${pvc.namespace}
```

# Using an Encrypted Volume
Expand All @@ -96,9 +100,14 @@ A newly-created PVC remains in the `Pending` state until the associated Secret i

# Filesystem Expansion

Longhorn supports [offline expansion](../../../nodes-and-volumes/volumes/expansion/#encrypted-volume) for encrypted volumes.
Longhorn supports [both online and offline expansion](../../../nodes-and-volumes/volumes/expansion/#encrypted-volume) for encrypted volumes.

StorageClass parameters are needed to enable online expansion:

- `csi.storage.k8s.io/node-expand-secret-name`
- `csi.storage.k8s.io/node-expand-secret-namespace`

# History
- Encryption of volumes in `Filesystem` mode available starting v1.2.0 ([#1859](https://github.com/longhorn/longhorn/issues/1859))

- Encryption of volumes in `Block` mode available starting v1.6.0 ([#4883](https://github.com/longhorn/longhorn/issues/4883))
- Encryption of volumes in `Block` mode available starting v1.6.0 ([#4883](https://github.com/longhorn/longhorn/issues/4883))
12 changes: 9 additions & 3 deletions content/docs/1.8.0/nodes-and-volumes/volumes/expansion.md
Original file line number Diff line number Diff line change
Expand Up @@ -116,7 +116,13 @@ If a volume is reverted to a snapshot with smaller size, the frontend of the vol

#### Encrypted volume

Due to [the upstream limitation](https://kubernetes.io/blog/2022/09/21/kubernetes-1-25-use-secrets-while-expanding-csi-volumes-on-node-alpha/), Longhorn cannot handle **online** expansion for encrypted volumes automatically unless you enable the feature gate `CSINodeExpandSecret`.
Longhorn support for online expansion depends on Kubernetes.
- Kubernetes natively supports [authenticated CSI storage resizing](https://kubernetes.io/blog/2023/12/15/csi-node-expand-secret-support-ga/) starting in v1.29.
- In [Kubernetes v1.25 to v1.28](https://kubernetes.io/blog/2022/09/21/kubernetes-1-25-use-secrets-while-expanding-csi-volumes-on-node-alpha/), the feature gate `CSINodeExpandSecret` is required.
You can enable online expansion for encrypted volumes by specifying the following [encryption parameters in the StorageClass](../../../advanced-resources/security/volume-encryption#setting-up-kubernetes-secrets-and-storageclasses):

- `csi.storage.k8s.io/node-expand-secret-name`
- `csi.storage.k8s.io/node-expand-secret-namespace`

If you cannot enable it but still prefer to do online expansion, you can:
1. Login the node host the encrypted volume is attached to.
Expand All @@ -131,9 +137,9 @@ From v1.8.0, Longhorn supports fully automatic online expansion of the filesyste
- CSI plugin
- Share Manager, which manages the NFS export

If you have upgraded from a previous version, the Share Manager pods (one for each RWX volume) are not upgraded automatically, to avoid disruption during the upgrade.
If you have upgraded from a previous version, the Share Manager pods (one for each RWX volume) are not upgraded automatically, to avoid disruption during the upgrade.

After growing the block device, the CSI layer sends a resize command to the Share Manager to grow the filesystem within the block device. With a down-rev share-manager, the command fails with an "unimplemented" error code and so no expansion happens. To get the right image before the expansion, the simplest thing is to force a restart of the pod. Identify the Share Manager pod of the RWX volume (typically named `share-manager-<volume name>`) and delete it:
After growing the block device, the CSI layer sends a resize command to the Share Manager to grow the filesystem within the block device. With a down-rev share-manager, the command fails with an "unimplemented" error code and so no expansion happens. To get the right image before the expansion, the simplest thing is to force a restart of the pod. Identify the Share Manager pod of the RWX volume (typically named `share-manager-<volume name>`) and delete it:

```shell
kubectl -n longhorn-system delete pod <the share manager pod>
Expand Down
Loading