Skip to content
This repository has been archived by the owner on Feb 12, 2024. It is now read-only.

feature/en289 Add a one-volume option for persistence using subPath #301

Merged
merged 1 commit into from
Jun 15, 2023
Merged

feature/en289 Add a one-volume option for persistence using subPath #301

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 6 additions & 3 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,8 +35,9 @@ The following items can be set via `--set` flag during installation or configure

#### Configure how to persist data

- **Disable**: The data does not survive the termination of a pod.
- **Persistent Volume Claim(default)**: A default `StorageClass` is needed in the Kubernetes cluster to dynamically provision the volumes. Specify another StorageClass in the `storageClass` or set `existingClaim` if you have already existing persistent volumes to use.
- **Disable(default)**: The data does not survive the termination of a pod.
banzo marked this conversation as resolved.
Show resolved Hide resolved
- **Persistent Volume Claim**: Enable persistence so that data survives termination of the pod. There is the choice of using one large persistent volume (using subPath) or seven separate persistent volumes for config, data, logs, repos, etc.
A default `StorageClass` is needed in the Kubernetes cluster to dynamically provision the volumes. Specify another StorageClass in the `persistence.storageClass` setting.

#### Configure authentication

Expand Down Expand Up @@ -184,14 +185,16 @@ The following table lists the configurable parameters of the nifi chart and the
| `persistence.enabled` | Use persistent volume to store data | `false` |
| `persistence.storageClass` | Storage class name of PVCs (use the default type if unset) | `nil` |
| `persistence.accessMode` | ReadWriteOnce or ReadOnly | `[ReadWriteOnce]` |
| `persistence.subPath.enabled` | Use only one persistent volume with subPath instead of seven separate persistent volumes | `false` |
| `persistence.subPath.name` | Name of the one persistent volume claim when using subPath | `data` |
| `persistence.subPath.size` | Size of the one persistent volume claim when using subPath | `36Gi` |
| `persistence.configStorage.size` | Size of persistent volume claim | `100Mi` |
| `persistence.authconfStorage.size` | Size of persistent volume claim | `100Mi` |
| `persistence.dataStorage.size` | Size of persistent volume claim | `1Gi` |
| `persistence.flowfileRepoStorage.size` | Size of persistent volume claim | `10Gi` |
| `persistence.contentRepoStorage.size` | Size of persistent volume claim | `10Gi` |
| `persistence.provenanceRepoStorage.size` | Size of persistent volume claim | `10Gi` |
| `persistence.logStorage.size` | Size of persistent volume claim | `5Gi` |
| `persistence.existingClaim` | Use an existing PVC to persist data | `nil` |
banzo marked this conversation as resolved.
Show resolved Hide resolved
| **jvmMemory** |
| `jvmMemory` | bootstrap jvm size | `2g` |
| **SideCar** |
Expand Down
106 changes: 85 additions & 21 deletions templates/statefulset.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -453,20 +453,55 @@ spec:
tcpSocket:
port: {{ .Values.properties.httpsPort }}
volumeMounts:
- name: "logs"
mountPath: /opt/nifi/nifi-current/logs
- name: "data"
mountPath: /opt/nifi/data
- name: "auth-conf"
mountPath: /opt/nifi/nifi-current/auth-conf/
- name: "config-data"
mountPath: /opt/nifi/nifi-current/config-data
- name: "flowfile-repository"
mountPath: /opt/nifi/flowfile_repository
- name: "content-repository"
mountPath: /opt/nifi/content_repository
- name: "provenance-repository"
mountPath: /opt/nifi/provenance_repository
- mountPath: /opt/nifi/nifi-current/logs
{{- if and .Values.persistence.enabled .Values.persistence.subPath.enabled }}
name: {{ .Values.persistence.subPath.name }}
subPath: logs
{{- else }}
name: "logs"
{{- end }}
- mountPath: /opt/nifi/data
{{- if and .Values.persistence.enabled .Values.persistence.subPath.enabled }}
name: {{ .Values.persistence.subPath.name }}
subPath: data
{{- else }}
name: "data"
{{- end }}
- mountPath: /opt/nifi/nifi-current/auth-conf/
{{- if and .Values.persistence.enabled .Values.persistence.subPath.enabled }}
name: {{ .Values.persistence.subPath.name }}
subPath: auth-conf
{{- else }}
name: "auth-conf"
{{- end }}
- mountPath: /opt/nifi/nifi-current/config-data
{{- if and .Values.persistence.enabled .Values.persistence.subPath.enabled }}
name: {{ .Values.persistence.subPath.name }}
subPath: config-data
{{- else }}
name: "config-data"
{{- end }}
- mountPath: /opt/nifi/flowfile_repository
{{- if and .Values.persistence.enabled .Values.persistence.subPath.enabled }}
name: {{ .Values.persistence.subPath.name }}
subPath: flowfile-repository
{{- else }}
name: "flowfile-repository"
{{- end }}
- mountPath: /opt/nifi/content_repository
{{- if and .Values.persistence.enabled .Values.persistence.subPath.enabled }}
name: {{ .Values.persistence.subPath.name }}
subPath: content-repository
{{- else }}
name: "content-repository"
{{- end }}
- mountPath: /opt/nifi/provenance_repository
{{- if and .Values.persistence.enabled .Values.persistence.subPath.enabled }}
name: {{ .Values.persistence.subPath.name }}
subPath: provenance-repository
{{- else }}
name: "provenance-repository"
{{- end }}
- name: "bootstrap-conf"
mountPath: /opt/nifi/nifi-current/conf/bootstrap.conf
subPath: "bootstrap.conf"
Expand Down Expand Up @@ -541,8 +576,13 @@ spec:
resources:
{{ toYaml .Values.logresources | indent 10 }}
volumeMounts:
- name: logs
mountPath: /var/log
- mountPath: /var/log
{{- if and .Values.persistence.enabled .Values.persistence.subPath.enabled }}
name: {{ .Values.persistence.subPath.name }}
subPath: logs
{{- else }}
name: "logs"
{{- end }}
- name: bootstrap-log
imagePullPolicy: {{ .Values.sidecar.imagePullPolicy | default "Always" | quote }}
image: "{{ .Values.sidecar.image }}:{{ .Values.sidecar.tag }}"
Expand All @@ -553,8 +593,13 @@ spec:
resources:
{{ toYaml .Values.logresources | indent 10 }}
volumeMounts:
- name: logs
mountPath: /var/log
- mountPath: /var/log
{{- if and .Values.persistence.enabled .Values.persistence.subPath.enabled }}
name: {{ .Values.persistence.subPath.name }}
subPath: logs
{{- else }}
name: "logs"
{{- end }}
- name: user-log
imagePullPolicy: {{ .Values.sidecar.imagePullPolicy | default "Always" | quote }}
image: "{{ .Values.sidecar.image }}:{{ .Values.sidecar.tag }}"
Expand All @@ -565,8 +610,13 @@ spec:
resources:
{{ toYaml .Values.logresources | indent 10 }}
volumeMounts:
- name: logs
mountPath: /var/log
- mountPath: /var/log
{{- if and .Values.persistence.enabled .Values.persistence.subPath.enabled }}
name: {{ .Values.persistence.subPath.name }}
subPath: logs
{{- else }}
name: "logs"
{{- end }}
{{- if .Values.certManager.enabled }}
- name: cert-manager
imagePullPolicy: {{ .Values.image.pullPolicy | quote }}
Expand Down Expand Up @@ -786,7 +836,21 @@ spec:
{{- if .Values.extraVolumes }}
{{ toYaml .Values.extraVolumes | indent 6 }}
{{- end }}
{{- if .Values.persistence.enabled }}
{{- if and .Values.persistence.enabled .Values.persistence.subPath.enabled }}
volumeClaimTemplates:
- metadata:
name: {{ .Values.persistence.subPath.name }}
spec:
accessModes:
{{- range .Values.persistence.accessModes }}
- {{ . | quote }}
{{- end }}
storageClassName: {{ .Values.persistence.storageClass | quote }}
resources:
requests:
storage: {{ .Values.persistence.subPath.size }}
{{- end }}
{{- if and .Values.persistence.enabled (not .Values.persistence.subPath.enabled) }}
volumeClaimTemplates:
- metadata:
name: logs
Expand Down
11 changes: 10 additions & 1 deletion values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -253,7 +253,16 @@ persistence:
# The default storage class is used if this variable is not set.

accessModes: [ReadWriteOnce]
## Storage Capacities for persistent volumes

## Use subPath and have 1 persistent volume instead of 7 volumes - use when your k8s nodes have limited volume slots, to limit waste of space,
## or your available volume sizes are quite large
# The one disk will have a directory folder for each volumeMount, but this is hidden. Run 'mount' to view each mount.
subPath:
enabled: false
name: data
size: 30Gi

## Storage Capacities for persistent volumes (these are ignored if using one volume with subPath)
configStorage:
size: 100Mi
authconfStorage:
Expand Down