Skip to content

Commit

Permalink
Increase the default PVC size for logstash (#7540)
Browse files Browse the repository at this point in the history
Increase the default PVC size for Logstash

---------

Co-authored-by: Rob Bavey <rob.bavey@elastic.co>
Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
Co-authored-by: Michael Morello <michael.morello@gmail.com>
  • Loading branch information
4 people authored Feb 9, 2024
1 parent 309ab6d commit b13e71f
Show file tree
Hide file tree
Showing 3 changed files with 24 additions and 13 deletions.
15 changes: 13 additions & 2 deletions docs/orchestrating-elastic-stack-applications/logstash.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -301,7 +301,11 @@ WARNING: Volume support for Logstash is a breaking change to earlier versions of
[discrete]
== Specifying the volume claim settings

By default, a PersistentVolume called `logstash-data` is created, that maps to `/usr/share/logstash/data` for persistent storage, typically used for storage from plugins. The `logstash-data` volume claim is, by default, a small (1Gi) volume, using the standard StorageClass of your Kubernetes cluster, but can be overridden by adding a `spec.volumeClaimTemplate` section named `logstash-data`.
A PersistentVolume called `logstash-data` is created by default.
It maps to `/usr/share/logstash/data` for persistent storage, which is typically used for storage from plugins.

By default, the `logstash-data` volume claim is a `1.5Gi` volume, using the standard StorageClass of your Kubernetes cluster.
You can override the default by adding a `spec.volumeClaimTemplate` section named `logstash-data`.

For production workloads, you should define your own volume claim template with the desired storage capacity and (optionally) the Kubernetes link:https://kubernetes.io/docs/concepts/storage/storage-classes/[storage class] to associate with the persistent volume. To override this volume claim for `data` usages, the name of this volume claim must be `logstash-data`.

Expand All @@ -326,8 +330,15 @@ spec:
storage: 2Gi
----

The default volume size will likely be insufficient for production workloads, especially when you are using:

* the persistent queue (PQ) feature
* dead letter queues (DLQ), or
* {ls} plugins that make heavy use of temporary storage.

Increase the storage capacity, or consider creating separate volumes for these use cases.

Separate storage, for example for Logstash configurations using persistent queues (PQ) and/or dead letter queues (DLQ), can be added by including an additional `spec.volumeClaimTemplate` along with a corresponding `spec.podTemplate.spec.containers.volumeMount` for each requested volume.
You can add separate storage by including an additional `spec.volumeClaimTemplate` along with a corresponding `spec.podTemplate.spec.containers.volumeMount` for each requested volume.

This example shows how to setup separate storage for a PQ:

Expand Down
18 changes: 9 additions & 9 deletions pkg/controller/logstash/logstash_controller_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -191,7 +191,7 @@ func TestReconcileLogstash_Reconcile(t *testing.T) {
StorageClassName: ptr.To[string](sampleStorageClass.Name),
Resources: corev1.VolumeResourceRequirements{
Requests: corev1.ResourceList{
corev1.ResourceStorage: resource.MustParse("1Gi"),
corev1.ResourceStorage: resource.MustParse("1.5Gi"),
},
},
},
Expand Down Expand Up @@ -303,7 +303,7 @@ func TestReconcileLogstash_Reconcile(t *testing.T) {
StorageClassName: ptr.To[string](sampleStorageClass.Name),
Resources: corev1.VolumeResourceRequirements{
Requests: corev1.ResourceList{
corev1.ResourceStorage: resource.MustParse("1Gi"),
corev1.ResourceStorage: resource.MustParse("1.5Gi"),
},
},
},
Expand Down Expand Up @@ -432,7 +432,7 @@ func TestReconcileLogstash_Reconcile(t *testing.T) {
StorageClassName: ptr.To[string](sampleStorageClass.Name),
Resources: corev1.VolumeResourceRequirements{
Requests: corev1.ResourceList{
corev1.ResourceStorage: resource.MustParse("1Gi"),
corev1.ResourceStorage: resource.MustParse("1.5Gi"),
},
},
},
Expand Down Expand Up @@ -550,7 +550,7 @@ func TestReconcileLogstash_Reconcile(t *testing.T) {
StorageClassName: ptr.To[string](sampleStorageClass.Name),
Resources: corev1.VolumeResourceRequirements{
Requests: corev1.ResourceList{
corev1.ResourceStorage: resource.MustParse("1Gi"),
corev1.ResourceStorage: resource.MustParse("1.5Gi"),
},
},
},
Expand Down Expand Up @@ -650,15 +650,15 @@ func TestReconcileLogstash_Resize(t *testing.T) {
}{
{
name: "Cannot increase storage with fixed storage class",
initialCapacity: "1Gi",
initialCapacity: "1.5Gi",
desiredCapacity: "3Gi",
storageClass: fixedStorageClass,
wantErr: true,
},
{
name: "Cannot decrease storage with resizable storage class",
initialCapacity: "3Gi",
desiredCapacity: "1Gi",
desiredCapacity: "1.5Gi",
storageClass: resizableStorageClass,
wantErr: true,
},
Expand All @@ -671,14 +671,14 @@ func TestReconcileLogstash_Resize(t *testing.T) {
},
{
name: "Nothing happens when keeping the storage the same with resizable storage class",
initialCapacity: "1Gi",
desiredCapacity: "1Gi",
initialCapacity: "1.5Gi",
desiredCapacity: "1.5Gi",
storageClass: resizableStorageClass,
wantErr: false,
},
{
name: "Can successfully resize the storage with resizable storage class",
initialCapacity: "1Gi",
initialCapacity: "1.5Gi",
desiredCapacity: "3Gi",
storageClass: resizableStorageClass,
extraVerify: func(r ReconcileLogstash, desiredCapacity string) (reconcile.Result, error) {
Expand Down
4 changes: 2 additions & 2 deletions pkg/controller/logstash/volume/defaults.go
Original file line number Diff line number Diff line change
Expand Up @@ -21,10 +21,10 @@ var (
ContainerMountPath: ConfigMountPath,
}

DefaultPersistentVolumeSize = resource.MustParse("1Gi")
DefaultPersistentVolumeSize = resource.MustParse("1.5Gi")

// DefaultDataVolumeClaim is the default data volume claim for Logstash pods.
// We default to a 1GB persistent volume, using the default storage class.
// We default to a 1.5Gi persistent volume, using the default storage class.
DefaultDataVolumeClaim = corev1.PersistentVolumeClaim{
ObjectMeta: metav1.ObjectMeta{
Name: LogstashDataVolumeName,
Expand Down

0 comments on commit b13e71f

Please sign in to comment.