Persistence queue

CHANGELOG.md

@@ -10,6 +10,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 
 ### Added
 
+- Configuration of persistent buffering for agent [861](https://github.com/signalfx/splunk-otel-collector-chart/pull/861)
 - Add option to disable Openshift SecurityContextConstraint resource [#843](https://github.com/signalfx/splunk-otel-collector-chart/pull/843)
 
 ## [0.83.0] - 2023-08-18

docs/advanced-configuration.md

@@ -720,3 +720,63 @@ rbac:
 ```
 helm install my-splunk-otel-collector -f my_values.yaml splunk-otel-collector-chart/splunk-otel-collector
 ```
+
+## Data Persistence
+
+By default, without any configuration, data is queued in memory only. When data cannot be sent it is retried a few times (up to 5 mins. by default) and then dropped.
+
+If for any reason, the collector is restarted in this period, the queued data will be gone.
+
+If you want the queue to be persisted on disk across collector restarts, set `splunkPlatform.sendingQueue.persistentQueue.enabled` to enable support for logs, metrics and traces.
+
+By default, data is persisted in `/var/addon/splunk/exporter_queue` directory.
+Override this behaviour by setting `splunkPlatform.sendingQueue.persistentQueue.storagePath` option.
+
+Check [Data Persistence in the OpenTelemetry Collector
+](https://community.splunk.com/t5/Community-Blog/Data-Persistence-in-the-OpenTelemetry-Collector/ba-p/624583) for detailed explantion.
+
+Note: Data Persistence is only applicable for agent daemonset.
+
+Use following in values.yaml to disable data persistense for logs or metrics or traces:
+
+```yaml
+agent:
+  config:
+    exporters:
+       splunk_hec/platform_logs:
+         sending_queue:
+           storage: null
+```
+or
+```yaml
+agent:
+  config:
+    exporters:
+       splunk_hec/platform_metrics:
+         sending_queue:
+           storage: null
+```
+or
+```yaml
+agent:
+  config:
+    exporters:
+       splunk_hec/platform_traces:
+         sending_queue:
+           storage: null
+```
+
+### Support for persistent queue
+
+* `GKE/Autopilot` and `EKS/Fargate` support
+  * Both of the above distributions doesn't allow volume mounts, as they are kind of `serverless` and we don't manage the underlying infrastructure.
+  * Persistent buffering is not supported for them, as directory needs to be mounted via `hostPath`.
+  * Refer [aws/fargate](https://docs.aws.amazon.com/eks/latest/userguide/fargate.html) and [gke/autopilot](https://cloud.google.com/kubernetes-engine/docs/concepts/autopilot-security#built-in-security).
+* Gateway support
+  * The filestorage extention acquires an exclusive lock for the queue directory.
+  * It is not possible to run the persistent buffering if there are multiple replicas of a pod and `gateway` runs 3 replicas by default.
+  * Even if support is somehow provided, only one of the pods will be able to acquire the lock and run, while the others will be blocked and unable to operate.
+* Cluster Receiver support
+  * Cluster receiver is a 1-replica deployment of Open-temlemetry collector.
+  * As any available node can be selected by the Kubernetes control plane to run the cluster receiver pod (unless we explicitly specify the `clusterReceiver.nodeSelector` to pin the pod to a specific node), `hostPath` or `local` volume mounts wouldn't work for such envrionments.
+  * Data Persistence is currently not applicable to the k8s cluster metrics and k8s events.

examples/disable-persistence-queue-traces/README.md

@@ -0,0 +1,7 @@
+# Example of chart configuration
+
+## Disable Persistent Queue for traces only
+
+This example will show how to disable data persistence for traces data.
+
+Refer to: https://github.com/signalfx/splunk-otel-collector-chart/blob/main/docs/advanced-configuration.md#data-persistence

examples/disable-persistence-queue-traces/disable-persistence-queue-traces-values.yaml

@@ -0,0 +1,21 @@
+clusterName: CHANGEME
+splunkPlatform:
+  endpoint: CHANGEME
+  token: CHANGEME
+  index: CHANGEME
+  metricsIndex: CHANGEME
+  metricsEnabled: true
+  tracesEnabled: true
+  logsEnabled: true
+  sendingQueue:
+    persistentQueue:
+      enabled: true
+
+logsEngine: otel
+
+agent:
+  config:
+    exporters:
+       splunk_hec/platform_traces:
+         sending_queue:
+           storage: null

examples/disable-persistence-queue-traces/rendered_manifests/clusterRole.yaml

@@ -0,0 +1,83 @@
+---
+# Source: splunk-otel-collector/templates/clusterRole.yaml
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: default-splunk-otel-collector
+  labels:
+    app.kubernetes.io/name: splunk-otel-collector
+    helm.sh/chart: splunk-otel-collector-0.83.0
+    app.kubernetes.io/managed-by: Helm
+    app.kubernetes.io/instance: default
+    app.kubernetes.io/version: "0.83.0"
+    app: splunk-otel-collector
+    chart: splunk-otel-collector-0.83.0
+    release: default
+    heritage: Helm
+rules:
+- apiGroups:
+  - ""
+  resources:
+  - events
+  - namespaces
+  - namespaces/status
+  - nodes
+  - nodes/spec
+  - nodes/stats
+  - nodes/proxy
+  - pods
+  - pods/status
+  - persistentvolumeclaims
+  - persistentvolumes
+  - replicationcontrollers
+  - replicationcontrollers/status
+  - resourcequotas
+  - services
+  verbs:
+  - get
+  - list
+  - watch
+- apiGroups:
+  - apps
+  resources:
+  - daemonsets
+  - deployments
+  - replicasets
+  - statefulsets
+  verbs:
+  - get
+  - list
+  - watch
+- apiGroups:
+  - extensions
+  resources:
+  - daemonsets
+  - deployments
+  - replicasets
+  verbs:
+  - get
+  - list
+  - watch
+- apiGroups:
+  - batch
+  resources:
+  - jobs
+  - cronjobs
+  verbs:
+  - get
+  - list
+  - watch
+- apiGroups:
+    - autoscaling
+  resources:
+    - horizontalpodautoscalers
+  verbs:
+    - get
+    - list
+    - watch
+- nonResourceURLs:
+  - /metrics
+  verbs:
+  - get
+  - list
+  - watch

examples/disable-persistence-queue-traces/rendered_manifests/clusterRoleBinding.yaml

@@ -0,0 +1,24 @@
+---
+# Source: splunk-otel-collector/templates/clusterRoleBinding.yaml
+kind: ClusterRoleBinding
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: default-splunk-otel-collector
+  labels:
+    app.kubernetes.io/name: splunk-otel-collector
+    helm.sh/chart: splunk-otel-collector-0.83.0
+    app.kubernetes.io/managed-by: Helm
+    app.kubernetes.io/instance: default
+    app.kubernetes.io/version: "0.83.0"
+    app: splunk-otel-collector
+    chart: splunk-otel-collector-0.83.0
+    release: default
+    heritage: Helm
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: default-splunk-otel-collector
+subjects:
+- kind: ServiceAccount
+  name: default-splunk-otel-collector
+  namespace: default