Error while upgrading fluentd using helm chart in k8s 1.31.7

[monduofficial]

What is a problem?

Hi All,

I have been trying to add a new mount to the fluentd pod so i had to upgrade the helm image inorder to fetch the change i need to trigger the pipeline .

helm repo add fluent https://fluent.github.io/helm-charts
envsubst < kubernetes/logging/fluentd/helm_values.yaml | helm upgrade --install fluentd fluent/fluentd -n logging -f -

Any help or clue would be helpfull

Regards
Mondu

Describe the configuration of Fluentd

nameOverride: ""
fullnameOverride: ""

DaemonSet, Deployment or StatefulSet

kind: "DaemonSet"

# Only applicable for Deployment or StatefulSet

replicaCount: 1

image:
repository: "fluent/fluentd-kubernetes-daemonset"
pullPolicy: "IfNotPresent"
tag: "v1.16.5-debian-kafka-1.0"

Optional array of imagePullSecrets containing private registry credentials

Ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/

imagePullSecrets: []

serviceAccount:
create: true
annotations: {}
name: null

rbac:
create: true

Configure podsecuritypolicy

Ref: https://kubernetes.io/docs/concepts/policy/pod-security-policy/

podSecurityPolicy:
enabled: false
annotations: {}

Security Context policies for controller pods

See https://kubernetes.io/docs/tasks/administer-cluster/sysctl-cluster/ for

notes on enabling and using sysctls

podSecurityContext: {}

seLinuxOptions:

type: "spc_t"

securityContext: {}

capabilities:

drop:

- ALL

readOnlyRootFilesystem: true

runAsNonRoot: true

runAsUser: 1000

#runAsUser: 0
#runAsGroup: 0
#fsGroup: 0

Configure the livecycle

Ref: https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/

lifecycle: {}

preStop:

exec:

command: ["/bin/sh", "-c", "sleep 20"]

Configure the livenessProbe

Ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/

#commenting it as there is a bug in the fluentd dedployment
livenessProbe:
httpGet:
path: /metrics
port: metrics
#initialDelaySeconds: 0
#periodSeconds: 10
#timeoutSeconds: 1
#successThreshold: 1
#failureThreshold: 3

Configure the readinessProbe

Ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/

#commenting it as there is a bug in the fluentd dedployment
readinessProbe:
httpGet:
path: /metrics
port: metrics
#initialDelaySeconds: 0
#periodSeconds: 10
#timeoutSeconds: 1
#successThreshold: 1
#failureThreshold: 3
resources: {}

requests:

cpu: 10m

memory: 128Mi

limits:

memory: 128Mi

only available if kind is Deployment

autoscaling:
enabled: false
minReplicas: 1
maxReplicas: 100
targetCPUUtilizationPercentage: 80

targetMemoryUtilizationPercentage: 80

see https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/#autoscaling-on-multiple-metrics-and-custom-metrics

customRules: []
# - type: Pods
# pods:
# metric:
# name: packets-per-second
# target:
# type: AverageValue
# averageValue: 1k

see https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#support-for-configurable-scaling-behavior

behavior:

scaleDown:

policies:

- type: Pods

value: 4

periodSeconds: 60

- type: Percent

value: 10

periodSeconds: 60

priorityClassName: "system-node-critical"

nodeSelector: {}

Node tolerations for server scheduling to nodes with taints

Ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/

tolerations: []

- key: null

operator: Exists

effect: "NoSchedule"

Affinity and anti-affinity

Ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity

affinity: {}

Annotations to be added to fluentd DaemonSet/Deployment

annotations: {}

Labels to be added to fluentd DaemonSet/Deployment

labels: {}

Annotations to be added to fluentd pods

podAnnotations: {}

Labels to be added to fluentd pods

podLabels: {}

How long (in seconds) a pods needs to be stable before progressing the deployment

minReadySeconds:

How long (in seconds) a pod may take to exit (useful with lifecycle hooks to ensure lb deregistration is done)

terminationGracePeriodSeconds:

Deployment strategy / DaemonSet updateStrategy

updateStrategy: {}

type: RollingUpdate

rollingUpdate:

maxUnavailable: 1

Additional environment variables to set for fluentd pods

env:

• name: "FLUENTD_CONF"
  value: "../../../etc/fluent/fluent.conf"
• name: K8S_NODE_NAME
  valueFrom:
  fieldRef:
  fieldPath: spec.nodeName
  #- name: FLUENT_UID

value: "0"

- name: FLUENT_ELASTICSEARCH_HOST

value: "elasticsearch-master"

- name: FLUENT_ELASTICSEARCH_PORT

value: "9200"

envFrom: []

volumes:

• name: varlog
  hostPath:
  path: /var/log
• name: varlibdockercontainers
  hostPath:
  path: /var/lib/docker/containers
• name: etcfluentd-main
  configMap:
  name: fluentd-main
  defaultMode: 0777
• name: etcfluentd-config
  configMap:
  name: fluentd-config
  defaultMode: 0777
• name: nfs-direct-volume
  persistentVolumeClaim:
  claimName: nfs-direct-pvc-for-logs
• name: nfs-direct-volume-media
  persistentVolumeClaim:
  claimName: nfs-direct-pvc-media # New PVC for /opt/backend-files/shared/media

volumeMounts:

• name: varlog
  mountPath: /var/log
• name: varlibdockercontainers
  mountPath: /var/lib/docker/containers
  readOnly: true
• name: etcfluentd-main
  mountPath: /etc/fluent
• name: etcfluentd-config
  mountPath: /etc/fluent/config.d/
• name: nfs-direct-volume
  mountPath: /srv/
• name: nfs-direct-volume-media
  mountPath: /srv/media # New mount for the additional PVC

resources:
requests:
cpu: 200m
memory: 256Mi
limits:
cpu: 500m
memory: 512Mi

Only available if kind is StatefulSet

Fluentd persistence

persistence:
enabled: false
storageClass: ""
accessMode: ReadWriteOnce
size: 10Gi

Fluentd service

service:
type: "ClusterIP"
annotations: {}
ports: []

- name: "forwarder"

protocol: TCP

containerPort: 24224

Prometheus Monitoring

metrics:
serviceMonitor:
enabled: true
additionalLabels:
release: fluentd
prometheus: "true"
namespace: "monitoring"
namespace: "monitoring"
namespaceSelector: {}
## metric relabel configs to apply to samples before ingestion.
##
metricRelabelings:
- sourceLabels: [name]
separator: ;
regex: ^fluentd_output_status_buffer_(oldest|newest)_.+
replacement: $1
action: drop
# relabel configs to apply to samples after ingestion.
#
relabelings:
- sourceLabels: [__meta_kubernetes_pod_node_name]
separator: ;
regex: ^(.*)$
targetLabel: nodename
replacement: $1
action: replace

jobLabel: fluentd
scrapeInterval: 30s
scrapeTimeout: 5s
honorLabels: true

prometheusRule:
enabled: true
additionalLabels: {}
namespace: "monitoring"
rules:
- alert: FluentdDown
expr: up{job="fluentd"} == 0
for: 5m
labels:
context: fluentd
severity: warning
annotations:
summary: "Fluentd Down"
description: "{{ $labels.pod }} on {{ $labels.nodename }} is down"
- alert: FluentdScrapeMissing
expr: absent(up{job="fluentd"} == 1)
for: 15m
labels:
context: fluentd
severity: warning
annotations:
summary: "Fluentd Scrape Missing"
description: "Fluentd instance has disappeared from Prometheus target discovery"

Grafana Monitoring Dashboard

dashboards:
enabled: "true"
namespace: "monitoring"
labels:
grafana_dashboard: '"1"'

Fluentd list of plugins to install

plugins:

• fluent-plugin-grafana-loki
• fluent-plugin-prometheus
• fluent-plugin-kubernetes_metadata_filter

- fluent-plugin-out-http

Add fluentd config files from K8s configMaps

configMapConfigs:

• fluentd-prometheus-conf

- fluentd-systemd-conf

Fluentd configurations:

fileConfigs:
01_sources.conf: |-

<source>
  @type tail
  @id in_tail_container_logs
  path /var/log/containers/test-*.log
  tag test-backend.*
  @label @test-backend
  read_from_head true
  <parse>
    @type multi_format
    <pattern>
      format json
      time_key time
      time_type string
      time_format "%Y-%m-%dT%H:%M:%S.%NZ"
      keep_time_key false
    </pattern>
    <pattern>
      format regexp
      expression /^(?<time>.+) (?<stream>stdout|stderr)( (.))? (?<log>.*)$/
      time_format '%Y-%m-%dT%H:%M:%S.%NZ'
      keep_time_key false
    </pattern>
  </parse>
  emit_unmatched_lines true
</source>

<source>
  @type tail
  path /var/log/containers/*.log
  tag all.*
  @label @ALL
  read_from_head true
  <parse>
    @type multi_format
    <pattern>
      format json
      time_key time
      time_type string
      time_format "%Y-%m-%dT%H:%M:%S.%NZ"
      keep_time_key false
    </pattern>
    <pattern>
      format regexp
      expression /^(?<time>.+) (?<stream>stdout|stderr)( (.))? (?<log>.*)$/
      time_format '%Y-%m-%dT%H:%M:%S.%NZ'
      keep_time_key false
    </pattern>
  </parse>
  emit_unmatched_lines true
</source>

02_filters.conf: |-
<label @test-backend>

  <match **>
    @type relabel
    @label @DISPATCH
  </match>


    
</label>

<label @ALL>
<filter **>
    @type kubernetes_metadata
    @id filter_kube_metadata
    skip_labels false
    skip_container_metadata false
    skip_namespace_metadata true
    skip_master_url true
 </filter>
<filter **>
  @type grep
  <exclude>
  key log
  pattern /security_event/
</exclude>
</filter>
  <match **>
  @type loki
  url "http://$INFRA_METRICS_SERVER_IP:3100"
  
  flush_interval 10s
  flush_at_shutdown true
  buffer_chunk_limit 1m    
  extract_kubernetes_labels true
  remove_keys kubernetes
  <label>
    container $.kubernetes.container
   </label>
   </match>

</label>

<match all.**>

@type loki

url "http://$INFRA_METRICS_SERVER_IP:3100"

flush_interval 10s

flush_at_shutdown true

buffer_chunk_limit 1m

03_dispatch.conf: |-
<label @dispatch>
<filter **>
@type prometheus

name fluentd_input_status_num_records_total
type counter
desc The total number of incoming records

tag ${tag}
hostname ${hostname}

  <match **>
    @type relabel
    @label @OUTPUT
  </match>
</label>

04_outputs.conf: |-

<label @OUTPUT>
<filter **>
  @type grep
  <regexp>
  key log
  pattern /security_event/
  </regexp>
</filter>
  <match **>
     @type file
     path /srv/log
     compress gzip
    <buffer>
      @type file
      timekey 15m
      timekey_wait 10m
      path /buffer
      retry_type exponential_backoff
      
    </buffer>
  </match>
</label>

Describe the logs of Fluentd

"fluent" already exists with the same configuration, skipping
Error: UPGRADE FAILED: current release manifest contains removed kubernetes api(s) for this kubernetes version and it is therefore unable to build the kubernetes objects for performing the diff. error from kubernetes: unable to recognize "": no matches for kind "PodSecurityPolicy" in version "policy/v1beta1"

##[error]Bash exited with code '1'.
Finishing: Bash

Environment

- Fluentd version:fluentd 1.14.5
- TD Agent version: 
- Fluent Package version: fluent https://fluent.github.io/helm-charts   
- Docker image (tag): fluent/fluentd-kubernetes-daemonset/v1.16.5-debian-kafka-1.0
- Operating system: k8s v1.31.7 - Ubuntu 22.04.5 LTS
- Kernel version:  5.15.0-144-generic