The first app run in kubernetes pod throws runtime error

[falkolab]

I found a strange behavior within k8s pod, the app throws error but the next try executes app without runtime error

/usr/sbin # /usr/sbin/timescaledb-event-streamer
CDC (Chance Data Capture) for TimescaleDB Hypertable
timescaledb-event-streamer version 0.11.0 (git revision unknown; branch unknown)
Using configuration file from environment variable
Loading configuration file: /etc/db_event_streamer/config.yml
runtime error: invalid memory address or nil pointer dereference

I didn't found any additional info in app logs

[noctarius]

Hey @falkolab! Thanks for bug report. Is it possible to share the deployment descriptor and the configuration (remember to remove the password) for me to try and test it myself?

There isn't an immediate idea what it could be. Especially if it runs the second time.

[falkolab]

This is exported deployment (there are additional properties args and command under container section for debugging purpose)

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "10"
  creationTimestamp: "2023-11-07T17:31:27Z"
  generation: 38
  name: timescaledb-event-streamer
  namespace: default
  resourceVersion: "196048"
  uid: 49dc819d-85c8-40c4-9d9f-bbdf2b5c39fa
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: timescaledb-event-streamer
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      annotations:
        kubectl.kubernetes.io/restartedAt: "2023-11-08T14:50:13+03:00"
      creationTimestamp: null
      labels:
        app: timescaledb-event-streamer
    spec:
      containers:
      - args:
        - while true; do sleep 30; done;
        command:
        - /bin/sh
        - -c
        - --
        env:
        - name: POSTGRESQL_CONNECTION
          valueFrom:
            configMapKeyRef:
              key: postgresql_connection
              name: f-general
        - name: SINK_NATS_ADDRESS
          valueFrom:
            configMapKeyRef:
              key: nats_address
              name: f-general
        - name: POSTGRESQL_PASSWORD
          valueFrom:
            secretKeyRef:
              key: postgres_password
              name: f-general
        - name: SINK_NATS_USERINFO_USERNAME
          valueFrom:
            secretKeyRef:
              key: nats_user
              name: f-general
        - name: SINK_NATS_USERINFO_PASSWORD
          valueFrom:
            secretKeyRef:
              key: nats_password
              name: f-general
        - name: TIMESCALEDB_EVENT_STREAMER_CONFIG
          value: /etc/db_event_streamer/config.yml
        image: ghcr.io/noctarius/timescaledb-event-streamer:latest
        imagePullPolicy: Always
        name: timescaledb-event-streamer
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /etc/db_event_streamer
          name: config-volume
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - configMap:
          defaultMode: 420
          name: timescaledb-event-streamer-config
        name: config-volume
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: "2023-11-08T11:42:18Z"
    lastUpdateTime: "2023-11-08T13:17:31Z"
    message: ReplicaSet "timescaledb-event-streamer-f95df576c" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  - lastTransitionTime: "2023-11-08T13:39:23Z"
    lastUpdateTime: "2023-11-08T13:39:23Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  observedGeneration: 38
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1

configmap:

apiVersion: v1
data:
  config.yml: |
    postgresql:
    #  connection: 'postgres://repl_user@localhost:5432/postgres'
    #  password: '...'
      publication:
        name: 'db_event_streamer'

      tables:
        excludes:
          - 'pg_catalog.*'
        includes:
          - 'public.*'
      events:
        read: true
        insert: true
        update: true
        delete: true
        truncate: true
        message: true

    stateStorage:
      type: file
      file:
        path: '/var/lib/db_event_streamer/statestorage.dat'

    sink:
    #  filters:
    #    filterName:
    #      condition: 'value.op == "u" && value.before.id == 2'
    #      default: true
      tombstone: true
    #  type: 'stdout'
      type: 'nats'
      nats:
        authorization: 'userinfo'
        userInfo:
          username: 'publisher'
          password: '...'

    topic:
      namingStrategy:
        type: 'debezium'
      prefix: 'timescaledb'

    timescaledb:
      hypertables:
        excludes:
          - 'pg_catalog.*'
        includes:
          - 'public.test'
      events:
        read: true
        insert: true
        update: true
        delete: true
        truncate: true
        message: false #deprecated: see postgresql\events\message
        compression: false
        decompression: false

    logging:
      level: 'info'
      outputs:
        console:
          enabled: true
        file:
          enabled: false
          path: '/var/logs/db_event_streamer'
          rotate: true
          maxSize: '5MB'
          maxDuration: 600 #seconds
          compress: true
      loggers:
        LogicalReplicationResolver:
          level: 'debug'
          outputs:
            console:
              enabled: false
kind: ConfigMap
metadata:
  creationTimestamp: "2023-11-07T17:18:10Z"
  name: timescaledb-event-streamer-config
  namespace: default
  resourceVersion: "196010"
  uid: d5de049c-9440-4ecd-9aa5-d62470bc0d13

I enter to the container like:
kubectl exec --stdin --tty timescaledb-event-streamer-*** -- /bin/sh

and executing the app from /usr/sbin

This is an output of printenv command:

TIMESCALEDB_EVENT_STREAMER_CONFIG=/etc/db_event_streamer/config.yml
KUBERNETES_PORT=tcp://10.43.0.1:443
KUBERNETES_SERVICE_PORT=443
SINK_NATS_ADDRESS=nats://***:14222
HOSTNAME=timescaledb-event-streamer-f95df576c-2nsss
NATS_SERVICE_PORT_NATS=4222
SHLVL=1
HOME=/root
SINK_NATS_USERINFO_PASSWORD=***
NATS_PORT_4222_TCP_ADDR=10.43.66.40
NATS_PORT_4222_TCP_PORT=4222
TERM=xterm
KUBERNETES_PORT_443_TCP_ADDR=10.43.0.1
NATS_PORT_4222_TCP_PROTO=tcp
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
POSTGRESQL_PASSWORD=***
NATS_SERVICE_HOST=10.43.66.40
KUBERNETES_PORT_443_TCP_PORT=443
KUBERNETES_PORT_443_TCP_PROTO=tcp
SINK_NATS_USERINFO_USERNAME=***
NATS_PORT_4222_TCP=tcp://10.43.66.40:4222
NATS_SERVICE_PORT=4222
NATS_PORT=tcp://10.43.66.40:4222
POSTGRESQL_CONNECTION=postgres://***@***:5432/***
KUBERNETES_PORT_443_TCP=tcp://10.43.0.1:443
KUBERNETES_SERVICE_PORT_HTTPS=443
KUBERNETES_SERVICE_HOST=10.43.0.1
PWD=/usr/sbin

You can safely remove env mappings from deployment and specify them within the container. I think it's overhead to place here secrets and cm.

If you point me to some direction to test, I can do it in the nearest couple days on the current pod.

[noctarius]

Let me try a bit tomorrow. If I can reproduce it, it's probably easy to fix since it's most likely a missing directory or something (which could make sense) 🤷‍♂️

[falkolab]

I found a solution. You are right the reason is in absence mounted pvc for the storing : stateStorage.file.path

pvc.yaml for single pod usage

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: timescaledbevent-streamer-claim
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 100M

and mounting in deployment like:

        volumeMounts:
        - name: pvc-volume
          mountPath: /var/lib/db_event_streamer
      volumes:
      - name: pvc-volume
        persistentVolumeClaim:
          claimName: timescaledbevent-streamer-claim

You probably need to check the folder for existence or thomething.
Could you explain a little how to properly calculate volume size?

[noctarius]

Perfect! Good to know. That was my guess as well. Haven't had time to have a look yet.

I'll keep it open to remember to add a meaningful check and a better error message.

Thanks for testing and the help 🙏

[noctarius]

Could you explain a little how to properly calculate volume size?

The file's fairly small. It just keeps a few kilobytes state per TimescaleDB chunk, plus some overall data. It basically stores what the last known xid (transaction id) and the last index values were. Nothing large.

I guess if you use a 100 megs you'll be golden into all eternity 🤣