Backups that don't trigger any jobs never complete and aren't cleaned up

[haasad]

Description

Backups that don't trigger any jobs are not marked as finished. Because they aren't marked as finished they are never cleaned up and accumulate endlessly on the cluster.

Additional Context

We automatically distribute a "default" k8up schedule to new namespaces on some of our clusters. If there's no PVCs (or BACKUP_SKIP_WITHOUT_ANNOTATION is true) and no other means of triggering a backup job (pods with k8up.io/backupcommand, prebackuppods etc.) the backup objects are created and never cleaned up. They accumulate endlessly.

Logs

No response

Expected Behavior

Backup objects that don't trigger any jobs should be marked as finished and cleaned up like regular finished backups.

Steps To Reproduce

1. create a new empty namespace
2. create a schedule:

apiVersion: k8up.io/v1
kind: Schedule
metadata:
  name: noop-schedule
spec:
  backup:
    failedJobsHistoryLimit: 2
    resources: {}
    schedule: "* * * * *"
    successfulJobsHistoryLimit: 2

3. watch the backup objects accumulate:

kubectl get backups.k8up.io --sort-by .metadata.creationTimestamp
NAME                         SCHEDULE REF    COMPLETION   PREBACKUP              AGE
noop-schedule-backup-pr7z4   noop-schedule                NoPreBackupPodsFound   27m
noop-schedule-backup-4l9wt   noop-schedule                NoPreBackupPodsFound   26m
noop-schedule-backup-262bq   noop-schedule                NoPreBackupPodsFound   25m
<...>
noop-schedule-backup-9f7b6   noop-schedule                NoPreBackupPodsFound   2m5s
noop-schedule-backup-tvszx   noop-schedule                NoPreBackupPodsFound   65s
noop-schedule-backup-8kkt4   noop-schedule                NoPreBackupPodsFound   5s

Version of K8up

v2.11.1

Version of Kubernetes

v1.28

Distribution of Kubernetes

OpenShift 4.15

[TheBigLee]

Hi @haasad The problem should now be resolved with the latest k8up version

[haasad]

Unfortunately it looks like the fix does not work consistently. Some backups are correctly marked as completed (and failed) and some backups behave the same way as before.

kubectl -n syn-backup-k8up get po k8up-69fb78f479-hn7bv -oyaml | yq '.spec.containers[].image'
ghcr.io/k8up-io/k8up:v2.11.3

kubectl -n ahaas-test get schedule noop-schedule -oyaml
apiVersion: k8up.io/v1
kind: Schedule
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"k8up.io/v1","kind":"Schedule","metadata":{"annotations":{},"name":"noop-schedule","namespace":"ahaas-test"},"spec":{"backup":{"failedJobsHistoryLimit":2,"resources":{},"schedule":"* * * * *","successfulJobsHistoryLimit":2}}}
  creationTimestamp: "2024-12-20T09:12:33Z"
  finalizers:
  - k8up.io/schedule
  generation: 2
  name: noop-schedule
  namespace: ahaas-test
  resourceVersion: "2031059042"
  uid: 84fd3c3d-50e4-4935-a4dd-57317bcb02cd
spec:
  backup:
    failedJobsHistoryLimit: 2
    resources: {}
    schedule: '* * * * *'
    successfulJobsHistoryLimit: 2
  resourceRequirementsTemplate: {}
status:
  conditions:
  - lastTransitionTime: "2024-12-20T09:12:33Z"
    message: The resource is ready
    reason: Ready
    status: "True"
    type: Ready

kubectl -n ahaas-test get backups.k8up.io --sort-by .metadata.creationTimestamp
NAME                         SCHEDULE REF    COMPLETION   PREBACKUP              AGE
noop-schedule-backup-pw6xs   noop-schedule                NoPreBackupPodsFound   3h48m
noop-schedule-backup-9w8s6   noop-schedule                NoPreBackupPodsFound   3h44m
noop-schedule-backup-pvwsx   noop-schedule                NoPreBackupPodsFound   3h34m
noop-schedule-backup-497pr   noop-schedule                NoPreBackupPodsFound   3h30m
noop-schedule-backup-kdbcf   noop-schedule                NoPreBackupPodsFound   3h26m
noop-schedule-backup-6x8gr   noop-schedule                NoPreBackupPodsFound   3h24m
noop-schedule-backup-d849p   noop-schedule                NoPreBackupPodsFound   3h23m
noop-schedule-backup-6qshf   noop-schedule                NoPreBackupPodsFound   3h20m
noop-schedule-backup-jndlr   noop-schedule                NoPreBackupPodsFound   3h16m
noop-schedule-backup-p5dkh   noop-schedule                NoPreBackupPodsFound   3h11m
noop-schedule-backup-cs7h2   noop-schedule                NoPreBackupPodsFound   3h9m
noop-schedule-backup-fw4m8   noop-schedule                NoPreBackupPodsFound   3h8m
noop-schedule-backup-n9b58   noop-schedule                NoPreBackupPodsFound   3h4m
noop-schedule-backup-fp862   noop-schedule                NoPreBackupPodsFound   3h2m
noop-schedule-backup-vj84q   noop-schedule                NoPreBackupPodsFound   179m
noop-schedule-backup-56q5g   noop-schedule                NoPreBackupPodsFound   178m
noop-schedule-backup-cpxpr   noop-schedule                NoPreBackupPodsFound   176m
noop-schedule-backup-6bbxz   noop-schedule                NoPreBackupPodsFound   173m
noop-schedule-backup-s8v5s   noop-schedule                NoPreBackupPodsFound   170m
noop-schedule-backup-qdbvt   noop-schedule                NoPreBackupPodsFound   169m
noop-schedule-backup-qvrlh   noop-schedule                NoPreBackupPodsFound   167m
noop-schedule-backup-5svj9   noop-schedule                NoPreBackupPodsFound   163m
noop-schedule-backup-frpdl   noop-schedule                NoPreBackupPodsFound   162m
noop-schedule-backup-2m6rs   noop-schedule                NoPreBackupPodsFound   158m
noop-schedule-backup-kht4s   noop-schedule                NoPreBackupPodsFound   155m
noop-schedule-backup-plj77   noop-schedule                NoPreBackupPodsFound   152m
noop-schedule-backup-ktkr2   noop-schedule                NoPreBackupPodsFound   149m
noop-schedule-backup-xt9z7   noop-schedule                NoPreBackupPodsFound   148m
noop-schedule-backup-j8v5b   noop-schedule                NoPreBackupPodsFound   146m
noop-schedule-backup-gx9n2   noop-schedule                NoPreBackupPodsFound   142m
noop-schedule-backup-v75b7   noop-schedule                NoPreBackupPodsFound   140m
noop-schedule-backup-jzkbx   noop-schedule                NoPreBackupPodsFound   136m
noop-schedule-backup-4jwzv   noop-schedule                NoPreBackupPodsFound   134m
noop-schedule-backup-2lfj9   noop-schedule                NoPreBackupPodsFound   130m
noop-schedule-backup-wv9bv   noop-schedule                NoPreBackupPodsFound   127m
noop-schedule-backup-tfdtt   noop-schedule                NoPreBackupPodsFound   124m
noop-schedule-backup-h9knq   noop-schedule                NoPreBackupPodsFound   121m
noop-schedule-backup-5q89l   noop-schedule                NoPreBackupPodsFound   118m
noop-schedule-backup-8k5dh   noop-schedule                NoPreBackupPodsFound   115m
noop-schedule-backup-586k7   noop-schedule                NoPreBackupPodsFound   109m
noop-schedule-backup-ctlhr   noop-schedule                NoPreBackupPodsFound   107m
noop-schedule-backup-gj7mb   noop-schedule                NoPreBackupPodsFound   103m
noop-schedule-backup-wbf68   noop-schedule                NoPreBackupPodsFound   100m
noop-schedule-backup-mj4ws   noop-schedule                NoPreBackupPodsFound   97m
noop-schedule-backup-2w67l   noop-schedule                NoPreBackupPodsFound   94m
noop-schedule-backup-gp946   noop-schedule                NoPreBackupPodsFound   91m
noop-schedule-backup-2bclc   noop-schedule                NoPreBackupPodsFound   89m
noop-schedule-backup-rj7hk   noop-schedule                NoPreBackupPodsFound   88m
noop-schedule-backup-nw6m6   noop-schedule                NoPreBackupPodsFound   83m
noop-schedule-backup-tw6ff   noop-schedule                NoPreBackupPodsFound   77m
noop-schedule-backup-v8v9z   noop-schedule                NoPreBackupPodsFound   76m
noop-schedule-backup-h4ssg   noop-schedule                NoPreBackupPodsFound   75m
noop-schedule-backup-pwr6w   noop-schedule                NoPreBackupPodsFound   72m
noop-schedule-backup-gssww   noop-schedule                NoPreBackupPodsFound   70m
noop-schedule-backup-7r95b   noop-schedule                NoPreBackupPodsFound   68m
noop-schedule-backup-mh98r   noop-schedule                NoPreBackupPodsFound   67m
noop-schedule-backup-pzfv9   noop-schedule                NoPreBackupPodsFound   66m
noop-schedule-backup-ncxk4   noop-schedule                NoPreBackupPodsFound   65m
noop-schedule-backup-wh4t2   noop-schedule                NoPreBackupPodsFound   64m
noop-schedule-backup-4qvvv   noop-schedule                NoPreBackupPodsFound   61m
noop-schedule-backup-kw9r6   noop-schedule                NoPreBackupPodsFound   59m
noop-schedule-backup-llm8r   noop-schedule                NoPreBackupPodsFound   58m
noop-schedule-backup-8pjwr   noop-schedule                NoPreBackupPodsFound   55m
noop-schedule-backup-ww5q5   noop-schedule                NoPreBackupPodsFound   53m
noop-schedule-backup-lbtr7   noop-schedule                NoPreBackupPodsFound   52m
noop-schedule-backup-9dc8b   noop-schedule                NoPreBackupPodsFound   48m
noop-schedule-backup-8qqpm   noop-schedule                NoPreBackupPodsFound   47m
noop-schedule-backup-j72zj   noop-schedule                NoPreBackupPodsFound   46m
noop-schedule-backup-79rbj   noop-schedule                NoPreBackupPodsFound   43m
noop-schedule-backup-59vrj   noop-schedule                NoPreBackupPodsFound   41m
noop-schedule-backup-qsgdn   noop-schedule                NoPreBackupPodsFound   38m
noop-schedule-backup-k7g29   noop-schedule                NoPreBackupPodsFound   37m
noop-schedule-backup-vf4b6   noop-schedule                NoPreBackupPodsFound   35m
noop-schedule-backup-5pv88   noop-schedule                NoPreBackupPodsFound   34m
noop-schedule-backup-sg4q8   noop-schedule                NoPreBackupPodsFound   31m
noop-schedule-backup-pfph6   noop-schedule                NoPreBackupPodsFound   28m
noop-schedule-backup-6925l   noop-schedule                NoPreBackupPodsFound   25m
noop-schedule-backup-4xthd   noop-schedule                NoPreBackupPodsFound   22m
noop-schedule-backup-fvhhp   noop-schedule                NoPreBackupPodsFound   19m
noop-schedule-backup-fqkc7   noop-schedule                NoPreBackupPodsFound   16m
noop-schedule-backup-tqbtn   noop-schedule                NoPreBackupPodsFound   13m
noop-schedule-backup-gwl95   noop-schedule   Failed       NoPreBackupPodsFound   9m23s
noop-schedule-backup-gf95p   noop-schedule                NoPreBackupPodsFound   8m23s
noop-schedule-backup-rd272   noop-schedule                NoPreBackupPodsFound   7m23s
noop-schedule-backup-98w4v   noop-schedule   Failed       NoPreBackupPodsFound   6m23s
noop-schedule-backup-fjrs4   noop-schedule                NoPreBackupPodsFound   5m23s
noop-schedule-backup-dkw4m   noop-schedule                NoPreBackupPodsFound   4m23s
noop-schedule-backup-gpcdj   noop-schedule                NoPreBackupPodsFound   3m23s
noop-schedule-backup-q5fln   noop-schedule                NoPreBackupPodsFound   2m23s
noop-schedule-backup-smw6h   noop-schedule                NoPreBackupPodsFound   83s
noop-schedule-backup-xb7qf   noop-schedule                NoPreBackupPodsFound   23s

Failed backups are correctly pruned, however only ca. 50% of the backups are marked as Failed. The remaining one keep accumulating like they did before.