[🐛 Bug]: chrome-node-scaledjobs.yaml don't apply ttlSecondsAfterFinished and minReplicaCount

[cr-liorholtzman]

What happened?

The scaledJob charts don't apply ttlSecondsAfterFinished see link and therefore jobs are stuck in completed once container is removed

chart 0.19.0

Command used to start Selenium Grid with Docker

autoscaling:
  # Enable autoscaling. Implies installing KEDA
  enabled: false
  # Enable autoscaling without automatically installing KEDA
  enableWithExistingKEDA: yes
  # Which typ of KEDA scaling to use: job or deployment
  scalingType: job
  # Annotations for KEDA resources: ScaledObject and ScaledJob
  annotations:
    helm.sh/hook: post-install,post-upgrade
  # Options for KEDA ScaledJobs
  scaledJobOptions:
    ttlSecondsAfterFinished: 100
    pollingInterval: 10
    scalingStrategy:
      strategy: accurate
  deregisterLifecycle:
    preStop:
      exec:
        command:
          - bash
          - -c
          - |
            curl -X POST 127.0.0.1:5555/se/grid/node/drain --header 'X-REGISTRATION-SECRET;' && \
            while curl 127.0.0.1:5555/status; do sleep 1; done;

Relevant log output

Error: UPGRADE FAILED: post-upgrade hooks failed: unable to build kubernetes object for post-upgrade hook selenium-grid/templates/chrome-node-scaledjobs.yaml: error validating "": error validating data: ValidationError(ScaledJob.spec): unknown field "ttlSecondsAfterFinished" in sh.keda.v1alpha1.ScaledJob.spec

selenium-chrome-node-24h6v-sd26s          0/1     Completed   0          16h
selenium-chrome-node-2c7v7-5zpxh          0/1     Completed   0          17h
selenium-chrome-node-2hvxv-58vxp          0/1     Completed   0          16h
selenium-chrome-node-2khwt-gw26g          0/1     Completed   0          18h
selenium-chrome-node-2lj7f-9ff25          0/1     Completed   0          15h
selenium-chrome-node-2thlx-qlgxq          0/1     Completed   0          17h
selenium-chrome-node-4f5jx-2lr7v          0/1     Completed   0          17h
selenium-chrome-node-4fmjk-dd6jp          0/1     Completed   0          16h
selenium-chrome-node-4hgn2-vt7bq          0/1     Completed   0          15h
selenium-chrome-node-4sqbj-vzxfn          0/1     Completed   0          17h
selenium-chrome-node-544c4-7gf6m          0/1     Completed   0          15h
selenium-chrome-node-54xc8-s8hp4          0/1     Completed   0          15h
selenium-chrome-node-58kpc-sg2xx          0/1     Completed   0          16h
selenium-chrome-node-5h2j5-c68x8          0/1     Completed   0          16h
selenium-chrome-node-67lmp-lbh7m          0/1     Completed   0          15h
selenium-chrome-node-686j4-7n5d4          0/1     Completed   0          17h
selenium-chrome-node-6gdxq-6j6bc          0/1     Completed   0          15h
selenium-chrome-node-6lbw5-kd2hp          0/1     Completed   0          16h
selenium-chrome-node-6mxqw-qqhv8          0/1     Completed   0          15h
selenium-chrome-node-72c8l-q2dz8          0/1     Completed   0          17h
selenium-chrome-node-76dkw-4m2wk          0/1     Completed   0          15h
selenium-chrome-node-7n9wn-n6nhf          0/1     Completed   0          17h
selenium-chrome-node-7td89-262jx          0/1     Completed   0          16h
selenium-chrome-node-7vhv6-ndgm8          0/1     Completed   0          15h
selenium-chrome-node-896cb-7hxrg          0/1     Completed   0          15h
selenium-chrome-node-8kcf2-d9bb7          0/1     Completed   0          17h
selenium-chrome-node-8nftg-7rqws          0/1     Completed   0          17h
selenium-chrome-node-8pjh2-zcjb8          0/1     Completed   0          16h
selenium-chrome-node-8xmkx-k942v          0/1     Completed   0          15h
selenium-chrome-node-8xrkx-4k9xp          0/1     Completed   0          16h
selenium-chrome-node-8zhlz-kw2dr          0/1     Completed   0          16h
selenium-chrome-node-9g76r-5vwl9          0/1     Completed   0          15h
selenium-chrome-node-9mrvj-g5f5j          0/1     Completed   0          15h
selenium-chrome-node-bdgfx-6pnx4          0/1     Completed   0          14h
selenium-chrome-node-bgbzt-bl2sj          0/1     Completed   0          16h
selenium-chrome-node-bx9rp-kw9c7          0/1     Completed   0          16h
selenium-chrome-node-c9l7p-bxv2j          0/1     Completed   0          16h
selenium-chrome-node-cb9sn-x5w5b          0/1     Completed   0          16h
selenium-chrome-node-cc9m8-49mm8          0/1     Completed   0          15h
selenium-chrome-node-crbjj-gv4q4          0/1     Completed   0          15h
selenium-chrome-node-d4jd2-fcv6x          0/1     Completed   0          41m
selenium-chrome-node-ddnl4-l7k2c          0/1     Completed   0          18h
selenium-chrome-node-dk57n-rnrtv          0/1     Completed   0          15h
selenium-chrome-node-gldrr-qtv9t          0/1     Completed   0          15h
selenium-chrome-node-hxmf5-t9mdp          0/1     Completed   0          15h
selenium-chrome-node-k4pdq-b5v22          0/1     Completed   0          15h
selenium-chrome-node-kfcbb-9ptlp          0/1     Completed   0          16h
selenium-chrome-node-kg74k-4bk28          0/1     Completed   0          15h
selenium-chrome-node-kvfqq-2s7p7          0/1     Completed   0          15h
selenium-chrome-node-kxsqp-cqmfn          0/1     Completed   0          16h
selenium-chrome-node-l2642-fhqxn          0/1     Completed   0          15h
selenium-chrome-node-l4ltr-dwp28          0/1     Completed   0          14h
selenium-chrome-node-l6x96-5458w          0/1     Completed   0          17h
selenium-chrome-node-l9v5j-bgqjs          0/1     Completed   0          17h
selenium-chrome-node-ldmg4-gzt7v          0/1     Completed   0          17h
selenium-chrome-node-m97ml-c72jn          0/1     Completed   0          15h
selenium-chrome-node-mljrd-k89v5          0/1     Completed   0          16h
selenium-chrome-node-n4pxn-mft8h          0/1     Completed   0          17h
selenium-chrome-node-n8dw6-76mr5          0/1     Completed   0          15h
selenium-chrome-node-nkgz8-hqgxc          0/1     Completed   0          16h
selenium-chrome-node-p5gkr-2fz8p          0/1     Completed   0          16h
selenium-chrome-node-pmjt8-lwqzt          0/1     Completed   0          15h
selenium-chrome-node-q2xwc-z764d          0/1     Completed   0          17h
selenium-chrome-node-q75mp-b2rnn          0/1     Completed   0          15h
selenium-chrome-node-q8jjj-zsz5k          0/1     Completed   0          16h
selenium-chrome-node-qhx99-j54jk          0/1     Completed   0          15h
selenium-chrome-node-qpcn9-2r7mf          0/1     Completed   0          18h
selenium-chrome-node-r5tt9-9n4xx          0/1     Completed   0          16h
selenium-chrome-node-s72ww-hzl8b          0/1     Completed   0          15h
selenium-chrome-node-s8m4r-4lzkr          0/1     Completed   0          16h
selenium-chrome-node-sgrfw-4hwpr          0/1     Completed   0          18h
selenium-chrome-node-ssdm2-g67nk          0/1     Completed   0          14h
selenium-chrome-node-stxjg-jx5nb          0/1     Completed   0          16h
selenium-chrome-node-szndq-wcs5l          0/1     Completed   0          16h
selenium-chrome-node-t7c7k-z7rgj          0/1     Completed   0          15h
selenium-chrome-node-tf4ft-2q56f          0/1     Completed   0          15h
selenium-chrome-node-tt9bv-j6zlt          0/1     Completed   0          15h
selenium-chrome-node-v84g5-qzpwz          0/1     Completed   0          18h
selenium-chrome-node-vbgkr-hxcd9          0/1     Completed   0          17h
selenium-chrome-node-vf45l-7p8b8          0/1     Completed   0          15h
selenium-chrome-node-vljdf-2tjhh          0/1     Completed   0          15h
selenium-chrome-node-vlm5c-zhzds          0/1     Completed   0          16h
selenium-chrome-node-vrhn7-zwkpx          0/1     Completed   0          15h
selenium-chrome-node-wb24j-8kctp          0/1     Completed   0          17h
selenium-chrome-node-wbcpr-47w5d          0/1     Completed   0          16h
selenium-chrome-node-wc4jt-nr9hw          0/1     Completed   0          15h
selenium-chrome-node-wcnt8-st5br          0/1     Completed   0          14h
selenium-chrome-node-wcnxn-mlp9z          0/1     Completed   0          16h
selenium-chrome-node-wfrr2-vw472          0/1     Completed   0          15h
selenium-chrome-node-wg46n-vgqhv          0/1     Completed   0          15h
selenium-chrome-node-whtmt-nhvfd          0/1     Completed   0          15h
selenium-chrome-node-wmb7s-njsl9          0/1     Completed   0          15h
selenium-chrome-node-wsdzl-rcltr          0/1     Completed   0          41m
selenium-chrome-node-x2qdj-84bmd          0/1     Completed   0          18h
selenium-chrome-node-x9lvp-d2n22          0/1     Completed   0          16h
selenium-chrome-node-xhtml-7wcbt          0/1     Completed   0          15h
selenium-chrome-node-xl8sd-4d2sf          0/1     Completed   0          17h
selenium-chrome-node-xldf9-fx9jc          0/1     Completed   0          16h
selenium-chrome-node-z7mdd-46mdl          0/1     Completed   0          16h
selenium-chrome-node-zk4lv-thvpq          0/1     Completed   0          15h
selenium-distributor-5f559d7b4b-4z282     1/1     Running     0          2d
selenium-event-bus-74f48cb47d-6bjxm       1/1     Running     0          2d
selenium-router-7f455cc464-krtdn          1/1     Running     0          2d
selenium-session-map-77f54c9dc6-7j7ff     1/1     Running     0          2d
selenium-session-queue-54fcc874b9-497kv   1/1     Running     0          2d

Operating System

k8s

Docker Selenium version (tag)

4.10.0-20230607

[github-actions]

@cr-liorholtzman, thank you for creating this issue. We will troubleshoot it as soon as we can.

---

Info for maintainers

Triage this issue by using labels.

If information is missing, add a helpful comment and then I-issue-template label.

If the issue is a question, add the I-question label.

If the issue is valid but there is no time to troubleshoot it, consider adding the help wanted label.

If the issue requires changes or fixes from an external project (e.g., ChromeDriver, GeckoDriver, MSEdgeDriver, W3C), add the applicable G-* label, and it will provide the correct link and auto-close the issue.

After troubleshooting the issue, please add the R-awaiting answer label.

Thank you!

[diemol]

@msvticket do you know?

[cr-liorholtzman]

@diemol Just a small correction, I think it should be under jobTargetRef section according this exmaple - kedacore/keda#630

[cr-liorholtzman]

Another available solution which does seem to work is to use these two on the scaledJobOptions section:

    successfulJobsHistoryLimit: 1
    failedJobsHistoryLimit: 1

[msvticket]

It is not a bug that jobs stay around for a while. There are default values for successfulJobsHistoryLimit and failedJobsHistoryLimit so they will be deleted automatically by default.

If you want to be able to add more fields to scaledJobOptions that would be a feature request.

[amardeep2006]

How long the jobs and pods stay specifically? As per official kubernetes documentation these may have adverse impact on cluster.

https://kubernetes.io/docs/concepts/workloads/controllers/job/#ttl-mechanism-for-finished-jobs

I tried passing below config via value file and it seems effective in cleaning up the jobs.

autoscaling:
  scaledJobOptions:
    successfulJobsHistoryLimit: 5
    failedJobsHistoryLimit: 5

[VietND96]

Via PR #1989, these configs and their default value added

• autoscaling.scaledJobOptions.successfulJobsHistoryLimit: 0 - clear all Completed job by default
• autoscaling.scaledOptions.minReplicaCount: 0 - applicable for all nodes in both scalingType job and deployment
• The template is more flexible for user to add all spec that KEDA ScaledObject and ScaledJob supports.
• The template also enables the capability for the use case that the user would like to set different scaled options for individual browser nodes.

[diemol]

Seems this issue can be closed?