Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(clouddriver): make fetching properties file more resilient for k8s jobs #4783

Merged
merged 4 commits into from
Sep 27, 2024
Merged

feat(clouddriver): make fetching properties file more resilient for k8s jobs #4783

merged 4 commits into from
Sep 27, 2024

Commits on Sep 27, 2024

  1. tests(orca/clouddriver): refactor tests to remove some test code dupl… 

    …ications and make them more descriptive
    @apoorv-mahajan @kirangodishala
    apoorv-mahajan authored and kirangodishala committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    1d1b42d View commit details
    Browse the repository at this point in the history

  2. feat(orca/clouddriver): make waitOnJobCompletion retries configurable

    @apoorv-mahajan @kirangodishala
    apoorv-mahajan authored and kirangodishala committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    4cef464 View commit details
    Browse the repository at this point in the history

  3. feat(clouddriver): make fetching properties file more resilient for k… 

    …8s jobs

If a k8s run job is marked as succeeded, and property file is defined in the stage context,
then it can so happen that multiple pods are created for that job.
 See https://kubernetes.io/docs/concepts/workloads/controllers/job/#handling-pod-and-container-failures

In extreme edge cases, the first pod may be around before the second one succeeds.
That leads to kubectl logs job/ command failing as seen below:
kubectl -n test logs job/test-run-job-5j2vl -c parser
Found 2 pods, using pod/test-run-job-5j2vl-fj8hd
Error from server (BadRequest): container "parser" in pod "test-run-job-5j2vl-fj8hd" is terminated
or
Found 2 pods, using pod/test-run-job-5j2vl-fj8hd
Error from server (BadRequest): container "parser" in pod "test-run-job-5j2vl-fj8hd" is waiting to start: PodInitializing

where that commands defaults to using one of the two pods.

To fix this issue, if we encounter an error from the kubectl logs job/ command, we
find a successful pod in the job and directly query it for logs.
    @apoorv-mahajan @kirangodishala
    apoorv-mahajan authored and kirangodishala committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    7b27660 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'master' into multiple-pod-runjob-case

    @mergify
    mergify[bot] authored Sep 27, 2024
    Configuration menu
    Copy the full SHA
    9004d4f View commit details
    Browse the repository at this point in the history