feat: add finalizer to workflow pod to prevent `pod deleted`. Fixes #8783 #9058

rohankmr414 · 2022-06-25T14:22:58Z

Signed-off-by: Rohan Kumar rohan@pipekit.io

dpadhiar · 2022-06-27T22:16:21Z

@rohankmr414 could you write a test case or add an example of this to your PR description?

@juliev0 @sarabala1979 the e2e tests for this PR are all failing with 15m0 timeout, something to note I assume

juliev0 · 2022-06-27T23:23:58Z

@rohankmr414 Regarding the test failures, we should determine if they're "real" or intermittently failing. Have you tried to run them locally? You should be able to do "make [testname]" like "make TestMarkDaemonedPodSucceeded"

sarabala1979 · 2022-06-29T15:00:10Z

@rohankmr414 It is a nice fix. Please add unit test and e2e test. Fix the test failures.

rohankmr414 · 2022-07-13T14:42:37Z

=== RUN   TestDaemonPodSuite/TestMarkDaemonedPodSucceeded
Submitting workflow  daemoned-pod-completed-
Waiting 1m0s for workflow metadata.name=daemoned-pod-completed-x6jsz
 ? daemoned-pod-completed-x6jsz Workflow 0s      

 ● daemoned-pod-completed-x6jsz   Workflow 0s      
 └ ● daemoned-pod-completed-x6jsz DAG      0s      
 └ ◷ daemoned                     Pod      0s      

 ● daemoned-pod-completed-x6jsz   Workflow 0s      
 └ ● daemoned-pod-completed-x6jsz DAG      0s      
 └ ◷ daemoned                     Pod      0s      PodInitializing

 ● daemoned-pod-completed-x6jsz   Workflow 0s      
 └ ● daemoned-pod-completed-x6jsz DAG      0s      
 └ ● daemoned                     Pod      3s      
 └ ◷ daemon-dependent             Pod      0s      

 ● daemoned-pod-completed-x6jsz   Workflow 0s      
 └ ● daemoned-pod-completed-x6jsz DAG      0s      
 └ ● daemoned                     Pod      3s      
 └ ◷ daemon-dependent             Pod      0s      PodInitializing

 ● daemoned-pod-completed-x6jsz   Workflow 0s      
 └ ● daemoned-pod-completed-x6jsz DAG      0s      
 └ ● daemoned                     Pod      3s      
 └ ● daemon-dependent             Pod      0s      

 ● daemoned-pod-completed-x6jsz   Workflow 0s      
 └ ● daemoned-pod-completed-x6jsz DAG      0s      
 └ ● daemoned                     Pod      3s      
 └ ● daemon-dependent             Pod      0s      

 ✔ daemoned-pod-completed-x6jsz   Workflow 9s      
 └ ✔ daemoned-pod-completed-x6jsz DAG      9s      
 └ ✔ daemoned                     Pod      3s      
 └ ✔ daemon-dependent             Pod      5s      

 ✔ daemoned-pod-completed-x6jsz   Workflow 9s      
 └ ✔ daemoned-pod-completed-x6jsz DAG      9s      
 └ ✔ daemoned                     Pod      3s      
 └ ✔ daemon-dependent             Pod      5s      

Condition "to be Succeeded" met after 9s
Checking expectation daemoned-pod-completed-x6jsz
daemoned-pod-completed-x6jsz : Succeeded 
=== PASS: DaemonPodSuite/TestMarkDaemonedPodSucceeded
    --- PASS: TestDaemonPodSuite/TestMarkDaemonedPodSucceeded (9.54s)
PASS

@sarabala1979 @juliev0 For some reason, the tests are passing on my local env with the finalizer, not sure what's happening here

juliev0 · 2022-07-13T16:20:46Z

=== RUN   TestDaemonPodSuite/TestMarkDaemonedPodSucceeded
Submitting workflow  daemoned-pod-completed-
Waiting 1m0s for workflow metadata.name=daemoned-pod-completed-x6jsz
 ? daemoned-pod-completed-x6jsz Workflow 0s      

 ● daemoned-pod-completed-x6jsz   Workflow 0s      
 └ ● daemoned-pod-completed-x6jsz DAG      0s      
 └ ◷ daemoned                     Pod      0s      

 ● daemoned-pod-completed-x6jsz   Workflow 0s      
 └ ● daemoned-pod-completed-x6jsz DAG      0s      
 └ ◷ daemoned                     Pod      0s      PodInitializing

 ● daemoned-pod-completed-x6jsz   Workflow 0s      
 └ ● daemoned-pod-completed-x6jsz DAG      0s      
 └ ● daemoned                     Pod      3s      
 └ ◷ daemon-dependent             Pod      0s      

 ● daemoned-pod-completed-x6jsz   Workflow 0s      
 └ ● daemoned-pod-completed-x6jsz DAG      0s      
 └ ● daemoned                     Pod      3s      
 └ ◷ daemon-dependent             Pod      0s      PodInitializing

 ● daemoned-pod-completed-x6jsz   Workflow 0s      
 └ ● daemoned-pod-completed-x6jsz DAG      0s      
 └ ● daemoned                     Pod      3s      
 └ ● daemon-dependent             Pod      0s      

 ● daemoned-pod-completed-x6jsz   Workflow 0s      
 └ ● daemoned-pod-completed-x6jsz DAG      0s      
 └ ● daemoned                     Pod      3s      
 └ ● daemon-dependent             Pod      0s      

 ✔ daemoned-pod-completed-x6jsz   Workflow 9s      
 └ ✔ daemoned-pod-completed-x6jsz DAG      9s      
 └ ✔ daemoned                     Pod      3s      
 └ ✔ daemon-dependent             Pod      5s      

 ✔ daemoned-pod-completed-x6jsz   Workflow 9s      
 └ ✔ daemoned-pod-completed-x6jsz DAG      9s      
 └ ✔ daemoned                     Pod      3s      
 └ ✔ daemon-dependent             Pod      5s      

Condition "to be Succeeded" met after 9s
Checking expectation daemoned-pod-completed-x6jsz
daemoned-pod-completed-x6jsz : Succeeded 
=== PASS: DaemonPodSuite/TestMarkDaemonedPodSucceeded
    --- PASS: TestDaemonPodSuite/TestMarkDaemonedPodSucceeded (9.54s)
PASS

@sarabala1979 @juliev0 For some reason, the tests are passing on my local env with the finalizer, not sure what's happening here

Hmm, sometimes there are intermittent failures unfortunately. We have a bug logged for that (there's an idea that the issue is caused by some overall timeout being too short). Can you push an "empty commit" (i.e. add a space or line or something) to re-trigger the tests? Then you can see if the same tests are failing every time or are intermittent? (Sorry for the trouble)

rohankmr414 · 2022-07-13T16:51:21Z

@juliev0 actually I did a rebase and force push which re-triggered the tests, the same tests are failing again. I think the reason behind this is that in some tests pods are being terminated and they are being stuck in a terminating state due to the finalizer which somehow prevents the next test from being run. It happened in my local env as well.
This is after running make test-functional E2E_SUITE_TIMEOUT=15m STATIC_FILES=false after the test TestDaemonTemplateRef which prevents TestMarkDaemonedPodSucceeded from being run in https://github.com/argoproj/argo-workflows/runs/7320582957?check_suite_focus=true

If I remove the finalizer manually, after which the stuck pods are terminated, the tests are running fine.

juliev0 · 2022-07-13T18:42:06Z

Oh, so it was after you did a rebase, meaning some recent commit in master caused it to change for you? Sorry about that.

When you say "some test pods are being terminated" do you mean the Controller is not the one deleting them and therefore they're not going through your logic to remove the finalizer? Unfortunately, I'm not an expert on those tests. Can you investigate and modify the tests if needed?

rohankmr414 · 2022-07-13T18:44:07Z

@juliev0 the error is not because of the rebase, ig it is because of some bug in pod deletion logic

juliev0 · 2022-07-13T19:52:03Z

@juliev0 the error is not because of the rebase, ig it is because of some bug in pod deletion logic

Okay, thanks for the clarification. If you are able to investigate the bug that would be great.

rohankmr414 · 2022-07-24T18:24:07Z

@juliev0 @sarabala1979 the issue occurs when running multiple tests, some pods are being terminated without being labelled as completed from the previous test, preventing the next test from being run. On running the tests individually the pods are being labelled as completed successfully and getting cleaned up later, not really sure what's happening here

juliev0 · 2022-07-26T22:50:39Z

@juliev0 @sarabala1979 the issue occurs when running multiple tests, some pods are being terminated without being labelled as completed from the previous test, preventing the next test from being run. On running the tests individually the pods are being labelled as completed successfully and getting cleaned up later, not really sure what's happening here

Sorry for the lack of response. I'm new to the project so don't have any insider information. :) It looks like you've made some recent commits - are you able to address this issue of the test?

alexec

I don't think any e2e tests should be changed. If you changed them, then maybe code does not work.

Code looks correct to me otherwise.

* fix: Fix test fail flakey on finalizer branch Signed-off-by: Atsushi Sakai <sakai.at24@gmail.com> Signed-off-by: Rohan Kumar <rohan@pipekit.io>

Signed-off-by: Rohan Kumar <rohan@pipekit.io>

juliev0 · 2023-11-29T16:53:33Z

Sorry for the late response. Thanks for fixing the DCO. I know that can be annoying. (For the future, it seems reasonable that we could make an exception here and there for that but I’m not sure what the precedent is.)

sakai-ast · 2023-11-30T07:10:08Z

Thank you for fixing the DCO error. However, another CI test is now failing...
I have tried to reproduce the failed test, but could not fully reproduce the problem this time.

@juliev0
I believe this is the first time this CI test has failed on this PR, and it looks flaky.
I think this PR is ready for merge, should we still address this failed test?

juliev0 · 2023-11-30T15:49:11Z

Unfortunately there are flakey tests. The typical thing to do in that case is repeated “empty commits” until it all passes

sakai-ast · 2023-11-30T23:47:04Z

@rohankmr414
If you have time, it would be greatly appreciated if you could address the above until the CI passes.

sakai-ast · 2023-12-01T11:15:40Z

Thank you for your help.

@juliev0
All tests have been passed. Could you please confirm?

juliev0 · 2023-12-06T17:44:56Z

workflow/controller/controller.go

+					Operation: "replace",
+					Path:      "/metadata/labels/workflows.argoproj.io~1completed",
+					Value:     "true",
+				},


is this label right? It's not supposed to be "workflows.argoproj.io/completed", is it?

if so, you can use common.LabelKeyCompleted

~1 is escape for JSON Pointer.
https://fdatatracker.ietf.org/doc/html/rfc6901#section-3

okay, thanks

juliev0 · 2023-12-06T18:04:30Z

workflow/controller/controller.go

 			propagation := metav1.DeletePropagationBackground
-			err := pods.Delete(ctx, podName, metav1.DeleteOptions{
+			err = pods.Delete(ctx, podName, metav1.DeleteOptions{


I assume that in general, a given Pod reaches labelPodCompleted and then deletePod logic? Do you know if it's possible for that not to happen?

The execution of the deletePod logic is decided by the following function.

argo-workflows/workflow/controller/pod_cleanup.go

Lines 46 to 72 in 5b6ad2b

func determinePodCleanupAction(

selector labels.Selector,

podLabels map[string]string,

strategy wfv1.PodGCStrategy,

workflowPhase wfv1.WorkflowPhase,

podPhase apiv1.PodPhase,

) podCleanupAction {

switch {

case !selector.Matches(labels.Set(podLabels)): // if the pod will never be deleted, label it now

return labelPodCompleted

case strategy == wfv1.PodGCOnPodNone:

return labelPodCompleted

case strategy == wfv1.PodGCOnWorkflowCompletion && workflowPhase.Completed():

return deletePod

case strategy == wfv1.PodGCOnWorkflowSuccess && workflowPhase == wfv1.WorkflowSucceeded:

return deletePod

case strategy == wfv1.PodGCOnPodCompletion:

return deletePod

case strategy == wfv1.PodGCOnPodSuccess && podPhase == apiv1.PodSucceeded:

return deletePod

case strategy == wfv1.PodGCOnPodSuccess && podPhase == apiv1.PodFailed:

return labelPodCompleted

case workflowPhase.Completed():

return labelPodCompleted

}

return ""

}

This belongs by Pod GC strategy.
https://argoproj.github.io/argo-workflows/fields/#:~:text=pod%20GC%20queue.-,strategy,-string

argo-workflows/examples/pod-gc-strategy.yaml

Lines 10 to 15 in dbf4215

# Pod GC strategy must be one of the following:

# * OnPodCompletion - delete pods immediately when pod is completed (including errors/failures)

# * OnPodSuccess - delete pods immediately when pod is successful

# * OnWorkflowCompletion - delete pods when workflow is completed

# * OnWorkflowSuccess - delete pods when workflow is successful

# Default: do not delete pods

juliev0 · 2023-12-06T18:15:48Z

workflow/common/common.go

@@ -168,6 +168,9 @@ const (
 	// Finalizer to block deletion of the workflow if deletion of artifacts fail for some reason.
 	FinalizerArtifactGC = workflow.WorkflowFullName + "/artifact-gc"

+	// Finalizer blocks deletion of pods until we're captured their status.
+	Finalizer = workflow.WorkflowFullName + "/status"
+


Maybe these 2 finalizers should be named to distinguish what they're being applied on? Like:

FinalizerWorkflowArtifactGC FinalizerPodStatus

What do you think?

juliev0 · 2023-12-06T19:53:56Z

There is a danger in this PR of there being some edge case which causes that finalizer to remain on the pod and for the pod to not be deletable unless manually patched...hence Alex's request to add a feature flag (I think that would generally be an environment variable - not sure if it should default to false or true).

Also, do you know for a fact that the deletePod logic in operator.go is the only way in the Controller for a pod to be deleted? (For example, I notice that the Pod is owned by the Workflow (OwnerReference here), which means there could theoretically be a scenario in which it's being automatically deleted when the Workflow gets deleted some of the time? Could you help alleviate any concern for that by talking through the flow, including any edge cases you can think of?

…rgoproj#8783 Continuing Work of argoproj#9058 Signed-off-by: Atsushi Sakai <sakai.at24@gmail.com>

sakai-ast · 2023-12-26T07:22:59Z

I still have a strong need for the finalizer logic in my organization. However, since this PR is not mine, it is difficult for me to contribute to it quickly. Therefore, I have created the following PR in respect of this one.
#12413

hence Alex's request to add a feature flag (I think that would generally be an environment variable - not sure if it should default to false or true).

I implemented a feature flag in the above PR.

Also, do you know for a fact that the deletePod logic in operator.go is the only way in the Controller for a pod to be deleted?

As far as I know, yes.

(For example, I notice that the Pod is owned by the Workflow (OwnerReference here), which means there could theoretically be a scenario in which it's being automatically deleted when the Workflow gets deleted some of the time?

Currently, Argo Workflows allows users to choose the pod deletion logic, as noted in the comments below.
#9058 (comment)

If the pod deletion logic were solely reliant on OwnerReference, users would not have this choice. Therefore, I believe the current pod deletion logic is necessary.

…8783 Continuing Work of #9058 (#12413) Signed-off-by: Atsushi Sakai <sakai.at24@gmail.com>

agilgur5 · 2024-01-29T05:34:35Z

Closing this as superseded by #12413, which was just merged. Big thanks to everyone who worked on it!

juliev0 · 2024-01-29T05:54:06Z

Closing this as superseded by #12413, which was just merged. Big thanks to everyone who worked on it!

Good call

…rgoproj#8783 Continuing Work of argoproj#9058 (argoproj#12413) Signed-off-by: Atsushi Sakai <sakai.at24@gmail.com>

acornegruta · 2024-06-12T15:33:03Z

Hi, can you give us an approximate date when this will be released please? Even with automatic retries, pods are being randomly deleted, happens almost on a daily basis.

zonybob · 2024-08-19T12:31:36Z

For anyone waiting for 3.6 to fix this issue (like I was/am), I thought I'd drop a Kyverno policy that works well for us to block OpenShift from evicting "Completed" pods that haven't been seen/accounted for by Argo. It blocks the evict call until Argo labels the pod with workflows.argoproj.io/completed: true

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: disable-eviction-too-fast
spec:
  admission: true
  background: false
  rules:
    - context:
        - apiCall:
            method: GET
            urlPath: /api/v1/namespaces/{{request.object.metadata.namespace}}/pods/{{request.object.metadata.name}}
          name: wfpod
      match:
        any:
          - resources:
              kinds:
                - v1/Pod.eviction
              operations:
                - CREATE
      name: Prevent race condition on Workflow Pods
      skipBackgroundRequests: true
      validate:
        deny:
          conditions:
            all:
              - key: "{{ wfpod.metadata.ownerReferences[].kind }}"
                operator: AnyIn
                value:
                  - Workflow
              - key: "{{ wfpod.status.phase }}"
                operator: Equals
                value: Succeeded
              - key:
                  '{{ wfpod.metadata.labels."workflows.argoproj.io/completed" || foo
                  | to_string(@) }}'
                operator: Equals
                value: "false"
        message: Eviction is too soon after job completion
  validationFailureAction: Enforce

rohankmr414 force-pushed the finalizer branch from 773492c to 8e0297e Compare June 25, 2022 16:47

sarabala1979 assigned juliev0, sarabala1979 and dpadhiar Jun 27, 2022

rohankmr414 force-pushed the finalizer branch from 8e0297e to d1d0cee Compare July 13, 2022 12:04

rohankmr414 force-pushed the finalizer branch 4 times, most recently from 762c791 to b164fd4 Compare July 21, 2022 20:49

rohankmr414 force-pushed the finalizer branch from 5c00841 to df001a3 Compare July 26, 2022 06:21

rohankmr414 force-pushed the finalizer branch 7 times, most recently from 68ff951 to f1ff236 Compare August 1, 2022 14:38

alexec reviewed Aug 2, 2022

View reviewed changes

alexec added the pinned label Aug 10, 2022

fix: Fix test fail flaky on finalizer branch (#191)

091a766

* fix: Fix test fail flakey on finalizer branch Signed-off-by: Atsushi Sakai <sakai.at24@gmail.com> Signed-off-by: Rohan Kumar <rohan@pipekit.io>

rohankmr414 force-pushed the finalizer branch from da8d6c8 to 091a766 Compare November 29, 2023 15:47

Merge branch 'master' into finalizer

606ad1e

Signed-off-by: Rohan Kumar <rohan@pipekit.io>

Merge branch 'main' into finalizer

f011046

juliev0 reviewed Dec 6, 2023

View reviewed changes

sakai-ast added a commit to sakai-ast/argo-workflows that referenced this pull request Dec 22, 2023

feat: Add finalizer to workflow pod to prevent 'pod deleted'. Fixes a…

6319b77

…rgoproj#8783 Continuing Work of argoproj#9058 Signed-off-by: Atsushi Sakai <sakai.at24@gmail.com>

sakai-ast added a commit to sakai-ast/argo-workflows that referenced this pull request Dec 26, 2023

feat: Add finalizer to workflow pod to prevent 'pod deleted'. Fixes a…

33954b7

…rgoproj#8783 Continuing Work of argoproj#9058 Signed-off-by: Atsushi Sakai <sakai.at24@gmail.com>

sakai-ast mentioned this pull request Dec 26, 2023

feat: Add finalizer to workflow pod to prevent pod deleted. Fixes #8783. Continuing Work of #9058 #12413

Merged

agilgur5 added the area/controller Controller issues, panics label Dec 29, 2023

juliev0 pushed a commit that referenced this pull request Jan 29, 2024

feat: Add finalizer to workflow pod to prevent 'pod deleted'. Fixes #…

6abe8a9

…8783 Continuing Work of #9058 (#12413) Signed-off-by: Atsushi Sakai <sakai.at24@gmail.com>

agilgur5 closed this Jan 29, 2024

isubasinghe mentioned this pull request Feb 27, 2024

Release v3.5 patch releases discussion #11997

Open

agilgur5 added this to the v3.6.0 milestone Apr 12, 2024

agilgur5 added the solution/superseded This PR or issue has been superseded by another one (slightly different from a duplicate) label Jun 12, 2024

agilgur5 changed the title ~~feat: add finalizer to workflow pod to prevent "pod deleted". Fixes #8783~~ feat: add finalizer to workflow pod to prevent pod deleted. Fixes #8783 Jun 12, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add finalizer to workflow pod to prevent `pod deleted`. Fixes #8783 #9058

feat: add finalizer to workflow pod to prevent `pod deleted`. Fixes #8783 #9058

rohankmr414 commented Jun 25, 2022 •

edited

Loading

dpadhiar commented Jun 27, 2022 •

edited

Loading

juliev0 commented Jun 27, 2022 •

edited

Loading

sarabala1979 commented Jun 29, 2022

rohankmr414 commented Jul 13, 2022

juliev0 commented Jul 13, 2022 •

edited

Loading

rohankmr414 commented Jul 13, 2022

juliev0 commented Jul 13, 2022

rohankmr414 commented Jul 13, 2022

juliev0 commented Jul 13, 2022

rohankmr414 commented Jul 24, 2022

juliev0 commented Jul 26, 2022

alexec left a comment

juliev0 commented Nov 29, 2023

sakai-ast commented Nov 30, 2023 •

edited

Loading

juliev0 commented Nov 30, 2023 •

edited

Loading

sakai-ast commented Nov 30, 2023

sakai-ast commented Dec 1, 2023

juliev0 Dec 6, 2023

juliev0 Dec 6, 2023

sakai-ast Dec 26, 2023

juliev0 Dec 27, 2023

juliev0 Dec 6, 2023

sakai-ast Dec 26, 2023

juliev0 Dec 6, 2023

juliev0 commented Dec 6, 2023

sakai-ast commented Dec 26, 2023

agilgur5 commented Jan 29, 2024 •

edited

Loading

juliev0 commented Jan 29, 2024

acornegruta commented Jun 12, 2024

zonybob commented Aug 19, 2024 •

edited by agilgur5

Loading

	func determinePodCleanupAction(
	selector labels.Selector,
	podLabels map[string]string,
	strategy wfv1.PodGCStrategy,
	workflowPhase wfv1.WorkflowPhase,
	podPhase apiv1.PodPhase,
	) podCleanupAction {
	switch {
	case !selector.Matches(labels.Set(podLabels)): // if the pod will never be deleted, label it now
	return labelPodCompleted
	case strategy == wfv1.PodGCOnPodNone:
	return labelPodCompleted
	case strategy == wfv1.PodGCOnWorkflowCompletion && workflowPhase.Completed():
	return deletePod
	case strategy == wfv1.PodGCOnWorkflowSuccess && workflowPhase == wfv1.WorkflowSucceeded:
	return deletePod
	case strategy == wfv1.PodGCOnPodCompletion:
	return deletePod
	case strategy == wfv1.PodGCOnPodSuccess && podPhase == apiv1.PodSucceeded:
	return deletePod
	case strategy == wfv1.PodGCOnPodSuccess && podPhase == apiv1.PodFailed:
	return labelPodCompleted
	case workflowPhase.Completed():
	return labelPodCompleted
	}
	return ""
	}

	# Pod GC strategy must be one of the following:
	# * OnPodCompletion - delete pods immediately when pod is completed (including errors/failures)
	# * OnPodSuccess - delete pods immediately when pod is successful
	# * OnWorkflowCompletion - delete pods when workflow is completed
	# * OnWorkflowSuccess - delete pods when workflow is successful
	# Default: do not delete pods

feat: add finalizer to workflow pod to prevent pod deleted. Fixes #8783 #9058

feat: add finalizer to workflow pod to prevent pod deleted. Fixes #8783 #9058

Conversation

rohankmr414 commented Jun 25, 2022 • edited Loading

dpadhiar commented Jun 27, 2022 • edited Loading

juliev0 commented Jun 27, 2022 • edited Loading

sarabala1979 commented Jun 29, 2022

rohankmr414 commented Jul 13, 2022

juliev0 commented Jul 13, 2022 • edited Loading

rohankmr414 commented Jul 13, 2022

juliev0 commented Jul 13, 2022

rohankmr414 commented Jul 13, 2022

juliev0 commented Jul 13, 2022

rohankmr414 commented Jul 24, 2022

juliev0 commented Jul 26, 2022

alexec left a comment

Choose a reason for hiding this comment

juliev0 commented Nov 29, 2023

sakai-ast commented Nov 30, 2023 • edited Loading

juliev0 commented Nov 30, 2023 • edited Loading

sakai-ast commented Nov 30, 2023

sakai-ast commented Dec 1, 2023

juliev0 Dec 6, 2023

Choose a reason for hiding this comment

juliev0 Dec 6, 2023

Choose a reason for hiding this comment

sakai-ast Dec 26, 2023

Choose a reason for hiding this comment

juliev0 Dec 27, 2023

Choose a reason for hiding this comment

juliev0 Dec 6, 2023

Choose a reason for hiding this comment

sakai-ast Dec 26, 2023

Choose a reason for hiding this comment

juliev0 Dec 6, 2023

Choose a reason for hiding this comment

juliev0 commented Dec 6, 2023

sakai-ast commented Dec 26, 2023

agilgur5 commented Jan 29, 2024 • edited Loading

juliev0 commented Jan 29, 2024

acornegruta commented Jun 12, 2024

zonybob commented Aug 19, 2024 • edited by agilgur5 Loading

feat: add finalizer to workflow pod to prevent `pod deleted`. Fixes #8783 #9058

feat: add finalizer to workflow pod to prevent `pod deleted`. Fixes #8783 #9058

rohankmr414 commented Jun 25, 2022 •

edited

Loading

dpadhiar commented Jun 27, 2022 •

edited

Loading

juliev0 commented Jun 27, 2022 •

edited

Loading

juliev0 commented Jul 13, 2022 •

edited

Loading

sakai-ast commented Nov 30, 2023 •

edited

Loading

juliev0 commented Nov 30, 2023 •

edited

Loading

agilgur5 commented Jan 29, 2024 •

edited

Loading

zonybob commented Aug 19, 2024 •

edited by agilgur5

Loading