Archived Workflow still appears in UI after adding ttlStrategy

[madhur-lyric]

Pre-requisites

• I have double-checked my configuration
• I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
• I have searched existing issues and could not find a match for this bug
• I'd like to contribute the fix myself (see contributing guide)

What happened/what did you expect to happen?

I have configured the ttlStrategy in my Argo Workflow as follows:

ttlStrategy:
    secondsAfterCompletion: 10

After the workflow completes, it is no longer shown when running the command:

kubectl get wf -n {namespace}

However, the workflow is still visible in the Argo UI and accessible through the Argo API. I would like the workflow to be completely deleted and not shown in the UI or API after the TTL expiration.

/argo-workflows/api/v1/workflows/{namespace}

Version(s)

v3.5.8

Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

spec:
  ttlStrategy:
    secondsAfterCompletion: 10

Logs from the workflow controller

kubectl logs -n argo deploy/workflow-controller | grep ${workflow}

Logs from in your workflow's wait container

kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded

[Joibel]

Do you have workflow archiving enabled?

[madhur-lyric]

@Joibel Yes workflow archive is enabled

controller:
  persistence:
    archive: true

[Joibel]

With enabled archiving, the behaviour you see is what is expected.

The argo-server, which responds to the API and provides the workflows to the UI will provide workflows from both the cluster and the archive. This is not totally clear in the documentation which could do with updating to explain this.

[lost11n]

@madhur-lyric Did you check the retentionPolicy of workflow-controller?It's a global setting ,has higher priority than "ttlStrategy" defined in workflow.For example, using default settings,argo will keep 10 completed workflow,even the time of ttlStrategy has expired.By the way my argo workflow version is 3.5.10

k -n argo describe cm workflow-controller

Default values of retentionPolicy:

retentionPolicy:
----
completed: 10
failed: 3
errored: 3