Please allow for defining dependency DAG (Directed Acyclic Graph)

[AceHack]

Summary

When defining Application please allow a new field that is very similar to SyncWaves but allows for more fine-grained control of dependencies, thus allowing more parallelism. For any Application, I should be able to define a DependsOn array that lists other applications that must be synced and correct before I am allowed to sync. This is very similar to SyncWaves but would allow for a much finer-grained parallelism.

Motivation

Below is our SyncWaves and they are very complex and we are unable to parallels everything to it's the fullest extent because of inherent limitation in the sync wave design. We would like to be able to completely express our fine-grained dependencies in a DAG so maximum parallelism can be reached.

  syncWaves:
    clusterBootstrap:
      namespaces:
        platformTopology: 0
        playgroundTopology: 0
      priorityClasses: 0
      storageClasses: 0
      sealedSecretsController: 10
      sealedSecrets: 20
      spotTerminationHandler: 10
    istio:
      controlPlane: 1000
      kiali: 1010
    infra:
      jaeger: 2000
      prometheus:
        operator: 2000
        alertRules: 2010
        grafanaDashboard: 2010
        thanos: 2010
      certManager: 2000
      argoCd: 2100
    operators:
      spark: 3000
      zookeeper: 3000
    knative:
      operator: 3000
      serving: 4000
      eventing: 4000
      source:
        config: 3000
        sqs: 4000
      kafka:
        zookeeper: 4000
    search:
      core:
        zookeeper: 5000
        solr: 5010
      services:
        api: 6000
    content:
      cassandra: 7000
      spark: 7000

Proposal

Please add a DependsOn array to the application so it can list out what other applications it depends on.

[sylvainOL]

Hello,
any news on this proposal?
What can I do to help?

[JohnMops]

would love to know as well

[danmx]

check out Skyscanner's project https://github.com/Skyscanner/applicationset-progressive-sync

[Vanuan]

I think Dagger does this.

[alexec]

Project lead here. I wrote Sync Waves, and know the Argo Workflows logic that determines how task dependencies are evaluated.

I've always wanted to be able to better understand the impact of a sync. Today, you press "sync" and hope for this best. Hence "sync plans" #8511.

A sync consists of a series of ordered "sync tasks":

https://github.com/argoproj/gitops-engine/blob/ed70eac8b7bd6b2f276502398fdbccccab5d189a/pkg/sync/sync_tasks.go#L70

This code already supports ordering. It would need to be updated to take into account the topological sorting of tasks.

One question would be - do dependencies take precedence over waves? I think either they have to (because you could create app which could not sync).

[alexec]

sync-deps.mov

[alexec]

I don't have enough bandwidth to continue this feature. Next steps:

• Someone to volunteer to take ownership (volunteer should be an experienced Argo CD contributor).
• Speak to @alexmt and @jessesuen to get grounding.
• Write an enhancement proposal.
• Take proposal to community meeting for review.

[alexec]

I’m not sure that just the functionality to sync as a DAG is enough (and Argo CD app that is a DAG is a lot like an Argo Workflow!). You probably need a way to view the DAG so that you can debug problematic DAGs.

[crenshaw-dev]

Closing this in favor of #7437 since there's more discussion there, and it's effectively the same request.