Best practices for syncPolicy for CI/CD

[musabmasood]

Hey folks. I was wondering what are the best practices for setting syncPolicy.automated when it comes to CI/CD for in-house developed services? I have two options in mind:

Option 1 - automated = true

If we do this, the CI/CD pipeline looks something like this:

1. Merge to master
2. Build
3. Push images
4. Update image.tag in deployments repo (triggers sync)
5. As a post-synchook, trigger another workflow somewhere
6. Continue from the other workflow (tests, promote etc)
7. ...

Pros:

• For config only changes (like resource bumps etc), we could simply create PRs and merge to master and argocd syncs it

Cons:

• If we have more workflows after the sync (e.g. integration tests, promotions etc), we can't see them in an end-to-end pipeline
• We need to be careful with CODEOWNERS

Option 1 - automated = false

CI/CD becomes something like this:

1. Merge to master
2. Build
3. Push images
4. Update image.tag in deployments repo
5. Run argocd app sync and wait
6. Run integration tests
7. Promote
8. ...

Pros:

• One pipeline with end-to-end view of the complete deployment
• We don't need to worry too much about CODEOWNERS as sync permission defines who/when/what gets deployed

Cons:

• Config only changes need manual sync

What are your thoughts and best practices?

[kostis-codefresh]

I tend for favor the second option if you want an end-to-end view.

You can easily fix the single con by creating a CI pipeline that monitors the deployments repo and runs "argo sync" when ANY change happens. This way all deployments are automated regardless of the change (source code or config code change).

[jemag]

I agree that Option 2 is probably easier to get an end-to-end view, although it seems at this point you are more doing regular CI/CD than taking full advantage of what GitOps has to offer. Especially, since you are not taking advantage of the selfHeal capability. Also, I am guessing you have to allow network traffic between your CI agent and your Argocd instance (for argocd app sync), which may not be necessary otherwise.

I think having automated = true is probably the way to go, but it does make it harder to get that end-to-end view. I am still investigating the best way to do it myself, but would be interested to hear other opinions. Perhaps you could have a general pipeline for the whole process and have your workflows communicate their end results to that general pipeline (e.g.: if integrations tests succeeded, etc.).

[musabmasood]

Option 2 kind of defeats the whole purpose of GitOps and makes you fall back on traditional CI/CD approach. Its more or less same as helm template | kubectl apply -f - in your CI pipeline. The advantage of course is the end to end view.

We prefer to keep autosync on, but that means multiple asynchronous workflows that are not linear. These workflows could also be in different tools (e.g. Jenkins, CircleCI etc) so there is a lack of visibility. We are trying to address some of these issues with pushing events to Datadog etc.