Declarative setup and configuration of ArgoCD

[mduarte]

I am really impressed with the approach that argo-cd has taken to GitOps. It is in my opinion the best implementation of this methodology so far.

However, the argo-cd setup itself is not declarative. I would expect to be able to setup ArgoCD with a single kubectl apply with all its resources, configuration and application definitions. ArgoCD would then, from those application definitions and confirations do the rest of the work as soon as it would start up.

Ideally it would be possible to have:

1. Declarative way to add the cluster that ArgoCD is in.
   Perhaps we could add the necessary serviceaccount/role/bindings necessary for argocd-manager in a declarative way at the same time we kubectl apply the argocd deployment.
2. Declarative definition of the git repositories.
   Perhaps we would need a CRD for repos? If the repo is private we could refer to a kubernetes secret with the ssh private key of the repo.

Is there plans to head in this direction and avoid the need of any procedural step to setup ArgoCD?

[jessesuen]

@mduarte thanks for the feedback. You bring up an excellent point about the ArgoCD configuration itself not being totally declarative. I think this is important discussion to have and I do think it can improved. Some points to know:

1. Application specs are ultimately in the end, k8s CRD specs, and can be created and updated from a yaml declaratively (as opposed to GUI). The spec is quite simple and just needs a pointer to the source repo and another pointer to the destination cluster/namespace:

spec:
  project: default
  source:
    repoURL: https://github.com/argoproj/argocd-example-apps.git
    targetRevision: HEAD
    path: guestbook
  destination:
    server: https://kubernetes.default.svc
    namespace: default

App specs can be created and updated declaratively using

argocd create -f path/to/argocd-app-spec.yaml --upsert

2. You'll notice in the above command still goes through ArgoCD CLI/API server. Though it's possible to kubectl apply the above spec, it is still preferable to send the spec through the ArgoCD API server as opposed to kubectl apply because (1) the spec is validated there before admittance, (2) authorized based on the auth credentials.

3. Another point is that it's possible to run ArgoCD itself a declarative way, at least partially. In fact, this is how we run ArgoCD here, with many ArgoCD instances for different development groups. Each instance is customized as different environments in ksonnet (this translates to different values.yaml files in helm, or different kustomization.yaml files in kustomize). Whenever we need to make a ArgoCD server configuration change (e.g. service loadbalancer updates, hostname change, RBAC policy, SSO configuration), we make the change in git, and sync the argocd server config changes. This includes any of its own configuration change. ArgoCD API server has a watch on the configmap and rbac configmap, such that that updates to the config are picked up automatically without requiring an API server restart.

NOTE that this only satisfies part of your concern, since application resources are still created imperatively instead of via git, as well as the cluster/repo secrets, which the API server relies on a naming convention due to k8s restrictions on resource names.

4. ArgoCD has a utility, argocd-util which does a complete data dump of the in the ArgoCD RAW API format (with credentials exposed) and is intended to be used either to make periodic backups of the entire argocd state, or to transfer the entire argocd instance into a different environment.

The way that this is used is as follows:

argocd-util export --kubeconfig config1 --namespace argocd > argocd-backup.yaml
argocd-util import --kubeconfig config2 --namespace argocd argocd-backup.yaml

I have also used it like:

argocd-util export --kubeconfig config1 --namespace argocd | argocd-util import --kubeconfig config2 -f -

[mduarte]

@jessesuen thanks for the quick reply.

I will try to reply to your points and give you some context about our setup so perhaps it will make more sense where I am coming from.

1. On my first point above I was actually referring to the fact that one needs to add the cluster where argocd is already running in by using the kubeconfig and running argocd cluster add <context> --in-cluster. A procedural step that seems avoidable in a typical setup where one just uses argocd to deploy applications in the same cluster as it is running and never in other clusters.

2. We are indeed already using CRD for the applications and using kubectl apply to create them (even though not ideal). In fact, one of these applications would then point to a path in the same repository where other applications were being defined.
   Basically the first kubectl apply would serve as a bootstrap application from where all the others would be defined and configured. We are not particularly happy about this solution, it really is fundamentaly a hacky way to get around the need for this procedural step. In this case, we are trying to avoid a procedural step every time a team wants to create a new application. In the GitOps world, the creation of an application to be monitored by our CD tool should itself be simply a PR that would be reviewed and merged by the people who have the permissions to do so. In our approach to GitOps, GitHub and the PR process is being used as the way to review operational changes (which is much easier than reviewing procedural steps to be taken), to audit them (same as before) and to manage access control (by the people who have permissions to merge pull requests).

3. I was not aware of argocd-util but unfortunately it doesn't really address my concerns. One of the main advantages of the GitOps approach is the ability to define the state of the whole cluster in a declarative way. Several tools that before required some sort of procedural step during setup (e.g. grafana) are now going in a direction to support some sort of declarative definition of their setup. Given that the configuration of all these applications can now be defined fully declaratively, one can create a new cluster and get to the same state by applying these resources to the cluster. There's no longer the need to back their configuration, it is fully backed up in GitHub. Ironically the tool we are using to help us with the CD of the GitOps approach is the one that doesn't support this.

That said, we've tried several tools out there for this task and in my opinion ArgoCD is the most promising! I am here giving this feedback to make ArgoCD even better :)

[jessesuen]

On my first point above I was actually referring to the fact that one needs to add the cluster where argocd is already running in by using the kubeconfig and running argocd cluster add --in-cluster. A procedural step that seems avoidable in a typical setup where one just uses argocd to deploy applications in the same cluster as it is running and never in other clusters.

I gave a little background about why it is the way it is, here: #527. Basically it boiled down to being able to run argocd services with a less privileged role, so that ArgoCD does not need to run with cluster-admin privileges as a default behavior, especially for the scenarios where ArgoCD is meant to be managing other clusters (not its own cluster).

That said, you're the second person to find this behavior a bit jarring. I do think there is room for an installation mode to support the in-cluster use case without the procedural step. Let's use #527 to discuss how to accomplish this. In the end, we will likely need to have an installation mode where we expand role of the application-controller to have cluster-admin privileges, and for the deploy process to presume access when deploying to kubernetes.default.svc.

Several tools that before required some sort of procedural step during setup (e.g. grafana) are now going in a direction to support some sort of declarative definition of their setup.
Ironically the tool we are using to help us with the CD of the GitOps approach is the one that doesn't support this.

Believe me, this doesn't sit right with me as well and I think we should find ways to address this. Today's implementation is not friendly to GitOps because of the way we store the credentials with a formulated secret name. The primary reason for the current design is because of the following constraints:

1. we need to store credentials securely (i.e. as a kubernetes secret)
2. In the application spec, sources/destinations should reference actual github URLs and K8s API server URLs, not by some referential kubernetes resource name.
3. kubernetes secret names need to be DNS friendly (cannot be URLs). Labels also have restrictions on what can be in a label value.
4. prefer a O(1) resource lookup when retrieving repo/cluster credentials. Because of K8s' restrictions on secret names and labels, we use a secret name formula.

If we can rework the repo and cluster credential management to satisfy the above requirements, in a more GitOps friendly way (without relying on the current secret naming recipe), I think it would be a huge win.

One question for you is: how are you dealing with kubernetes secrets and GitOps?

In our approach to GitOps, GitHub and the PR process is being used as the way to review operational changes (which is much easier than reviewing procedural steps to be taken), to audit them (same as before) and to manage access control (by the people who have permissions to merge pull requests).

Thanks for the explanation. This really helps us understand the use case better.

[mduarte]

One question for you is: how are you dealing with kubernetes secrets and GitOps?

At the moment we are looking at this solution by bitmani: https://github.com/bitnami-labs/sealed-secrets. However, we are also studying other possibilities.

[jessesuen]

Repasting from issue #527:

After discussion, we agree we can remove the procedural step of adding ArgoCD's own cluster credentials, and have the option of run ArgoCD with clusteradmin privileges, negating the need for the initial bootstrapping step.

[jessesuen]

Capturing some discussion from slack.

The current proposal is that repositories would be configured through the central argocd-configmap. So something like this would exist in the argocd-configmap:

data:
  repositories: |
  # private repository which references a secret
  - url: https://github.com/argoproj/my-private-repo.git
    usernameSecret:
      name: my-github-secret
      key: username
    passwordSecret:
      name: my-github-secret
      key: password

  # public repositories could also added to the repositories list but do not require a reference to a secret.
  # NOTE: it's not actually required for public repos to be inputed into ArgoCD for things to work.
  # The only benefit for adding public repos to ArgoCD, is that it is selectable from the UI.
  - url: https://github.com/argoproj/my-public-repo.git

When a user adds a repository through the API, in the backend, argocd might choose to continue to use the same formula to generate the secret name, but we would not require it like in the current design. The secret would simply need be able to be referenced by it's name in this config file (rather than the formulated name that ArgoCD uses).

[jessesuen]

To handle clusters, we will continue to use a secret to store all cluster information, but we will switch to use a better convention for the secret name, which will be friendlier for humans to create outside of ArgoCD API Server. The convention would simply be to use the hostname of the API server, after stripping out the https://.

If the cluster address has a port, then we would need swap the colon with a dash. Something like:

apiVersion: v1
kind: Secret
metadata:
  name: api-my-cluster-1124127088.us-west-2.elb.amazonaws.com-443

[alexmt]

Implemented