Git polling seems to be consuming a lot of resources

[bigjew92]

We currently have two git instances. One Gitlab and the other Github. ArgoCD pulls from both of them as different teams are using different environments.

We hooked ArgoCD up to Gitlab on 11/12/20 and Github a few months before that. Our Gitlab instance is running in AWS with 4 vCPU and 16 GB Ram. Our Github instance is running in GCP with 8 vCPU and 30 GB Ram.

We noticed that Gitlab was suffering and slowing down when browsing around. After some investigation, it looks like Gitlab is slowing down since we started polling from ArgoCD. Looking at the graphs, Github is about 15% vCPU whereas Gitlab is at about 99% vCPU almost constantly. The graph below for Gitlab shows a sudden spike up in mid Janurary, this is when a large amount of apps really started using ArgoCD within Gitlab.

Curious as to if anyone has seen this before or if anyone has noticed better performance with Github over Gitlab? Could it be our instance size not being able to handle the traffic? We have not deployed ArgoCD using "--app-resync", so that is defaulting to 3 minutes.

Gitlab instance CPU graph

Github instance CPU graph

[jannfis]

The amount of requests to your Git provider can be overwhelming with growing number of applications managed by Argo CD while using the default polling interval of 3 minutes. It can easily become several hundreds requests per minute. Argo CD exports Prometheus metrics on the number of Git requests, which are also visible in the example dashboard we provide (in case you use Prometheus/Grafana), in the "Repo Server Stats" section:

While I cannot really comment on the performance differences between GHE and GitLab, I would recommend to enable push events using web hooks and set the polling interval to a rather high value (e.g. 2 hours).

[bigjew92]

Oh thank you! Thanks good to know that the more applications that are in there, the more it polls.

I'm not sure we are using Prometheus, I'd have to double-check if this is set up. Is it something we would've had to enable or it's just on by default? Sorry, I didn't deploy this initially so I'm still figuring out where everything is.

For the webhooks, I was looking at that. Do I need to set that up AND change the --app-resync value? Ideally, we'd keep polling at 3 minutes for Github and change something for Gitlab (possibly though webhooks?).

[jannfis]

Currently, setting the polling interval is global and cannot be set on a per-repository basis. You could open an enhancement proposal so we can see whether it's a feature in high demand of the community.

For Grafana integration, the example dashboard can be found here: https://github.com/argoproj/argo-cd/blob/master/examples/dashboard.json

As for the exported metrics, they are enabled by default and available from the metrics services. I use the following ServiceMonitors for having Prometheus scrape them:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: argocd-api-monitor
  namespace: argocd
spec:
  endpoints:
  - interval: 5s
    port: metrics
  selector:
    matchLabels:
      app.kubernetes.io/name: argocd-server-metrics
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: argocd-controller-monitor
  namespace: argocd
spec:
  endpoints:
  - interval: 5s
    port: metrics
  selector:
    matchLabels:
      app.kubernetes.io/name: argocd-metrics
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: argocd-reposerver-monitor
  namespace: argocd
spec:
  endpoints:
  - interval: 5s
    port: metrics
  selector:
    matchLabels:
      app.kubernetes.io/name: argocd-repo-server-metrics