Source controller is unstable

[ANKAMBLE]

Describe the bug

We are seeing source controller pod is getting restarted frequently every hour and it take 15-30 mins to recover from below error
failed to download artifact, error: GET http://source-controller.flux-system.svc.cluster.local./gitrepository/flux-system/ocp-fleet/2998cb88140643569d3264e00b51a0ac552d856f.tar.gz giving up after 10 attempt(s): Get "http://source-controller.flux-system.svc.cluster.local./gitrepository/flux-system/ocp-fleet/2998cb88140643569d3264e00b51a0ac552d856f.tar.gz": dial tcp 172.30.237.29:80: i/o timeout

Steps to reproduce

We have flux muti tenancy enabled on Opensshift cluster with multiple namespace. We created below for repositories for maintaining infrastructure and application release files and helm charts

1. ocp-cluster repo - contains flux system cluster definition
2. ocp-fleet repo- contains infrastructure and tenants definitions
3. ocp-tenant repo - contains specific namespace related definitions
4. ocp-app repo - contains application specific helm charts

Expected behavior

flux should be able to connect to four repositories defined and update the cluster/namespace accordingly on each attempt, randomly we are seeing failures with below errors
failed to download artifact, error: GET http://source-controller.flux-system.svc.cluster.local./gitrepository/flux-system/ocp-fleet/2998cb88140643569d3264e00b51a0ac552d856f.tar.gz giving up after 10 attempt(s): Get "http://source-controller.flux-system.svc.cluster.local./gitrepository/flux-system/ocp-fleet/2998cb88140643569d3264e00b51a0ac552d856f.tar.gz": dial tcp 172.30.237.29:80: i/o timeout

Screenshots and recordings

OS / Distro

RHCOS, Openshift 4.6

Flux version

v0.17.2

Flux check

Git provider

No response

Container Registry provider

No response

Additional context

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[stefanprodan]

What's the reason for source-controller restarts? If it's OOM you should increase the memory limits.

[ANKAMBLE]

how can we check if it is failing due to OOM, i don't see it is showing OOM errors in logs. I do see pod memory utilization reaching around 98% memory utilization sometimes

[stefanprodan]

Please see https://sysdig.com/blog/troubleshoot-kubernetes-oom/ If the memory goes to 98% then you need to increase the limit, setting it to 2Gi should fix it.

[chanwit]

@ANKAMBLE my question might not be directly related to your issue.

did you use flux bootstrap or install Flux via the Operator Hub?

[ANKAMBLE]

we used flux bootstrap, the issue got resolved after increasing the memory

[chanwit]

@ANKAMBLE glad to know your problem got solved.

[stefanprodan]

source-controller memory usage is governed by the number of Helm charts it needs to build. The default limit of 1Gi works fine for charts that are already bundled (originating from Helm repositories). For charts in Git, users have to increase the limit. I think we could add this to the FAQ page.

[gecube]

Is it possible to add VPA object for governing the source controller memory size limit?