Panic Error Being Thrown

[m477r1x]

For some reason we are seeing errors being thrown on the image automation controller for one of our imageupdateautomations the error is:

{"level":"error","ts":"2023-05-25T13:40:54.637Z","msg":"Reconciler error","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"company","namespace":"company"},"namespace":"company","name":"company","reconcileID":"56bd72d1-8fdf-4945-b083-32c3a0e4ae6d","error":"panic: runtime error: invalid memory address or nil pointer dereference [recovered]"}

We are going to try removing some unnecassary yaml files from our github repo and see if that resolves the error, but in the mean-time @stefanprodan advised me to log this as an issue.

UPDATE:

We are seeing a penic error on a different namespace as well, but here is the full trace version of the error:

Trace Output

matt@AMS-LMPMLO01 tf-eng-infra-prod % k logs image-automation-controller-6b9b8ff777-4rk8q | grep d866a4c7-7a91-40f0-8cd4-4dfe609c6d96
{"level":"debug","ts":"2023-05-30T12:44:18.357Z","msg":"fetching git repository","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"company-web","namespace":"company-web"},"namespace":"company-web","name":"company-web","reconcileID":"d866a4c7-7a91-40f0-8cd4-4dfe609c6d96","gitrepository":{"name":"flux-system","namespace":"flux-system"}}
{"level":"Level(-2)","ts":"2023-05-30T12:44:18.357Z","msg":"using git repository ref from .spec.git.checkout","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"company-web","namespace":"company-web"},"namespace":"company-web","name":"company-web","reconcileID":"d866a4c7-7a91-40f0-8cd4-4dfe609c6d96","ref":{"branch":"main"}}
{"level":"Level(-2)","ts":"2023-05-30T12:44:18.357Z","msg":"using push branch from .spec.push.branch","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"company-web","namespace":"company-web"},"namespace":"company-web","name":"company-web","reconcileID":"d866a4c7-7a91-40f0-8cd4-4dfe609c6d96","branch":"main"}
{"level":"debug","ts":"2023-05-30T12:44:18.357Z","msg":"attempting to clone git repository","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"company-web","namespace":"company-web"},"namespace":"company-web","name":"company-web","reconcileID":"d866a4c7-7a91-40f0-8cd4-4dfe609c6d96","gitrepository":{"name":"flux-system","namespace":"flux-system"},"ref":{"branch":"main"},"working":"/tmp/flux-system-flux-system2903133355"}
{"level":"Level(-2)","ts":"2023-05-30T12:44:20.932Z","msg":"adjusting update path according to .spec.update.path","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"company-web","namespace":"company-web"},"namespace":"company-web","name":"company-web","reconcileID":"d866a4c7-7a91-40f0-8cd4-4dfe609c6d96","base":"/tmp/flux-system-flux-system2903133355","spec-path":"./apps/cnp-eu-dev/company-web/cms"}
{"level":"debug","ts":"2023-05-30T12:44:20.932Z","msg":"updating with setters according to image policies","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"company-web","namespace":"company-web"},"namespace":"company-web","name":"company-web","reconcileID":"d866a4c7-7a91-40f0-8cd4-4dfe609c6d96","count":1,"manifests-path":"/tmp/flux-system-flux-system2903133355/apps/cnp-eu-dev/company-web/cms"}
{"level":"Level(-2)","ts":"2023-05-30T12:44:20.932Z","msg":"found policy","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"company-web","namespace":"company-web"},"namespace":"company-web","name":"company-web","reconcileID":"d866a4c7-7a91-40f0-8cd4-4dfe609c6d96","namespace":"company-web","name":"company-web","latest-image":"837520006200.dkr.ecr.eu-west-2.amazonaws.com/company-web-dev:1.2.2"}
{"level":"Level(-2)","ts":"2023-05-30T12:44:20.932Z","msg":"adding setter","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"company-web","namespace":"company-web"},"namespace":"company-web","name":"company-web","reconcileID":"d866a4c7-7a91-40f0-8cd4-4dfe609c6d96","name":"company-web:company-web"}
{"level":"Level(-2)","ts":"2023-05-30T12:44:20.932Z","msg":"adding setter","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"company-web","namespace":"company-web"},"namespace":"company-web","name":"company-web","reconcileID":"d866a4c7-7a91-40f0-8cd4-4dfe609c6d96","name":"company-web:company-web:tag"}
{"level":"Level(-2)","ts":"2023-05-30T12:44:20.932Z","msg":"adding setter","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"company-web","namespace":"company-web"},"namespace":"company-web","name":"company-web","reconcileID":"d866a4c7-7a91-40f0-8cd4-4dfe609c6d96","name":"company-web:company-web:name"}
{"level":"Level(-2)","ts":"2023-05-30T12:44:20.932Z","msg":"scanning files","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"company-web","namespace":"company-web"},"namespace":"company-web","name":"company-web","reconcileID":"d866a4c7-7a91-40f0-8cd4-4dfe609c6d96","path":"/tmp/flux-system-flux-system2903133355/apps/cnp-eu-dev/company-web/cms","token":"\"$imagepolicy\""}
{"level":"debug","ts":"2023-05-30T12:44:20.932Z","msg":"ran updates to working dir","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"company-web","namespace":"company-web"},"namespace":"company-web","name":"company-web","reconcileID":"d866a4c7-7a91-40f0-8cd4-4dfe609c6d96","working":"/tmp/flux-system-flux-system2903133355"}
{"level":"error","ts":"2023-05-30T12:44:20.948Z","msg":"Reconciler error","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"company-web","namespace":"company-web"},"namespace":"company-web","name":"company-web","reconcileID":"d866a4c7-7a91-40f0-8cd4-4dfe609c6d96","error":"panic: runtime error: invalid memory address or nil pointer dereference [recovered]","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:324\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226"}

Flux Check Output

``` matt@AMS-LMPMLO01 tf-eng-infra-prod % flux check ► checking prerequisites ✗ flux 0.41.2 <2.0.0-rc.4 (new version is available, please upgrade) ✔ Kubernetes 1.27.1-eks-2f008fe >=1.20.6-0 ► checking controllers ✔ helm-controller: deployment ready ► ghcr.io/fluxcd/helm-controller:v0.34.0 ✔ image-automation-controller: deployment ready ► ghcr.io/fluxcd/image-automation-controller:v0.34.0 ✔ image-reflector-controller: deployment ready ► ghcr.io/fluxcd/image-reflector-controller:v0.28.0 ✔ kustomize-controller: deployment ready ► ghcr.io/fluxcd/kustomize-controller:v1.0.0-rc.4 ✔ notification-controller: deployment ready ► ghcr.io/fluxcd/notification-controller:v1.0.0-rc.4 ✔ source-controller: deployment ready ► ghcr.io/fluxcd/source-controller:v1.0.0-rc.4 ► checking crds ✔ alerts.notification.toolkit.fluxcd.io/v1beta2 ✔ buckets.source.toolkit.fluxcd.io/v1beta2 ✔ gitrepositories.source.toolkit.fluxcd.io/v1 ✔ helmcharts.source.toolkit.fluxcd.io/v1beta2 ✔ helmreleases.helm.toolkit.fluxcd.io/v2beta1 ✔ helmrepositories.source.toolkit.fluxcd.io/v1beta2 ✔ imagepolicies.image.toolkit.fluxcd.io/v1beta2 ✔ imagerepositories.image.toolkit.fluxcd.io/v1beta2 ✔ imageupdateautomations.image.toolkit.fluxcd.io/v1beta1 ✔ kustomizations.kustomize.toolkit.fluxcd.io/v1 ✔ ocirepositories.source.toolkit.fluxcd.io/v1beta2 ✔ providers.notification.toolkit.fluxcd.io/v1beta2 ✔ receivers.notification.toolkit.fluxcd.io/v1 ✔ all checks passed ```

Larger chunk of logs without any grepping

{"level":"error","ts":"2023-06-06T12:41:43.865Z","msg":"Reconciler error","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"dare-web","namespace":"dare-web"},"namespace":"dare-web","name":"dare-web","reconcileID":"dbbac68e-d8a2-4aa6-ad6e-8d9a9248ece6","error":"panic: runtime error: invalid memory address or nil pointer dereference [recovered]"}
{"level":"error","ts":"2023-06-06T12:44:11.481Z","logger":"runtime","msg":"Observed a panic: \"invalid memory address or nil pointer dereference\" (runtime error: invalid memory address or nil pointer dereference)\ngoroutine 77 [running]:\nk8s.io/apimachinery/pkg/util/runtime.logPanic({0x3fc340?, 0x2ea8600})\n\tk8s.io/apimachinery@v0.27.2/pkg/util/runtime/runtime.go:75 +0x99\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()\n\tsigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:107 +0xc5\npanic({0x3fc340, 0x2ea8600})\n\truntime/panic.go:884 +0x213\ngithub.com/fluxcd/image-automation-controller/internal/controller.(*ImageUpdateAutomationReconciler).getSigningEntity(_, {_, _}, {{{0x266dc2, 0x15}, {0xc0001190a0, 0x1f}}, {{0xc0008e9a60, 0x6}, {0x0, ...}, ...}, ...})\n\tgithub.com/fluxcd/image-automation-controller/internal/controller/imageupdateautomation_controller.go:577 +0x29a\ngithub.com/fluxcd/image-automation-controller/internal/controller.(*ImageUpdateAutomationReconciler).Reconcile(0xc000065680, {0x97ea20, 0xc0000f4060}, {{{0xc0008e8666?, 0x0?}, {0xc0008e8660?, 0x180a2c7?}}})\n\tgithub.com/fluxcd/image-automation-controller/internal/controller/imageupdateautomation_controller.go:358 +0x2ad5\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x97ea20?, {0x97ea20?, 0xc0000f4060?}, {{{0xc0008e8666?, 0x32db00?}, {0xc0008e8660?, 0x9698e8?}}})\n\tsigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:118 +0xc8\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000329400, {0x97e978, 0xc000326960}, {0x4a8200?, 0xc000583a60?})\n\tsigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:314 +0x377\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000329400, {0x97e978, 0xc000326960})\n\tsigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265 +0x1d9\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()\n\tsigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226 +0x85\ncreated by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2\n\tsigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:222 +0x587\n"}
{"level":"error","ts":"2023-06-06T12:44:11.481Z","msg":"Reconciler error","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"trainr","namespace":"trainr"},"namespace":"trainr","name":"trainr","reconcileID":"fad3f689-3f0c-49da-b7c8-34f226dc9178","error":"panic: runtime error: invalid memory address or nil pointer dereference [recovered]"}
{"level":"error","ts":"2023-06-06T12:56:44.000Z","logger":"runtime","msg":"Observed a panic: \"invalid memory address or nil pointer dereference\" (runtime error: invalid memory address or nil pointer dereference)\ngoroutine 78 [running]:\nk8s.io/apimachinery/pkg/util/runtime.logPanic({0x3fc340?, 0x2ea8600})\n\tk8s.io/apimachinery@v0.27.2/pkg/util/runtime/runtime.go:75 +0x99\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()\n\tsigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:107 +0xc5\npanic({0x3fc340, 0x2ea8600})\n\truntime/panic.go:884 +0x213\ngithub.com/fluxcd/image-automation-controller/internal/controller.(*ImageUpdateAutomationReconciler).getSigningEntity(_, {_, _}, {{{0x266dc2, 0x15}, {0xc00005c600, 0x1f}}, {{0xc000062f68, 0x8}, {0x0, ...}, ...}, ...})\n\tgithub.com/fluxcd/image-automation-controller/internal/controller/imageupdateautomation_controller.go:577 +0x29a\ngithub.com/fluxcd/image-automation-controller/internal/controller.(*ImageUpdateAutomationReconciler).Reconcile(0xc000065680, {0x97ea20, 0xc0008be900}, {{{0xc000062f70?, 0x0?}, {0xc000062f68?, 0x180a2c7?}}})\n\tgithub.com/fluxcd/image-automation-controller/internal/controller/imageupdateautomation_controller.go:358 +0x2ad5\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x97ea20?, {0x97ea20?, 0xc0008be900?}, {{{0xc000062f70?, 0x32db00?}, {0xc000062f68?, 0x9698e8?}}})\n\tsigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:118 +0xc8\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000329400, {0x97e978, 0xc000326960}, {0x4a8200?, 0xc0004b23c0?})\n\tsigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:314 +0x377\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000329400, {0x97e978, 0xc000326960})\n\tsigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265 +0x1d9\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()\n\tsigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226 +0x85\ncreated by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2\n\tsigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:222 +0x587\n"}
{"level":"error","ts":"2023-06-06T12:56:44.000Z","msg":"Reconciler error","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"insights","namespace":"insights"},"namespace":"insights","name":"insights","reconcileID":"cdfee0c3-b683-47ed-bae7-aebbbf2c04e8","error":"panic: runtime error: invalid memory address or nil pointer dereference [recovered]"}
{"level":"error","ts":"2023-06-06T12:56:45.920Z","logger":"runtime","msg":"Observed a panic: \"invalid memory address or nil pointer dereference\" (runtime error: invalid memory address or nil pointer dereference)\ngoroutine 79 [running]:\nk8s.io/apimachinery/pkg/util/runtime.logPanic({0x3fc340?, 0x2ea8600})\n\tk8s.io/apimachinery@v0.27.2/pkg/util/runtime/runtime.go:75 +0x99\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()\n\tsigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:107 +0xc5\npanic({0x3fc340, 0x2ea8600})\n\truntime/panic.go:884 +0x213\ngithub.com/fluxcd/image-automation-controller/internal/controller.(*ImageUpdateAutomationReconciler).getSigningEntity(_, {_, _}, {{{0x266dc2, 0x15}, {0xc000aec000, 0x1f}}, {{0xc000062e10, 0x8}, {0x0, ...}, ...}, ...})\n\tgithub.com/fluxcd/image-automation-controller/internal/controller/imageupdateautomation_controller.go:577 +0x29a\ngithub.com/fluxcd/image-automation-controller/internal/controller.(*ImageUpdateAutomationReconciler).Reconcile(0xc000065680, {0x97ea20, 0xc000fac1b0}, {{{0xc000062e18?, 0x0?}, {0xc000062e10?, 0x180a2c7?}}})\n\tgithub.com/fluxcd/image-automation-controller/internal/controller/imageupdateautomation_controller.go:358 +0x2ad5\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x97ea20?, {0x97ea20?, 0xc000fac1b0?}, {{{0xc000062e18?, 0x32db00?}, {0xc000062e10?, 0x9698e8?}}})\n\tsigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:118 +0xc8\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000329400, {0x97e978, 0xc000326960}, {0x4a8200?, 0xc00037c140?})\n\tsigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:314 +0x377\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000329400, {0x97e978, 0xc000326960})\n\tsigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265 +0x1d9\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()\n\tsigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226 +0x85\ncreated by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2\n\tsigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:222 +0x587\n"}
{"level":"error","ts":"2023-06-06T12:56:45.920Z","msg":"Reconciler error","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"dare-web","namespace":"dare-web"},"namespace":"dare-web","name":"dare-web","reconcileID":"7a93209a-79bf-417a-9f2d-aee00d1b6a71","error":"panic: runtime error: invalid memory address or nil pointer dereference [recovered]"}
{"level":"error","ts":"2023-06-06T12:59:14.083Z","logger":"runtime","msg":"Observed a panic: \"invalid memory address or nil pointer dereference\" (runtime error: invalid memory address or nil pointer dereference)\ngoroutine 76 [running]:\nk8s.io/apimachinery/pkg/util/runtime.logPanic({0x3fc340?, 0x2ea8600})\n\tk8s.io/apimachinery@v0.27.2/pkg/util/runtime/runtime.go:75 +0x99\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()\n\tsigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:107 +0xc5\npanic({0x3fc340, 0x2ea8600})\n\truntime/panic.go:884 +0x213\ngithub.com/fluxcd/image-automation-controller/internal/controller.(*ImageUpdateAutomationReconciler).getSigningEntity(_, {_, _}, {{{0x266dc2, 0x15}, {0xc000a4e540, 0x1f}}, {{0xc0008e9a60, 0x6}, {0x0, ...}, ...}, ...})\n\tgithub.com/fluxcd/image-automation-controller/internal/controller/imageupdateautomation_controller.go:577 +0x29a\ngithub.com/fluxcd/image-automation-controller/internal/controller.(*ImageUpdateAutomationReconciler).Reconcile(0xc000065680, {0x97ea20, 0xc000e9b1a0}, {{{0xc0008e8666?, 0x0?}, {0xc0008e8660?, 0x180a2c7?}}})\n\tgithub.com/fluxcd/image-automation-controller/internal/controller/imageupdateautomation_controller.go:358 +0x2ad5\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x97ea20?, {0x97ea20?, 0xc000e9b1a0?}, {{{0xc0008e8666?, 0x32db00?}, {0xc0008e8660?, 0x9698e8?}}})\n\tsigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:118 +0xc8\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000329400, {0x97e978, 0xc000326960}, {0x4a8200?, 0xc000583980?})\n\tsigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:314 +0x377\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000329400, {0x97e978, 0xc000326960})\n\tsigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265 +0x1d9\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()\n\tsigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226 +0x85\ncreated by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2\n\tsigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:222 +0x587\n"}
{"level":"error","ts":"2023-06-06T12:59:14.083Z","msg":"Reconciler error","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"trainr","namespace":"trainr"},"namespace":"trainr","name":"trainr","reconcileID":"b065606a-a247-424b-88b4-bc8dd6eed3b9","error":"panic: runtime error: invalid memory address or nil pointer dereference [recovered]"}

[makkes]

I have a hypothesis where this might come from but I need your help to verify it:

Do you have .spec.git.commit.signingKey set?

Would you mind posting the status part of your ImageUpdateAutomation object here?

[hiddeco]

The missing return is likely not the culprit here, although I observed the same while looking at the log trace. As shared on Slack, the trigger seems to be change of path.

[m477r1x]

@makkes

I have a hypothesis where this might come from but I need your help to verify it:

Do you have .spec.git.commit.signingKey set?

Would you mind posting the status part of your ImageUpdateAutomation object here?

Yes, we do have the signing key set, here is the entire output of the resource

k get imageupdateuatomation -o yaml

apiVersion: v1
items:
- apiVersion: image.toolkit.fluxcd.io/v1beta1
  kind: ImageUpdateAutomation
  metadata:
    creationTimestamp: "2023-05-10T10:33:15Z"
    finalizers:
    - finalizers.fluxcd.io
    generation: 2
    labels:
      environment: dev
      kustomize.toolkit.fluxcd.io/name: apps-company-web
      kustomize.toolkit.fluxcd.io/namespace: company-web
    name: company-web
    namespace: company-web
    resourceVersion: "178133142"
    uid: 49d4d9f3-f892-4b2c-a6af-e07c28c0710c
  spec:
    git:
      checkout:
        ref:
          branch: main
      commit:
        author:
          email: fluxbot@company.global
          name: fluxbot
        messageTemplate: '{{range .Updated.Images}}{{println .}}{{end}}'
        signingKey:
          secretRef:
            name: flux-gpg-key
      push:
        branch: main
    interval: 5m0s
    sourceRef:
      kind: GitRepository
      name: flux-system
      namespace: flux-system
    update:
      path: ./apps/cnp-eu-dev/company-web/cms
      strategy: Setters
  status:
    conditions:
    - lastTransitionTime: "2023-05-25T13:35:55Z"
      message: no updates made
      reason: ReconciliationSucceeded
      status: "True"
      type: Ready
    lastAutomationRunTime: "2023-05-25T13:35:52Z"
    observedGeneration: 2
kind: List
metadata:
  resourceVersion: ""

[hiddeco]

If that's the case, I suspect the image mentioned in #527 will tell you the core of the issue. Although I would expect this same information to be available in the events for the affected resource.

[aryan9600]

could you please post the logs without the grep? i believe the grep is omitting the actual stack trace that should get logged on a panic.

[aryan9600]

could you please try ghcr.io/fluxcd/image-automation-controller:rc-faf265e7 and see if it prevents the panic from occurring? thanks