Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Crash "panic: invalid pointer handle: loop" with EXPERIMENTAL_GIT_TRANSPORT = true #358

Closed
johnp-stella opened this issue Apr 28, 2022 · 6 comments
Closed

Crash "panic: invalid pointer handle: loop" with EXPERIMENTAL_GIT_TRANSPORT = true #358

johnp-stella opened this issue Apr 28, 2022 · 6 comments
Assignees
@pjbgf
Labels
bug Something isn't working
Milestone
GA

Comments

@johnp-stella
Copy link

When trying out the EXPERIMENTAL_GIT_TRANSPORT we are seeing a crash loop. "panic: invalid pointer handle: loop" when pulling from github.

Have tried the experimental image from #339 without any success.

❯ flux version
flux: v0.27.4
helm-controller: v0.20.1
image-automation-controller: v0.22.0
image-reflector-controller: v0.17.2
kustomize-controller: v0.24.3
notification-controller: v0.23.4
source-controller: v0.24.1
image-automation-controller debug logs ( 
{"level":"Level(-2)","ts":"2022-04-28T04:36:51.974Z","logger":"managed-transport","msg":"[ssh]: cache hit","remoteAddress":"github.com:22"}
{"level":"Level(-2)","ts":"2022-04-28T04:36:51.974Z","logger":"managed-transport","msg":"[ssh]: creating new ssh session"}
{"level":"Level(-2)","ts":"2022-04-28T04:36:51.974Z","logger":"controller.imageupdateautomation","msg":"reading file","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageUpdateAutomation","name":"image-c","namespace":"flux-system","path":"image-a.yaml"}
{"level":"Level(-2)","ts":"2022-04-28T04:36:51.976Z","logger":"controller.imageupdateautomation","msg":"found schema extension","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageUpdateAutomation","name":"image-c","namespace":"flux-system","path":".metadata.labels.tags.datadoghq.com/version"}
{"level":"Level(-2)","ts":"2022-04-28T04:36:51.976Z","logger":"controller.imageupdateautomation","msg":"applying setter","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageUpdateAutomation","name":"image-c","namespace":"flux-system","setter":"flux-system:image-d:tag","old":"develop-22a966ba0a975e8f2858d88a8a5c2d216fbb742b-1943","new":"develop-22a966ba0a975e8f2858d88a8a5c2d216fbb742b-1943"}
{"level":"Level(-2)","ts":"2022-04-28T04:36:51.980Z","logger":"controller.imageupdateautomation","msg":"found schema extension","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageUpdateAutomation","name":"image-c","namespace":"flux-system","path":".spec.template.metadata.labels.tags.datadoghq.com/version"}
{"level":"Level(-2)","ts":"2022-04-28T04:36:51.980Z","logger":"controller.imageupdateautomation","msg":"applying setter","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageUpdateAutomation","name":"image-c","namespace":"flux-system","setter":"flux-system:image-d:tag","old":"develop-22a966ba0a975e8f2858d88a8a5c2d216fbb742b-1943","new":"develop-22a966ba0a975e8f2858d88a8a5c2d216fbb742b-1943"}
{"level":"Level(-2)","ts":"2022-04-28T04:36:51.981Z","logger":"controller.imageupdateautomation","msg":"found schema extension","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageUpdateAutomation","name":"image-c","namespace":"flux-system","path":".spec.template.spec.initContainers.image"}
{"level":"Level(-2)","ts":"2022-04-28T04:36:51.981Z","logger":"controller.imageupdateautomation","msg":"applying setter","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageUpdateAutomation","name":"image-c","namespace":"flux-system","setter":"flux-system:image-d","old":"xxxx.dkr.ecr.us-east-2.amazonaws.com/image-d:develop-22a966ba0a975e8f2858d88a8a5c2d216fbb742b-1943","new":"xxxx.dkr.ecr.us-east-2.amazonaws.com/image-d:develop-22a966ba0a975e8f2858d88a8a5c2d216fbb742b-1943"}
{"level":"Level(-2)","ts":"2022-04-28T04:36:51.981Z","logger":"controller.imageupdateautomation","msg":"found schema extension","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageUpdateAutomation","name":"image-c","namespace":"flux-system","path":".spec.template.spec.containers.image"}
{"level":"Level(-2)","ts":"2022-04-28T04:36:51.981Z","logger":"controller.imageupdateautomation","msg":"applying setter","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageUpdateAutomation","name":"image-c","namespace":"flux-system","setter":"flux-system:image-d","old":"xxxx.dkr.ecr.us-east-2.amazonaws.com/image-d:develop-22a966ba0a975e8f2858d88a8a5c2d216fbb742b-1943","new":"xxxx.dkr.ecr.us-east-2.amazonaws.com/image-d:develop-22a966ba0a975e8f2858d88a8a5c2d216fbb742b-1943"}
{"level":"Level(-2)","ts":"2022-04-28T04:36:51.981Z","logger":"controller.imageupdateautomation","msg":"found schema extension","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageUpdateAutomation","name":"image-c","namespace":"flux-system","path":".metadata.labels.tags.datadoghq.com/version"}
{"level":"Level(-2)","ts":"2022-04-28T04:36:51.981Z","logger":"controller.imageupdateautomation","msg":"applying setter","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageUpdateAutomation","name":"image-c","namespace":"flux-system","setter":"flux-system:image-c-preview:tag","old":"develop-22a966ba0a975e8f2858d88a8a5c2d216fbb742b-1943","new":"develop-22a966ba0a975e8f2858d88a8a5c2d216fbb742b-1943"}
{"level":"Level(-2)","ts":"2022-04-28T04:36:51.981Z","logger":"controller.imageupdateautomation","msg":"found schema extension","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageUpdateAutomation","name":"image-c","namespace":"flux-system","path":".spec.template.metadata.labels.tags.datadoghq.com/version"}
{"level":"Level(-2)","ts":"2022-04-28T04:36:51.981Z","logger":"controller.imageupdateautomation","msg":"applying setter","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageUpdateAutomation","name":"image-c","namespace":"flux-system","setter":"flux-system:image-c-preview:tag","old":"develop-22a966ba0a975e8f2858d88a8a5c2d216fbb742b-1943","new":"develop-22a966ba0a975e8f2858d88a8a5c2d216fbb742b-1943"}
{"level":"Level(-2)","ts":"2022-04-28T04:36:51.981Z","logger":"controller.imageupdateautomation","msg":"found schema extension","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageUpdateAutomation","name":"image-c","namespace":"flux-system","path":".spec.template.spec.containers.image"}
{"level":"Level(-2)","ts":"2022-04-28T04:36:51.981Z","logger":"controller.imageupdateautomation","msg":"applying setter","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageUpdateAutomation","name":"image-c","namespace":"flux-system","setter":"flux-system:image-c-preview","old":"xxxx.dkr.ecr.us-east-2.amazonaws.com/image-c-dev-preview:develop-22a966ba0a975e8f2858d88a8a5c2d216fbb742b-1943","new":"xxxx.dkr.ecr.us-east-2.amazonaws.com/image-c-sandbox-preview:develop-22a966ba0a975e8f2858d88a8a5c2d216fbb742b-1943"}
{"level":"Level(-2)","ts":"2022-04-28T04:36:51.982Z","logger":"controller.imageupdateautomation","msg":"found schema extension","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageUpdateAutomation","name":"image-c","namespace":"flux-system","path":".metadata.labels.tags.datadoghq.com/version"}
{"level":"Level(-2)","ts":"2022-04-28T04:36:51.982Z","logger":"controller.imageupdateautomation","msg":"applying setter","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageUpdateAutomation","name":"image-c","namespace":"flux-system","setter":"flux-system:image-c:tag","old":"develop-22a966ba0a975e8f2858d88a8a5c2d216fbb742b-1943","new":"develop-22a966ba0a975e8f2858d88a8a5c2d216fbb742b-1943"}
{"level":"Level(-2)","ts":"2022-04-28T04:36:51.982Z","logger":"controller.imageupdateautomation","msg":"found schema extension","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageUpdateAutomation","name":"image-c","namespace":"flux-system","path":".spec.template.metadata.labels.tags.datadoghq.com/version"}
{"level":"Level(-2)","ts":"2022-04-28T04:36:51.982Z","logger":"controller.imageupdateautomation","msg":"applying setter","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageUpdateAutomation","name":"image-c","namespace":"flux-system","setter":"flux-system:image-c:tag","old":"develop-22a966ba0a975e8f2858d88a8a5c2d216fbb742b-1943","new":"develop-22a966ba0a975e8f2858d88a8a5c2d216fbb742b-1943"}
{"level":"Level(-2)","ts":"2022-04-28T04:36:51.982Z","logger":"controller.imageupdateautomation","msg":"found schema extension","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageUpdateAutomation","name":"image-c","namespace":"flux-system","path":".spec.template.spec.containers.image"}
{"level":"Level(-2)","ts":"2022-04-28T04:36:51.982Z","logger":"controller.imageupdateautomation","msg":"applying setter","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageUpdateAutomation","name":"image-c","namespace":"flux-system","setter":"flux-system:image-c","old":"xxxx.dkr.ecr.us-east-2.amazonaws.com/image-c-dev:develop-22a966ba0a975e8f2858d88a8a5c2d216fbb742b-1943","new":"xxxx.dkr.ecr.us-east-2.amazonaws.com/image-c-sandbox:develop-22a966ba0a975e8f2858d88a8a5c2d216fbb742b-1943"}
{"level":"Level(-2)","ts":"2022-04-28T04:36:51.982Z","logger":"controller.imageupdateautomation","msg":"found schema extension","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageUpdateAutomation","name":"image-c","namespace":"flux-system","path":".metadata.labels.tags.datadoghq.com/version"}
{"level":"Level(-2)","ts":"2022-04-28T04:36:51.982Z","logger":"controller.imageupdateautomation","msg":"applying setter","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageUpdateAutomation","name":"image-c","namespace":"flux-system","setter":"flux-system:image-e:tag","old":"develop-22a966ba0a975e8f2858d88a8a5c2d216fbb742b-1943","new":"develop-22a966ba0a975e8f2858d88a8a5c2d216fbb742b-1943"}
{"level":"Level(-2)","ts":"2022-04-28T04:36:51.983Z","logger":"controller.imageupdateautomation","msg":"found schema extension","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageUpdateAutomation","name":"image-c","namespace":"flux-system","path":".spec.template.metadata.labels.tags.datadoghq.com/version"}
{"level":"Level(-2)","ts":"2022-04-28T04:36:51.983Z","logger":"controller.imageupdateautomation","msg":"applying setter","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageUpdateAutomation","name":"image-c","namespace":"flux-system","setter":"flux-system:image-e:tag","old":"develop-22a966ba0a975e8f2858d88a8a5c2d216fbb742b-1943","new":"develop-22a966ba0a975e8f2858d88a8a5c2d216fbb742b-1943"}
{"level":"Level(-2)","ts":"2022-04-28T04:36:51.983Z","logger":"controller.imageupdateautomation","msg":"found schema extension","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageUpdateAutomation","name":"image-c","namespace":"flux-system","path":".spec.template.spec.containers.image"}
{"level":"Level(-2)","ts":"2022-04-28T04:36:51.983Z","logger":"controller.imageupdateautomation","msg":"applying setter","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageUpdateAutomation","name":"image-c","namespace":"flux-system","setter":"flux-system:image-e","old":"xxxx.dkr.ecr.us-east-2.amazonaws.com/image-e:develop-22a966ba0a975e8f2858d88a8a5c2d216fbb742b-1943","new":"xxxx.dkr.ecr.us-east-2.amazonaws.com/image-e:develop-22a966ba0a975e8f2858d88a8a5c2d216fbb742b-1943"}
{"level":"Level(-2)","ts":"2022-04-28T04:36:51.983Z","logger":"controller.imageupdateautomation","msg":"found schema extension","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageUpdateAutomation","name":"image-c","namespace":"flux-system","path":".metadata.labels.tags.datadoghq.com/version"}
{"level":"Level(-2)","ts":"2022-04-28T04:36:51.983Z","logger":"controller.imageupdateautomation","msg":"applying setter","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageUpdateAutomation","name":"image-c","namespace":"flux-system","setter":"flux-system:image-b-preview:tag","old":"develop-22a966ba0a975e8f2858d88a8a5c2d216fbb742b-1943","new":"develop-22a966ba0a975e8f2858d88a8a5c2d216fbb742b-1943"}
{"level":"Level(-2)","ts":"2022-04-28T04:36:51.983Z","logger":"controller.imageupdateautomation","msg":"found schema extension","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageUpdateAutomation","name":"image-c","namespace":"flux-system","path":".spec.template.spec.containers.image"}
{"level":"Level(-2)","ts":"2022-04-28T04:36:51.983Z","logger":"controller.imageupdateautomation","msg":"applying setter","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageUpdateAutomation","name":"image-c","namespace":"flux-system","setter":"flux-system:image-b-preview","old":"xxxx.dkr.ecr.us-east-2.amazonaws.com/image-b-dev-preview:develop-22a966ba0a975e8f2858d88a8a5c2d216fbb742b-1943","new":"xxxx.dkr.ecr.us-east-2.amazonaws.com/image-b-sandbox-preview:develop-22a966ba0a975e8f2858d88a8a5c2d216fbb742b-1943"}
{"level":"Level(-2)","ts":"2022-04-28T04:36:51.984Z","logger":"controller.imageupdateautomation","msg":"found schema extension","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageUpdateAutomation","name":"image-c","namespace":"flux-system","path":".metadata.labels.tags.datadoghq.com/version"}
{"level":"Level(-2)","ts":"2022-04-28T04:36:51.984Z","logger":"controller.imageupdateautomation","msg":"applying setter","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageUpdateAutomation","name":"image-c","namespace":"flux-system","setter":"flux-system:image-b:tag","old":"develop-22a966ba0a975e8f2858d88a8a5c2d216fbb742b-1943","new":"develop-22a966ba0a975e8f2858d88a8a5c2d216fbb742b-1943"}
{"level":"Level(-2)","ts":"2022-04-28T04:36:51.984Z","logger":"controller.imageupdateautomation","msg":"found schema extension","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageUpdateAutomation","name":"image-c","namespace":"flux-system","path":".spec.template.metadata.labels.tags.datadoghq.com/version"}
{"level":"Level(-2)","ts":"2022-04-28T04:36:51.984Z","logger":"controller.imageupdateautomation","msg":"applying setter","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageUpdateAutomation","name":"image-c","namespace":"flux-system","setter":"flux-system:image-b:tag","old":"develop-22a966ba0a975e8f2858d88a8a5c2d216fbb742b-1943","new":"develop-22a966ba0a975e8f2858d88a8a5c2d216fbb742b-1943"}
{"level":"Level(-2)","ts":"2022-04-28T04:36:51.984Z","logger":"controller.imageupdateautomation","msg":"found schema extension","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageUpdateAutomation","name":"image-c","namespace":"flux-system","path":".spec.template.spec.containers.image"}
{"level":"Level(-2)","ts":"2022-04-28T04:36:51.984Z","logger":"controller.imageupdateautomation","msg":"applying setter","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageUpdateAutomation","name":"image-c","namespace":"flux-system","setter":"flux-system:image-b","old":"xxxx.dkr.ecr.us-east-2.amazonaws.com/image-b-dev:develop-22a966ba0a975e8f2858d88a8a5c2d216fbb742b-1943","new":"xxxx.dkr.ecr.us-east-2.amazonaws.com/image-b-sandbox:develop-22a966ba0a975e8f2858d88a8a5c2d216fbb742b-1943"}
{"level":"Level(-2)","ts":"2022-04-28T04:36:51.985Z","logger":"controller.imageupdateautomation","msg":"found schema extension","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageUpdateAutomation","name":"image-c","namespace":"flux-system","path":".metadata.labels.tags.datadoghq.com/version"}
{"level":"Level(-2)","ts":"2022-04-28T04:36:51.985Z","logger":"controller.imageupdateautomation","msg":"applying setter","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageUpdateAutomation","name":"image-c","namespace":"flux-system","setter":"flux-system:image-a-preview:tag","old":"develop-22a966ba0a975e8f2858d88a8a5c2d216fbb742b-1943","new":"develop-22a966ba0a975e8f2858d88a8a5c2d216fbb742b-1943"}
{"level":"Level(-2)","ts":"2022-04-28T04:36:51.985Z","logger":"controller.imageupdateautomation","msg":"found schema extension","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageUpdateAutomation","name":"image-c","namespace":"flux-system","path":".spec.template.metadata.labels.tags.datadoghq.com/version"}
{"level":"Level(-2)","ts":"2022-04-28T04:36:51.985Z","logger":"controller.imageupdateautomation","msg":"applying setter","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageUpdateAutomation","name":"image-c","namespace":"flux-system","setter":"flux-system:image-a-preview:tag","old":"develop-22a966ba0a975e8f2858d88a8a5c2d216fbb742b-1943","new":"develop-22a966ba0a975e8f2858d88a8a5c2d216fbb742b-1943"}
{"level":"Level(-2)","ts":"2022-04-28T04:36:51.985Z","logger":"controller.imageupdateautomation","msg":"found schema extension","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageUpdateAutomation","name":"image-c","namespace":"flux-system","path":".spec.template.spec.containers.image"}
{"level":"Level(-2)","ts":"2022-04-28T04:36:51.985Z","logger":"controller.imageupdateautomation","msg":"applying setter","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageUpdateAutomation","name":"image-c","namespace":"flux-system","setter":"flux-system:image-a-preview","old":"xxxx.dkr.ecr.us-east-2.amazonaws.com/image-a-dev-preview:develop-22a966ba0a975e8f2858d88a8a5c2d216fbb742b-1943","new":"xxxx.dkr.ecr.us-east-2.amazonaws.com/image-a-sandbox-preview:develop-22a966ba0a975e8f2858d88a8a5c2d216fbb742b-1943"}
{"level":"Level(-2)","ts":"2022-04-28T04:36:51.985Z","logger":"controller.imageupdateautomation","msg":"found schema extension","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageUpdateAutomation","name":"image-c","namespace":"flux-system","path":".metadata.labels.tags.datadoghq.com/version"}
{"level":"Level(-2)","ts":"2022-04-28T04:36:51.985Z","logger":"controller.imageupdateautomation","msg":"applying setter","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageUpdateAutomation","name":"image-c","namespace":"flux-system","setter":"flux-system:image-a:tag","old":"develop-22a966ba0a975e8f2858d88a8a5c2d216fbb742b-1943","new":"develop-22a966ba0a975e8f2858d88a8a5c2d216fbb742b-1943"}
{"level":"Level(-2)","ts":"2022-04-28T04:36:51.986Z","logger":"controller.imageupdateautomation","msg":"found schema extension","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageUpdateAutomation","name":"image-c","namespace":"flux-system","path":".spec.template.metadata.labels.tags.datadoghq.com/version"}
{"level":"Level(-2)","ts":"2022-04-28T04:36:51.986Z","logger":"controller.imageupdateautomation","msg":"applying setter","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageUpdateAutomation","name":"image-c","namespace":"flux-system","setter":"flux-system:image-a:tag","old":"develop-22a966ba0a975e8f2858d88a8a5c2d216fbb742b-1943","new":"develop-22a966ba0a975e8f2858d88a8a5c2d216fbb742b-1943"}
{"level":"Level(-2)","ts":"2022-04-28T04:36:51.986Z","logger":"controller.imageupdateautomation","msg":"found schema extension","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageUpdateAutomation","name":"image-c","namespace":"flux-system","path":".spec.template.spec.containers.image"}
{"level":"Level(-2)","ts":"2022-04-28T04:36:51.986Z","logger":"controller.imageupdateautomation","msg":"applying setter","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageUpdateAutomation","name":"image-c","namespace":"flux-system","setter":"flux-system:image-a","old":"xxxx.dkr.ecr.us-east-2.amazonaws.com/ƒg-dev:develop-22a966ba0a975e8f2858d88a8a5c2d216fbb742b-1943","new":"xxxx.dkr.ecr.us-east-2.amazonaws.com/image-a-sandbox:develop-22a966ba0a975e8f2858d88a8a5c2d216fbb742b-1943"}
{"level":"Level(-2)","ts":"2022-04-28T04:36:51.990Z","logger":"managed-transport","msg":"[ssh]: run on remote","cmd":"git-receive-pack '/xxx/flux'"}
{"level":"Level(-2)","ts":"2022-04-28T04:36:51.990Z","logger":"managed-transport","msg":"[ssh]: run on remote","cmd":"git-upload-pack '/xxx/flux'"}
{"level":"Level(-2)","ts":"2022-04-28T04:36:51.991Z","logger":"managed-transport","msg":"[ssh]: sshSmartSubtransport.Close()","server":"github.com:22"}
{"level":"Level(-2)","ts":"2022-04-28T04:36:51.991Z","logger":"managed-transport","msg":"[ssh]: session.Close()","server":"github.com:22"}
{"level":"debug","ts":"2022-04-28T04:36:51.995Z","logger":"controller.imageupdateautomation","msg":"ran updates to working dir","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageUpdateAutomation","name":"image-c","namespace":"flux-system","working":"/tmp/flux-system-flux-system2400661092"}
panic: invalid pointer handle: 0x7ff5fdce8500
goroutine 361 [running]:
github.com/libgit2/git2go/v33.(*HandleList).Get(0xc000480d00, 0x7ff5fdce8500)
github.com/libgit2/git2go/v33@v33.0.9/handles.go:64 +0x12e
github.com/libgit2/git2go/v33.credentialsCallback(0xc00010f000, 0xc0003e52b0, 0xc0008dc640, 0xc000d32500, 0x260, 0xc00007e8d0)
github.com/libgit2/git2go/v33@v33.0.9/remote.go:373 +0x45
github.com/libgit2/git2go/v33._Cfunc_git_transport_smart_credentials(0xc0003e52b0, 0x7ff5fdd6f490, 0x0, 0x40)
_cgo_gotypes.go:8811 +0x4c
github.com/libgit2/git2go/v33.(*Transport).SmartCredentials.func2(0xc0003e52b0, 0xc0003e52b0, 0x1af7300, 0x40)
github.com/libgit2/git2go/v33@v33.0.9/transport.go:112 +0x90
github.com/libgit2/git2go/v33.(*Transport).SmartCredentials(0x1dc50fd, {0x0, 0xc000278990}, 0x1)
github.com/libgit2/git2go/v33@v33.0.9/transport.go:112 +0xeb
github.com/fluxcd/source-controller/pkg/git/libgit2/managed.(*sshSmartSubtransport).Action(0xc000382000, {0xc000efd290, 0x29}, 0x2)
github.com/fluxcd/source-controller@v0.24.0/pkg/git/libgit2/managed/ssh.go:174 +0x32e
github.com/libgit2/git2go/v33.smartSubtransportActionCallback(0x4e31e7, 0x7ff5fd87e978, 0x1000c00199a600, 0x1, 0x2)
github.com/libgit2/git2go/v33@v33.0.9/transport.go:374 +0xb0
github.com/libgit2/git2go/v33._Cfunc_git_remote_free(0x7ff5fdf7ffe0)
_cgo_gotypes.go:7146 +0x45
github.com/libgit2/git2go/v33.(*Remote).free.func1(0x1cf9060)
github.com/libgit2/git2go/v33@v33.0.9/remote.go:603 +0x46
github.com/libgit2/git2go/v33.(*Remote).free(0xc0002be0c0)
github.com/libgit2/git2go/v33@v33.0.9/remote.go:603 +0x36
github.com/libgit2/git2go/v33.(*Remote).Free(0xc0002be0c0)
github.com/libgit2/git2go/v33@v33.0.9/remote.go:614 +0x48
github.com/fluxcd/image-automation-controller/controllers.push({0x2027560, 0xc0002be000}, {0xc0008fa300, 0xc00067be60}, {0xc000697ebc, 0x4}, {0xc00075c750, {0xc0007622a0, 0x0}})
github.com/fluxcd/image-automation-controller/controllers/imageupdateautomation_controller.go:824 +0x3c6
github.com/fluxcd/image-automation-controller/controllers.(*ImageUpdateAutomationReconciler).Reconcile(0xc00082eb80, {0x2027598, 0xc00067be30}, {{{0xc000697ea0, 0x1ce5a20}, {0xc0008c6618, 0x30}}})
github.com/fluxcd/image-automation-controller/controllers/imageupdateautomation_controller.go:383 +0x2d9a
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0xc000544000, {0x2027598, 0xc00067bda0}, {{{0xc000697ea0, 0x1ce5a20}, {0xc0008c6618, 0x414fd4}}})
sigs.k8s.io/controller-runtime@v0.11.2/pkg/internal/controller/controller.go:114 +0x26f
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000544000, {0x20274f0, 0xc00082e840}, {0x1c00ea0, 0xc000620540})
sigs.k8s.io/controller-runtime@v0.11.2/pkg/internal/controller/controller.go:311 +0x33e
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000544000, {0x20274f0, 0xc00082e840})
sigs.k8s.io/controller-runtime@v0.11.2/pkg/internal/controller/controller.go:266 +0x205
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
sigs.k8s.io/controller-runtime@v0.11.2/pkg/internal/controller/controller.go:227 +0x85
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2
sigs.k8s.io/controller-runtime@v0.11.2/pkg/internal/controller/controller.go:223 +0x357
@pjbgf
Copy link
Member

pjbgf commented May 3, 2022

@johnp-stella thank you for creating the issue and sharing the error logs.

Can you confirm this is consistent? Does it generally take the same amount of time for it to crash after restart? Does it happen every single time? Does the changes ever get pushed back into the repository?

Are you using what SSH key type (e.g. rsa, edcsa or ed25519)?

@johnp-stella
Copy link
Author

1.) It is consistently crashing

 restartCount: 74
    started: false
    state:
      waiting:
        message: back-off 5m0s restarting failed container=manager pod=image-automation-controller-756c98d579-qzk88_flux-system(876ab84b-8723-4531-93e2-2d912a68781f)
        reason: CrashLoopBackOff

2.) The timing is similar but not exact - and does not crash the same point each time

run 1
2022-05-04T00:13:13.166865664Z {"level":"info","ts":"2022-05-04T00:13:13.166Z","logger":"controller-runtime.metrics","msg":"Metrics server is starting to listen","addr":":8080"}
<big snip>
2022-05-04T00:14:02.673955064Z {"level":"Level(-2)","ts":"2022-05-04T00:14:02.673Z","logger":"managed-transport","msg":"[ssh]: run on remote","cmd":"git-receive-pack '/xxxxx/flux'"}
<crash>
run 2
2022-05-04T00:19:39.740145606Z {"level":"info","ts":"2022-05-04T00:19:39.739Z","logger":"controller-runtime.metrics","msg":"Metrics server is starting to listen","addr":":8080"}
<big snip>
2022-05-04T00:20:56.821584283Z {"level":"Level(-2)","ts":"2022-05-04T00:20:56.820Z","logger":"controller.imageupdateautomation","msg":"applying setter","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageUpdateAutomation","name":"candidate-web","namespace":"flux-system","setter":"flux-system:image-a","old":"yyyy.dkr.ecr.us-east-2.amazonaws.com/image-a:latest","new":"xxx.dkr.ecr.us-east-2.amazonaws.com/image-a:latest"}
2022-05-04T00:20:56.841039670Z panic: invalid pointer handle: 0x7f542b7ea110

run 3
2022-05-04T00:27:05.386479439Z {"level":"info","ts":"2022-05-04T00:27:05.384Z","logger":"controller-runtime.metrics","msg":"Metrics server is starting to listen","addr":":8080"}
<big snip>
2022-05-04T00:27:52.164733502Z {"level":"Level(-2)","ts":"2022-05-04T00:27:52.164Z","logger":"managed-transport","msg":"[ssh]: session.Close()","server":"github.com:22"}
2022-05-04T00:27:52.180042841Z panic: invalid pointer handle: 0x7f29cad68ce0

3.) It is pushing changes to repo - as the point it crashes is not consistent so is sometimes able to progress to a push.
4.) SSH key for github

algorithm   = "ECDSA"
ecdsa_curve = "P384"

@johnp-stella
Copy link
Author

I tied changing the GitHub ssh key type to ED25519 - Still saw the controller crashing

@pjbgf
Copy link
Member

pjbgf commented May 6, 2022

I was not able to reproduce this issue yet, but I have made a few changes recently and would be keen to see whether you still experience the same problem in your environment.

Here are two images created with latest source controller (which includes panic recovery) plus the removal of ssh connection caching. And the image-automation-controller using said source-controller version:

source-controller: quay.io/paulinhu/source-controller:v0.24.4-cacheless@sha256:61930cad1da900f209b396f20c2f7740ff32b5cf1bb4ab7892200790c00a5f4b

image-automation-controller: quay.io/paulinhu/image-automation-controller:v0.22.1-cacheless@sha256:a33a6fd7b40524dbc1a49333bcdb80f26803edc7f6a841f5005ba636beb68e4b

More details on:
fluxcd/source-controller#713 (comment)

@johnp-stella
Copy link
Author

Thanks @pjbgf The experimental docker images above have fixed the crashing issue. It has been stable and seems to be working as expected.

@kingdonb
Copy link
Member

Thanks for the information! I'm going to assign this to @pjbgf so it can be closed. 👍

@kingdonb kingdonb added the bug Something isn't working label May 12, 2022
@pjbgf pjbgf moved this to In Progress in Maintainers' Focus May 12, 2022
@pjbgf pjbgf added this to the GA milestone May 12, 2022
@pjbgf pjbgf closed this as completed May 12, 2022
Repository owner moved this from In Progress to Done in Maintainers' Focus May 12, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@pjbgf pjbgf

Labels
bug Something isn't working
Projects
Maintainers' Focus
Status: Done
Milestone
GA
Development

No branches or pull requests
3 participants
@kingdonb @pjbgf @johnp-stella