v0.14.0 not starting on k8s v1.20.2 and ARM64

[larsha]

Hi!

I upgraded to Flux v0.15.0 which includes the v0.14.0 of the source-controller. I can't get the pod to start, crashes immediately, no info in logs (tried both error and debug for log-level). I manually changed the image from v0.14.0 to v0.13.2 and it starts without any issues.

Can this be related to ARM64? (I read some info in the release notes regarding some ARMv7 issues?)

It exits with exit code 139, no other info that I can find.

[hiddeco]

It is probably related to ARM64 and the ARMv7 issues we experienced, I even think there is a chance that the ARMv7 patch conflicts with ARM64.

Do you by any chance make use of the automation controllers? If so, do they boot OK (image-automation-controller in particular)?

[larsha]

Yes I use them, and image-automation-controller seems to boot up just fine.

[hiddeco]

That gives me sufficient information to look for a solution, or rather, trace why the image-automation-controller succeeds to build without the patch that was required for this controller.

This may take some time, but I'll start tomorrow.

[hiddeco]

In the meantime: dropping the binutils-gold from https://github.com/fluxcd/source-controller/blob/v0.14.0/Dockerfile and rebuilding the image for your architecture should work and temporary resolve the issue.

[larsha]

Thank you! Let me know if you need anything from me or if I can assist you in some way.

[ViBiOh]

Same problem here on Kubernetes v1.21.1+k3s1, Manjaro ARM64 on Raspberry pi4.

I have no starting log, the kubectl describe give me this event. I reverted back to 0.13.2. If you need more details, I can give you more ;)

  Normal   Scheduled  60s                default-scheduler  Successfully assigned flux-system/source-controller-54bbf867c6-8jgzs to charlie
  Normal   Pulling    59s                kubelet            Pulling image "ghcr.io/fluxcd/source-controller:v0.14.0"
  Normal   Pulled     39s                kubelet            Successfully pulled image "ghcr.io/fluxcd/source-controller:v0.14.0" in 20.324590933s
  Normal   Pulled     17s (x2 over 33s)  kubelet            Container image "ghcr.io/fluxcd/source-controller:v0.14.0" already present on machine
  Normal   Created    16s (x3 over 35s)  kubelet            Created container manager
  Normal   Started    16s (x3 over 35s)  kubelet            Started container manager
  Warning  Unhealthy  16s (x3 over 34s)  kubelet            Readiness probe failed: Get "http://10.42.1.222:9090/": dial tcp 10.42.1.222:9090: connect: connection refused
  Warning  BackOff    10s (x6 over 31s)  kubelet            Back-off restarting failed container

[hiddeco]

Any chance one of you also has access to an ARMv7 machine? If so, can you please confirm the current image does work on that machine?

[larsha]

Any chance one of you also has access to an ARMv7 machine? If so, can you please confirm the current image does work on that machine?

I'm afraid I don't have access to an ARMv7 machine.