Entire git history being squashed and rewritten by v0.21.0

[jiphex]

I've recently updated the image-automation-controller in this setup to v0.21.0 to try and fix the GoRoutine leak issue noted in the release notes.

Since I've done this, it seems whenever the image-automation-controller tries to add commits to the repository, instead of tacking its new commits to the end, it's rewriting the entire git history of the repo into a single fat commit with its own commit message.

The Flux version is as follows:

❯ flux version
flux: v0.27.4
helm-controller: v0.17.2
image-automation-controller: v0.21.0 # (with EXPERIMENTAL_GIT_TRANSPORT=true)
image-reflector-controller: v0.16.0
kustomize-controller: v0.21.1
notification-controller: v0.22.3
source-controller: v0.22.0

The GitRepository and ImageUpdateAutomation are at the end of this issue, the GitRepository is configured to pull from the main branch of the Git(lab) repository, but to push to another branch so that a Merge Request can be created in the CI rather than applying its changes directly to main. To make sure that the new branch doesn't need to be rebased, it doesn't "normally" exist, instead it should be created by the controller (this is related to the behaviour described by @squaremo in this issue). The MRs created by Gitlab CI are configured to delete the branch on merge to keep things tidy.

Before the image-automation-controller update, this seemed to work fine, MRs would be opened automatically and the commits would be tacked on the end of the history of the main branch. Now however, it seems like whenever the controller does a push, it wipes out the whole git commit history and replaces it with a single commit containing the entire set of manifests, which ends up looking like this in Gitlab (shown as 333 commits behind main, 1 ahead, because the history now contains only a single commit):

The ImageUpdateAutomation and GitRepository resources are attached.

One thing I've noticed is that this seems to be affected by the presence (or not) of the branch at the time that the controller starts up - if the push branch exists when the controller starts, then the controller takes no action (but then if any commits are pushed to main between automation commits, they need to be rebased due to the lack of force push). If the push branch doesn't exist, then the behaviour above seems to apply.

[hiddeco]

Looking at the recent set of changes, I think it would be best to put any 💸 for bets on this being introduced by #324.

The good news is that this only seems to affect scenarios in which the push branch differs, which means that it does not try to overwrite the history of the branch it is consuming from.

[jiphex]

I've now updated to the flux v0.28.0 components:

❯ flux version
flux: v0.28.0
helm-controller: v0.18.0
image-automation-controller: v0.21.0
image-reflector-controller: v0.17.0
kustomize-controller: v0.22.0
notification-controller: v0.23.0
source-controller: v0.22.0

After I pushed this commit, it seems like the image-automation-controller got stuck in a loop and was pushing a new commit to the repo target branch with every controller loop, which quickly started to overwhelm our CI runner after 20 or so commits were added 😆 - I've suspended the automation for now

[stefanprodan]

@jiphex does this happens only for EXPERIMENTAL_GIT_TRANSPORT=true?

[hiddeco]

This happens at commit level, and should be unrelated to transport (which only transfers the commit data to remote).

[jiphex]

@jiphex does this happens only for EXPERIMENTAL_GIT_TRANSPORT=true?

That doesn't seem to make any difference

[jiphex]

After I pushed this commit, it seems like the image-automation-controller got stuck in a loop and was pushing a new commit to the repo target branch with every controller loop, which quickly started to overwhelm our CI runner after 20 or so commits were added 😆 - I've suspended the automation for now

Someone on the Slack mentioned the looping commits issue, which is potentially related to #166 or #135

[pjbgf]

The official version ghcr.io/fluxcd/image-automation-controller:v0.21.1 should contain a fix for this issue. Can you please test it and let us know whether it works it for you?

[jiphex]

Thanks, this seems resolved to me - the branch was created with a single new commit on top of the existing history:

[pjbgf]

Excellent, thank you for checking @jiphex. I will be closing this issue as the fix is already released under ghcr.io/fluxcd/image-automation-controller:v0.21.1.

If other users are still experiencing it with the latest release we can always open the issue back again.