Git repo state seems to be corrupt?

[musabshak]

[backend] | 2025-09-05T20:44:17.357Z error: [repo-manager] Failed to sync repository gerrit.hostname.com/MyRepo (id: 8) after 3 attempts. Error: Error: Failed to fetch repository: /data/.sourcebot/repos/8. Reason: fatal: couldn't find remote ref HEAD
[backend] |

I updated Sourcebot from v4.6.3 -> v4.6.6, running in k8s. terminationGracePeriod for my deployment was set to 15s (meaning k8s would send SIGTERM to Sourcebot process, wait 15 seconds, hope Sourcebot gracefully shut down, then send SIGKILL after 15 seconds regardless). Perhaps this was too low.

In either case, after Sourcebot came up with v4.6.6 image, certain repos are now failing to sync/index, and retrying doesn't help. All fail with the same error signature (above).

Not sure how to debug this / bandaid fix these failing repos. Redeploying SB from scratch will certainly fix things but that's sort of undesirable when there are 5000+ repos indexed already.

[brendan-kellam]

Hmm ok a regression was introduced in 4.6.5 but I (thought) was fixed in 4.6.6 but clearly not.

For context, the change in 4.6.6 removed the remote.origin.url from the config (see #488). Presumably these repos are going into a bad state since the origin is not set.

Will look into this in a bit when I'm back on my laptop. Thx for reporting.

[brendan-kellam]

How many repos are corrupted?

[brendan-kellam]

Also clarify: did you upgrade to v4.6.5 at any point?

[musabshak]

I went straight from v4.6.3 to v4.6.6, after which I ran into this issue.

I then went down to v4.6.4 (because I wanted the Gerrit branch indexing feature), and the issue resolved itself (ie I guess the repos weren't "corrupt", there are just some bad code changes in v4.6.5 / v4.6.6)

That is, I never deployed v4.6.5.

[brendan-kellam]

I went straight from v4.6.3 to v4.6.6, after which I ran into this issue.

Ok gotcha, thanks for clarifying.

Failed to sync repository gerrit.hostname.com/MyRepo (id: 8) after 3 attempts. Error: Error: Failed to fetch repository: /data/.sourcebot/repos/8. Reason: fatal: couldn't find remote ref HEAD

If you don't mind, could you try to spin up a sourcebot deployment locally on v4.6.6 with this single repository listed in the config? I'm wondering if this issue will consistently reproduce with the same repos.

[musabshak]

A fresh deploy from scratch at v4.6.6 works fine. I guess the problematic aspect then is that updating to v4.6.6 breaks things (at least it did for me).

I brought up a new instance at v4.6.4 from scratch and everything works fine now, I'm just hesitant to update to v4.6.6 now in prod because it may break again.

[musabshak]

A fresh deploy from scratch at v4.6.6 works fine.

UPDATE: the fresh deploy from scratch at v4.6.6 fails at the first sync for the same repo. That is, the initial clone + index seemed to work fine, but after an hour, when it resync+reindexed, that failed. This suggests v4.6.6 is actively broken.

[musabshak]

Same error signature as in top-level message. And suspiciously, the failing repo(s) have the characteristic that they're empty repos.

[brendan-kellam]

And suspiciously, the failing repo(s) have the characteristic that they're empty repos.

Hmm this is useful information thanks. Let me try on my side.

[brendan-kellam]

@musabshak when you get a second, could you try out ghcr.io/sourcebot-dev/sourcebot:main and see if you still run into this issue?