Possible hung git invocation causing presubmit failures

[Prabhuk]

Running into the following failure when uploading new PRs or updating existing PRs at times.

Running 'git gc'...
  | error: cannot lock ref 'refs/remotes/origin/users/matthias-springer/cse_filter': Unable to create '/var/lib/buildkite-agent/builds/service-855b8ffd5d-k7cg6-1/llvm-project/github-pull-requests/.git/refs/remotes/origin/users/matthias-springer/cse_filter.lock': File exists.
  |  

<br class="Apple-interchange-newline">

Links to two separate failing instances:

https://buildkite.com/llvm-project/github-pull-requests/builds/173742#01968337-b887-4d3e-99d9-6f5c9c3af89d

https://buildkite.com/llvm-project/github-pull-requests/builds/173749#0196834c-83a2-4bbe-b9c2-038795ea088f

[llvmbot]

@llvm/issue-subscribers-infrastructure

Author: Prabhu Rajasekaran (Prabhuk)

Details

Running into the following failure when uploading new PRs or updating existing PRs at times.

Running 'git gc'...
  | error: cannot lock ref 'refs/remotes/origin/users/matthias-springer/cse_filter': Unable to create '/var/lib/buildkite-agent/builds/service-855b8ffd5d-k7cg6-1/llvm-project/github-pull-requests/.git/refs/remotes/origin/users/matthias-springer/cse_filter.lock': File exists.
  |  

<br class="Apple-interchange-newline">

Links to two separate failing instances:

https://buildkite.com/llvm-project/github-pull-requests/builds/173742#01968337-b887-4d3e-99d9-6f5c9c3af89d

https://buildkite.com/llvm-project/github-pull-requests/builds/173749#0196834c-83a2-4bbe-b9c2-038795ea088f

[ilovepi]

I'm also seeing this. https://buildkite.com/llvm-project/github-pull-requests/builds/173773#01968387-d14d-4d7a-8da4-2315046de81f

BUILDKITE_REPO: https://github.com/llvm/llvm-project.git
current remote URL: https://github.com/llvm/llvm-project.git
GC counter 89/20
Running 'git gc'...
error: cannot lock ref 'refs/remotes/origin/users/matthias-springer/cse_filter': Unable to create '/var/lib/buildkite-agent/builds/service-855b8ffd5d-k7cg6-1/llvm-project/github-pull-requests/.git/refs/remotes/origin/users/matthias-springer/cse_filter.lock': File exists.

Do we have any idea what would be causing it? The user branch bit seems suspicious, but I'm not sure how/why that would happen.

[boomanaiden154]

It seems like this was impacting pretty much every premerge job. From what I can tell the jobs were running inside the pod service-855b8ffd5d-k7cg6service-855b8ffd5d-k7cg6. I've killed that pod and a new one is being brought up. I will monitor and see if that fixes the issue.

[boomanaiden154]

Seems like things are recovering. The following jobs are running and have gotten past the pipeline generation phase:

1. https://buildkite.com/llvm-project/github-pull-requests/builds/173781
2. https://buildkite.com/llvm-project/github-pull-requests/builds/173782
3. https://buildkite.com/llvm-project/github-pull-requests/builds/173783

[boomanaiden154]

Looks like things are returning to normal.. This is a failure mode that theoretically is not possible with the new system which is nice.