Creation of new Workspaces stoped working

[konne]

Bug description

We can still open existing workspaces from our Dashboard but if we try to open a new one everybody in the company get this screen

Steps to reproduce

go to github and click the gitpod button

Workspace affected

No response

Expected behavior

No response

Example repository

No response

Anything else?

No response

[pawlean]

Do you happen to have a repo I can look at? Did you change anything in your .gitpod.yml file? I sometimes see this issue when I've formatted my .gitpod.yml incorrectly.

[konne]

no this are private repos. This is now already working again. I have here another workspace id from another user that had the
same issue.
We had no changes in the gitpod.yml and the underlying dockerfile and it was working yesterday and had issues today morning and now its working again. So I expect you had a temporary issue.

[pawlean]

Thanks for the update @konne, let me know if anything else shows up. Can you close this issue?

[konne]

Yes, you can do something: analyze why this happens!

We just rolled starting Monday gitpod to another 35 developers with Unleashed plan out, and this incident created a really bad sign for the devs to fully switch to gitpod.

[pawlean]

Sorry about that - I've informed our team internally. 🙏🏼

I'll close this issue for now, but feel free to ping us again anytime.

[csweichel]

I'm sorry you're seeing this. I've dug into the issue and found that it's the prebuild initializer that's failing while trying to do a git fetch (internal logs)

specifically:

git fetch  failed (exit status 1): From https://github.com/v****b/*****
error: cannot lock ref 'refs/remotes/origin/D**********b/H********_****/documentation': 'refs/remotes/origin/D**********b/H********_****' exists; cannot create 'refs/remotes/origin/D**********b/H********_****/documentation'

We'll work on a fix asap.

[csweichel]

We have merged a fix and are rolling it out as we speak. You should be able to start workspaces again in the coming hours.

One contributing factor was that there were branches with overlapping names, i.e. at some point a branch b existed which was subsequently deleted and replaced by a branch b/something. This overlap in names combined with the Git working copy of prebuilds led to this issue.

[konne]

@csweichel can you please also add that we get a better log in if the rebuild initializer fails.
For example on this issue we had even the chance that we could:

1. just fix in our repo
2. make clear that this is not a gitpod issue
   Again thanks for your search. We also had the same issue on our Jenkins build and someone fixed it just about two hours later and so I was then fixed. But there we had log files.

[csweichel]

@csweichel can you please also add that we get a better log in if the rebuild initializer fails.

That makes perfect sense - something we had in the back of our minds but should have done a long time ago.
I've filed an issue and made sure it has the appropriate priority: #7605