cloning repo is broken

[ilia-shipitsin]

brew gist-logs <formula> link OR brew config AND brew doctor output

# git clone https://github.com/Homebrew/homebrew-core
Cloning into 'homebrew-core'...
remote: Enumerating objects: 1532699, done.
remote: Counting objects: 100% (34185/34185), done.
remote: fatal: object 0acc93f8ce85f8561492a6b56256ed7f0035af76 cannot be read
remote: aborting due to possible repository corruption on the remote side.
fatal: protocol error: bad pack header
#

it would be nice to fix, because we use https://raw.githubusercontent.com/Homebrew/install/master/install.sh and it clones repo, thus fails :(

### Verification

- [X] My "`brew doctor` output" says `Your system is ready to brew.` and am still able to reproduce my issue.
- [X] I ran `brew update` and am still able to reproduce my issue.
- [X] I have resolved all warnings from `brew doctor` and that did not fix my problem.
- [X] I searched for recent similar issues at https://github.com/Homebrew/homebrew-core/issues?q=is%3Aissue and found no duplicates.

### What were you trying to do (and why)?

I'm trying to install brew using `https://raw.githubusercontent.com/Homebrew/install/master/install.sh`

### What happened (include all command output)?

git clone https://github.com/Homebrew/homebrew-core

Cloning into 'homebrew-core'...
remote: Enumerating objects: 1532699, done.
remote: Counting objects: 100% (34185/34185), done.
remote: fatal: object 0acc93f8ce85f8561492a6b56256ed7f0035af76 cannot be read
remote: aborting due to possible repository corruption on the remote side.
fatal: protocol error: bad pack header

### What did you expect to happen?

not to fail

### Step-by-step reproduction instructions (by running `brew` commands)

```shell
HOMEBREW_INSTALL_URL="https://raw.githubusercontent.com/Homebrew/install/master/install.sh"
/bin/bash -c "$(curl -fsSL ${HOMEBREW_INSTALL_URL})"

[Bo98]

This is a GitHub issue we have no control over. I am told it they have fixed the issue so give it another try.

If not, try a few times as it the error doesn't seem to happen every time.

[ilia-shipitsin]

it's not a github issue.

anyway, it is possible in theory to clone using depth 1

~# git clone --depth 1 https://github.com/Homebrew/homebrew-core
Cloning into 'homebrew-core'...
remote: Enumerating objects: 7052, done.
remote: Counting objects: 100% (7052/7052), done.
remote: Compressing objects: 100% (6719/6719), done.
remote: Total 7052 (delta 106), reused 1569 (delta 64), pack-reused 0
Receiving objects: 100% (7052/7052), 7.78 MiB | 7.38 MiB/s, done.
Resolving deltas: 100% (106/106), done.
~#

but I cannot pass "depth 1" to brew install script

[ilia-shipitsin]

please consider adjusting https://raw.githubusercontent.com/Homebrew/install/master/install.sh to clone with depth 1, it will prevent from such errors

[Bo98]

It is a GitHub issue. We are talking to them directly about it and some replicas on their end are missing parts of the repo, causing the errors you have reported.

[Bo98]

please consider adjusting https://raw.githubusercontent.com/Homebrew/install/master/install.sh to clone with depth 1, it will prevent from such errors

We don't do this because it causes severe performance issues that lead to GitHub asking us to stop using that flag. https://github.com/Homebrew/brew/blob/master/Library/Homebrew/cmd/update.sh#L474

[ilia-shipitsin]

I do not suggest to change default behaviour.

I'm ok with default clone, people may expect it.

however, if I need just the latest commit, I'm fine to say "depth 1" and I do not care of broken git indices.

[ilia-shipitsin]

maybe it makes sense to keep unshallow by default and a possibility to switch to shallow in case of broken index (obviously, depth 1 works with broken index)

[seannie-sharko]

Same issue. This is what ended up working for me:

My system:
mba m1
Ventura: 13.4.1 (22F82)

Removed and rebuilt core with @ilia-shipitsin suggestion to fetch the shallow image

cd /usr/local/Homebrew/Library/Taps/homebrew/
rm -rf /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core
git clone --depth 1 https://github.com/Homebrew/homebrew-core.git
git -C /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core fetch --unshallow

This will pull down ~400MB and resolve over 1 million deltas, be patient

brew update should now be up-to-date and brew upgrade should work

[rfay]

This is breaking routinely using the standard GH Actions homebrew action, https://github.com/ddev/ddev-platformsh/blob/ea9eb17449728c6dcb57dd95b75939a37da49da9/.github/workflows/tests.yml#L47-L49

It's breaking every repo (there are MANY) that uses it. I do note that it's using HOMEBREW_NO_INSTALL_FROM_API: 1 in order to use brew install --HEAD

Examples: https://github.com/ddev/ddev-platformsh/actions/runs/5415041350

[SMillerDev]

brew install --HEAD is unsupported so that's not a concern.

[SMillerDev]

maybe it makes sense to keep unshallow by default

GitHub has asked us explicitly not to do this.

[ilia-shipitsin]

thank you very much for closing this issue. you are very helpful

however, I still cannot clone repo

~# git clone https://github.com/Homebrew/homebrew-core
Cloning into 'homebrew-core'...
remote: Enumerating objects: 11082, done.
remote: Counting objects: 100% (7756/7756), done.
remote: fatal: object 0d7d72f3d46957a85ebbf915a1d2c78f001e486c cannot be read
remote: aborting due to possible repository corruption on the remote side.
fatal: protocol error: bad pack header
~#

[Bo98]

Latest update from GitHub is that they're rolling back code changes that caused this issue. It's being worked on.

We're well aware of the issue as it stalls all of our own CI, so we're not ignoring this by any means.

I do note that it's using HOMEBREW_NO_INSTALL_FROM_API: 1 in order to use brew install --HEAD

FWIW, this is one of the only reasons left to use HOMEBREW_NO_INSTALL_FROM_API. This scenario of brew install --HEAD will be fixed soon (either next release or the one after), at which point it should be safe to remove that env.

[Bo98]

If you want an indication of recovery, check https://github.com/Homebrew/formulae.brew.sh/actions/workflows/scheduled.yml which is a job that runs every 15 minutes and fetchs Homebrew/core twice. If either of those fail then the whole job fails.

[SMillerDev]

however, I still cannot clone repo

And that's still not something for us to fix. Do you also complain to your car dealer about road closures?

[ilia-shipitsin]

sorry, I did not realize that https://raw.githubusercontent.com/Homebrew/install/master/install.sh is not supposed to work.
is there a proper way ?

[Bo98]

I realise you're upset at us for this issue occurring but to follow up for the benefit for anyone else following along:

GitHub has identified and fixed what they believe to be the root cause, and has finished the repair operation on homebrew-core. This issue should now be resolved, and our own monitoring has confirmed this so far. Big thanks to the git systems team there who worked most of the day on resolving this issue.

I'll continue to monitor and keep an eye out to see if this regresses again, as the nature of the issue means it doesn't always happen on every fetch (I happened to be lucky enough to never experience it on my local machine despite trying) and may not resurface for a number of hours.

homebrew-core is a particularly complex repo which has been subject to even having its CPU usage featured on the GitHub blog. Not only is the repository massive, it has millions of people using it every day. This is why decisions like shallow clones are not as trivial many may think (e.g. if shallow clones were to make its way into a CI image, it would then spread a large server load when a bunch of repos run brew update). Most of the decisions taken to now has been done in conjunction with advice from GitHub to ensure the future health of homebrew-core, including a recent change to the way we merge pull requests.

Thankfully however, since Homebrew 4.0.0, homebrew-core clones are not required (except certain usages of --HEAD - but that'll be fixed in a few days) so we've not seen widespread impact of this issue beyond GitHub Actions. The Homebrew installer does not clone homebrew-core unless HOMEBREW_NO_INSTALL_FROM_API is set. But we obviously do ourselves need a homebrew-core clone for homebrew-core CI, so were working closely in getting this resolved with high priority.