Git 2.21.0 prints error message "Server does not allow request for unadvertised object"

[YngveNPettersen]

• [ x] I was not able to find an open or closed issue matching what I'm seeing

Setup

• Which version of Git for Windows are you using? Is it 32-bit or 64-bit?

git version 2.21.0.windows.1
cpu: x86_64
built from commit: 2481c4cbe949856f270a3ee80c802f5dd89381aa
sizeof-long: 4
sizeof-size_t: 8

• Which version of Windows are you running? Vista, 7, 8, 10? Is it 32-bit or 64-bit?

Win 7 and Win10, both 64-bit

• What options did you set as part of the installation? Or did you choose the
  defaults?

Editor Option: VIM
Custom Editor Path:
Path Option: BashOnly
Plink Path: C:\Program Files\PuTTY\plink.exe
SSH Option: Plink
CURL Option: OpenSSL
CRLF Option: CRLFAlways
Bash Terminal Option: MinTTY
Performance Tweaks FSCache: Enabled
Use Credential Manager: Disabled
Enable Symlinks: Disabled

• Any other interesting things about your environment that might be related
  to the issue you're seeing?

Lots of submodules

One submodule reference in parent module points to a reference in the submodule that has not been pushed, e.g because it was an intermediate branch while rebasing

Alternatively, reference a commit in an added sub module repository, which have not yet been cloned.

Details

• Which terminal/shell are you running Git from? e.g Bash/CMD/PowerShell/other

Git Bash

• What commands did you run to trigger this issue? If you can provide a
  Minimal, Complete, and Verifiable example
  this will help us understand the issue.

One of these

git fetch
git pull

• What did you expect to occur after running these commands?

Ordinary git fetch of current module and relevant submodules, optionally doing a pull of the current branch afterwards

• What actually happened instead?

Fetch prints this error message:

error: Server does not allow request for unadvertised object deadbeef....

And exits with an error

• If the problem was occurring with a specific repository, can you provide the
  URL to that repository to help us with testing?

Sorry, all URLs are internal

Further details:

I have three main objections about this error message:

1. It is printed (and exits with error) despite the fact the the missing commit is not necessary for the resulting checkout. I do not mind a failed checkout if the actual checkout references a missing commit. And I have git hooks server side that prevent this for our production branch.

2. The error message causes our autobuild buildsystem to fail (just the Windows workers currently, all others are using older git versions). Next build will work, since the missing commit is no longer an issue. If this error was a serious matter, then it should fail every fetch, not just the first (and no, I am not suggesting it should do so)

3. The error message provides absolutely no context. Which repository referenced the missing commit? Which commit in the repository did so? Which submodule reference? What is the URL of the repository that failed to provide the commit?

The most recent instance of this happened when I added a new module to a module set, one which had not yet been cloned (and in many cases was not going to be cloned in that location in the several weeks) by the checkout directory. This will result in builds failing on at least 9 different machines in the next few days.

IMO this error should be changed to a warning.

[YngveNPettersen]

Please note that this issue is problematic enough that I am now considering rolling our Git deployment back to 2.17 or 2.18, and keep it there until this issue have been fixed

Update: Rolled everything back to 2.17

[dscho]

This does not look like it has anything to do with your Git client version. If you install a portable Git v2.17.x, I would wager a bet that it now has the same problem!

This looks more like a submodule is pointing to a commit that used to be reachable from a ref on the server, but somebody force-pushed (or deleted) the ref, and that commit is no longer reachable.

In other words, this looks like a mistake in the project management to me, not like a bug in Git (either client or server).

It is just one of the many issues with submodules that you better know what you're doing there, force-pushing is almost never okay when you have submodules, or when the force-pushed branch is in a repository that is used as a submodule elsewhere.

[YngveNPettersen]

I doubt it, because I never observed this before 2.21

I do see that the the error message itself was added in 2.12 or 2.13, two years ago.

The error in all cases before yesterday have always been that the developer (others or myself) have rebased the branch (or amended the commit), in the submodule, added a new commit for that in the parent module, then in my cases, after a rebase the commit in the parent was updated without amending the old submodule pointer. Then the branches were pushed, and 2.21 instances started protesting about the missing commit in the submodule.

With 2.17 the only concerning thing I observe is this:

warning: Submodule in commit deadbee at path: '(NULL)' collides with a submodule named the same. Skipping it.

Which, while irritating, does not break the checkout.

Looking at the logs, one possible patch that seems to change the logic in this area is 22a1646, which is new in 2.20+

For reference, I just did two fetches, one with 2.17, and one with 2.21 on a different machine, same top module. 2.17 exited with 0, 2.21 exited with 1 (error).

I will also note that before I rolled git back, I had at least four autobuilds fail due to this error, none after the rollback (and at least two builds should have failed if 2.17 had the issue).

[dscho]

Sorry, all URLs are internal

Please note that this makes it rather impossible for me to do more on my side...

The error in all cases before yesterday have always been that the developer (others or myself) have rebased the branch (or amended the commit), in the submodule, added a new commit for that in the parent module, then in my cases, after a rebase the commit in the parent was updated without amending the old submodule pointer.

That makes sense.

Looking at the logs, one possible patch that seems to change the logic in this area is 22a1646, which is new in 2.20+

Possibly. If you ask me to debug this further, I would need an MCVE... Even if you don't ask me, but plan on investigating yourself a bit more, it would make sense to try to reduce the example (or recreate it with a fresh, toy example).

And yes, this commit sounds like it might become over-eager in trying to fetch something when it fails to understand that it has everything it needs locally already. (If that is what you are trying to say, that is...)

[YngveNPettersen]

The attached test case reproduces the problem in my remaining 2.21 installation. It does not reprocude in 2.17

git-unadvertise1.zip

[dscho]

Thanks for this recipe.

I simplified it down to

set -ex

rm -rf super clone

git init super

git init super/sub
git -C super/sub commit --allow-empty -m "Init submodule"

git -C super submodule add "$PWD/super/sub/.git" sub
git -C super commit -m "Added submodule"

git clone --recursive super clone

git -C super/sub commit --allow-empty -m "Modify submodule"

git -C super commit -m "Modified submodule" sub

git -C super/sub commit --allow-empty --amend -m "Amend" --no-edit

git -C super commit -m "Modified submodule again" sub

echo Error should occur now
git -C clone fetch

Then, I turned it into a script, ready for git bisect run (i.e. exit code 1 indicating the reported issue happens, exit code 0 indicating that the fetch succeeded, exit code 125 for everything else):

#!/bin/sh

set -x

cd "$(dirname "$0")"
make -j8 -C .. &&

export GIT_EXEC_PATH="$PWD/.." &&
PATH="$GIT_EXEC_PATH:$PATH" &&

rm -rf super clone &&
git init super &&
git init super/sub &&
git -C super/sub commit --allow-empty -m "Init submodule" &&

git -C super submodule add "$PWD/super/sub/.git" sub &&
git -C super commit -m "Added submodule" &&

git clone --recursive super clone &&

git -C super/sub commit --allow-empty -m "Modify submodule" &&

git -C super commit -m "Modified submodule" sub &&

git -C super/sub commit --allow-empty --amend -m "Amend" --no-edit &&

git -C super commit -m "Modified submodule again" sub || exit 125

git -C clone fetch 2>err || {
	grep unadvertised err || exit 125
	exit 1
}

With this, I was able to bisect it down to be76c21.

The commit message of this commit really sounds convincing, and I would argue that it is indeed a good idea to prevent unreachable commits from being fetched. The bigger question is probably why the wrong commit wants to be fetched...

[dscho]

Oh, and please note that this is not a Windows-specific problem, it also reproduces on Linux.

[dscho]

Okay, I analyzed this, and I think Git is actually correct. That commit should have been fetched, and it could not be fetched.

Let me first describe in high-level words what happened:

1. A super project with a submodule was cloned

2. In the upstream project, a change was made in the submodule, and committed in the super project.

3. In the upstream project, the tip commit of the submodule was amended, and again committed in the super project.

   Crucially, the submodule commit was "force-pushed", making the previous tip unreachable, but it is still in the super project's history.

4. In the clone, git fetch was called. Since it pulled down not only the latest super project commit, but also the one before, it wants to make sure that the submodule commits recorded in both are actually fetched in the submodule. But only the latest is reachable, the one that was "amended away" is no longer reachable, and cannot be fetched.

   As a consequence, git checkout HEAD^ && git submodule update would not work, as the submodule's commit could not be fetched.

I think this is a flaw in the submodules, by design.

What should have been done is to not amend the submodule's commit that was already recorded in the super project.

Alternatively, the super project's commit should have been amended, too, i.e. instead of the above

git -C super commit -m "Modified submodule again" sub

the command

git -C super commit --amend sub

should have been called. I understand that this is not always possible, and that's why projects usually enforce the "never force-push master" rule. This rule should have been heeded in the submodule, too (assuming that only commits from the submodule's master are committed in the super project).

In short, I think that this is not a bug in Git, it works as designed. It's just that submodules are a half-baked design that cannot be fixed... But that is my personal opinion about submodules.

If you want to discuss this issue further, I would like to redirect you to the Git mailing list (no HTML mails, not even alternative HTML parts, otherwise your mail will be rejected, plain text only please): as I said earlier, this issue is not specific to Windows (and therefore this here bug tracker isn't the best venue to discuss it).

[YngveNPettersen]

Note that your example is NOT the only case where this can happen.

We observed it when a submodule reference was committed locally, then the branch in the submodule was rebased (e.g. because updates had to be pulled from upstream, because git push failed; normal procedure for projects where branches are pull-rebased, not pull-merged), and the new submodule reference was commited, and then the branch in the submodule was pushed (the resulting commits with submodule reference updates were not squashed), followed by the super module's branch. NOTE: no forced push.

When another user pulls the updates in the super module, the first committed reference points at an unknown reference (because it was never pushed), tiggering the fetch failure, the second reference points to a known reference. The first reference would never be used, only the second one.

Additionally, the second fetch afterwards would not trigger the error. The result is that, for example, an autobuild operation would fail because the fetch failed, but the next operation would continue normally past the fetch operation.

[YngveNPettersen]

There is also my third problem with this message:

The error message provides absolutely no context. Which repository referenced the missing commit? Which commit in the repository did so? Which submodule reference? What is the URL of the repository that failed to provide the commit?

Also, the only ways to repair the problem is either pushing a discarded branch, or a forced push with the bad reference(s) removed from the branch.

[dscho]

Okay, but still, those issues are not specific to Windows. There is really only one active person in the Git for Windows project right now (me) and you will find more than one active one on the Git mailing list, in particular you will even find people who are knowledgeable in submodules...