git.exe parent process locks index file for child git.exe process

[hzshlomi]

I did see similar closed issues

Setup

MINGW64
$ git --version
git version 2.7.4.windows.1
Win7 64 bit

• What options did you set as part of the installation? Or did you choose the
  defaults?
  pull+rebase

Details

• Which terminal/shell are you running Git from?
  Bash (MINGW64), no other GUI tool installed
• What commands did you run to trigger this issue? If you can provide a
  git pull
• What did you expect to occur after running these commands?
  to be updated against remote
• What actually happened instead?
  it triggered "Auto packing" and failed (see output message below), while the process holding the lock was .. "git.exe":
  git.exe (pid=34040; "git pull")
  -> git.exe (pid=28552; "git fetch -update-head-ok")
  -> git.exe (pid=12732; "git gc -auto")
  -> git.exe (pid=12732; "git repack -d -l -A --unpack-unreachable=2.weeks.ago")

where 34040 holds "006efeeb01a6252adb97aafb1cbb04232f915794.idx",
and 12732 fails to unlink it.

The output:
Auto packing the repository in background for optimum performance.
See "git help gc" for manual housekeeping.
Counting objects: 309941, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (72671/72671), done.
Writing objects: 100% (309941/309941), done.
Total 309941 (delta 186940), reused 301892 (delta 181069)
Unlink of file '.git/objects/pack/pack-006efeeb01a6252adb97aafb1cbb04232f915794.idx' failed. Should I try again? (y/n) y

[dscho]

I did see similar closed issues

Which ones?

git version 2.7.4.windows.1

This version was released on March 18th, and there have been no less than four releases since then, each of which with its respective number of fixes.

The smart thing would be to verify that this is still an issue with Git for Windows v2.8.3.

What commands did you run to trigger this issue? If you can provide a
git pull

It is unlikely that this comes as a surprise to you: this does not reproduce the problem here.

[hzshlomi]

Which ones?

#500, #446, #233
https://github.com/git-for-windows/git/issues?q=Unlink+of+file+failed+is%3Aclosed

What commands

I am really sorry to inform you that the sequence of command used to trigger the issue is just "git pull". I will try harder next time.

To get serious, I thought the info I provided does give a good lead to find the issue. Now I dont mind installing another version, providing logs, or whatever, but I do not feel that you actually followed the information I did provide.

Do you want me to install v2.8.3?
Thanks.

[dscho]

I am really sorry to inform you that the sequence of command used to trigger the issue is just "git pull"

You do not need to be sorry about this. All it means is that I cannot reproduce this issue here, and therefore I am unlikely to be able to help. So it is me who is sorry.

[bwijen]

where 34040 holds "006efeeb01a6252adb97aafb1cbb04232f915794.idx",
and 12732 fails to unlink it.

@dscho: This looks like the same cause as #755 (which I think also holds for the other issues)

On both linux and windows file handles are inherited by child processes,
however deleting an inherited file on linux works, on windows it does not.

Might we extend the solution of #755 to all open calls?
(Opening with O_CLOEXEC / O_NOINHERIT that is.)

That would of course mean, figuring out if we can make git-upload-pack et al. work with this...

[dscho]

@bwijen but that O_NOINHERIT flag would not close the handle when spawning fetch, would it?

I guess that we need to insert a close_all_packs(); before https://github.com/git-for-windows/git/blob/v2.8.4.windows.1/builtin/pull.c#L851, but we would need a real MCVE to prove that this solves the problem. An MCVE that works on every computer, preferably as a test in Git's own regression test suite.

Which would require some real work... And somehow I wonder whether @hzshlomi will do that.

[hzshlomi]

@dscho you know, at this point, it actually gets interesting.
You need a "real MCVE"? from me?? you are the code maintainer -> this is a bug you should fix -> why won't you provide one? do you always expect users of your code to write code? in GIT's case that may work but I just dont understand that approach.

look here: http://stackoverflow.com/help/mcve.
""When asking a question about a problem caused by your code, you will get much better answers if you provide code people can use to reproduce the problem""
I think it's you that is expected to provide a "real MCVE", and yet if you really cant find the energy to do that, just define what you really need and I will.

[PhilipOakley]

@hzshlomi Just to say, 'be careful of what you wish for'.

Given that Git is not designed for Windows, and that without @dscho's continued perseverance, we would not even have what we have now.

Research does show that it's only by using tests first view of Test Driven Development (and in this case, particularly for tricky 'bug' cases) that you actually get any improvement in efficiency.

So this problem need a "Test", which is what an MVCE is. Without an easy way to poke the bug, we won't know if we've found it, or fixed it, nor let it come back via some regression in future change.

So anything you can do to even create a 30 (or however many it takes) line test script that will 'just work' in the SDK would be a start, so then it can be tightened and tightened like a noose around the bug. This may require a bit of google archaeology to bring together all the historic bits. Someone, anyone, need to provide some time into the problem, if they have it. For that is the limited resource. @dscho can do wonders when pointed in the right direction by a good MVCE, but he is a VOLUNTEER.

So if you can volunteer a short test script 'that just runs' easily, then that would be a start. I appreciate your time may be as limited as dscho's but if you need it fixed ...

[hzshlomi]

@PhilipOakley
Well, that is a totally different tone altogether
Truth be told, I don't "need" this fix as in I cant live without it.
Now I had no clue @dscho is a volunteer, and I do take this chance to say this as explicit as it gets:

@dscho, thank you very much for the wonderful job you guys are doing for the community. I mean every word, no cynicism, not one bit. I really do appreciate every second of your time put into it.

Having that said, the reason I reported is that I know other people that faced the issue and had varying degrees of success walking around the problem (I just use Win, Linux interchangeably).

So, to a lesser degree, I was too giving a very modest contribution, not looking for a free service but donating an input.

While doing so, I do not expect reactions like:

"It is unlikely that this comes as a surprise to you: this does not reproduce the problem here."

That tone is not called for. If you decide to contribute, go ahead. Mocking is not a contribution.
Now, I really just did a "pull" so I wasn't just being lazy collecting information.

I also did and still do am willing to put some time into it, just like I wrote before.

If I knew the code, I would do it, but I don't.

@PhilipOakley Thank you too!

[bwijen]

@bwijen but that O_NOINHERIT flag would not close the handle when spawning fetch, would it?

Hmm, you are right...

I guess that we need to insert a close_all_packs(); before https://github.com/git-for-windows/git/blob/v2.8.4.windows.1/builtin/pull.c#L851

@dscho: If that is the case, this needs to be fixed upstream right?

I will try to create a mcve, to see if I'm right about the cause...

[dscho]

If that is the case, this needs to be fixed upstream right?

@bwijen eventually, yes. The way I prefer to work on Windows-related issues is to get them fixed in Git for Windows first, to be able to verify the fix, and then (leisurely) submit the fixes upstream.

I will try to create a mcve, to see if I'm right about the cause...

Awesome, thank you! (happy)

[hzshlomi]

@dscho

$ cat mcve-770.sh
#!/bin/bash

git init origin
cd origin
touch test1.txt
git add test1.txt
git commit -m v1
originpath=pwd
cd ..
git clone file://$originpath clone
cd origin
cd ../clone
git config gc.autoPackLimit 1
git pull

$ ./mcve-770.sh
Initialized empty Git repository in C:/Program Files/Git/git-for-windows-issue-770/origin/.git/
[master (root-commit) e9c35dd] v1
1 file changed, 0 insertions(+), 0 deletions(-)
create mode 100644 test1.txt
Cloning into 'clone'...
remote: Counting objects: 3, done.
remote: Total 3 (delta 0), reused 0 (delta 0)
Receiving objects: 100% (3/3), done.
Checking connectivity... done.
Auto packing the repository in background for optimum performance.
See "git help gc" for manual housekeeping.
Counting objects: 3, done.
Writing objects: 100% (3/3), done.
Total 3 (delta 0), reused 3 (delta 0)
Unlink of file '.git/objects/pack/pack-7f065b66383f8af58ac120594ed27ba48fdb98ea.pack' failed. Should I try again? (y/n)

[hzshlomi]

@dscho

$ git --version
git version 2.8.4.windows.1

$ ./mcve-770.sh
Initialized empty Git repository in C:/Program Files/Git/git-for-windows-issue-770/origin/.git/
[master (root-commit) d484a7a] v1
1 file changed, 0 insertions(+), 0 deletions(-)
create mode 100644 test1.txt
Cloning into 'clone'...
remote: Counting objects: 3, done.
remote: Total 3 (delta 0), reused 0 (delta 0)
Receiving objects: 100% (3/3), done.
Checking connectivity... done.
Auto packing the repository in background for optimum performance.
See "git help gc" for manual housekeeping.
Counting objects: 3, done.
Writing objects: 100% (3/3), done.
Total 3 (delta 0), reused 3 (delta 0)
Unlink of file '.git/objects/pack/pack-fee13c2e332a865c17c4e4204fcab00e7b292894.pack' failed. Should I try again? (y/n)