Use symbolic links for libexec

[orgads]

• I was not able to find an open or closed issue matching what I'm seeing

Is it possible to deploy Git with symbolic links in libexec/git-core instead of many hard copies of the git executable?

[dscho]

Only if you build Git yourself.

However:

instead of many hard copies of the git executable?

This is incorrect. Those are not copies, those are hard links. It is true that Windows Explorer double-counts them, but they do not occupy more disk space than one copy + 127 directory entries pointing to the same start disk sector. You can verify this by running the following command:

$ ls -l /mingw64/bin/git.exe
-rwxr-xr-x 127 johannes 197121 2755352 Dec 15 09:52 /mingw64/bin/git.exe*

See that 127? That's the number of directory entries pointing at the same file contents.

[PhilipOakley]

@organds, Any suggestions as to where the best place(s) would be to highlight the information?

It is a question that does come up moderately frequently, so as some one who has been hit by the issue, maybe have a think about where you looked (or even if didn't 'look', why that was) so that we can educate folks and reduce the misunderstandings. Every little helps.

[Would it be worth adding a line to the 'Known Issues' of the Release notes file:///C:/Program%20Files/Git/ReleaseNotes.html, just the head the issue off at the pass]

[orgads]

See that 127? That's the number of directory entries pointing at the same file contents.

I see. So you have one extra executable: libexec/git-core/git.exe:

OrgadS@isr020134037 MINGW64 /c/Program Files/Git/mingw64/libexec/git-core
$ ls -li ../../bin/git.exe git-add.exe git.exe
 34902897112204028 -rwxr-xr-x 127 OrgadS 1049089 2755352 Dec 24 12:14 ../../bin/git.exe*
194499208907169829 -rwxr-xr-x   1 OrgadS 1049089 2755352 Dec 24 12:14 git.exe*
 34902897112204028 -rwxr-xr-x 127 OrgadS 1049089 2755352 Dec 24 12:14 git-add.exe*

@PhilipOakley : Maybe add an empty file inside libexec/git-core named ___MOST-EXECUTABLES-ARE-HARDLINKS___. That would have attracted my attention :)

[dscho]

@orgads yeah, no. I don't think that would be a good idea. First of all, this file would have to be generated only if the files really are hardlinks, and then that file would have to be left out e.g. when copying those files. That's just too fragile.

At some point, we have to bite the bullet and just accept that most users do not really understand their system and need the explanation. I just hope that the number of contributors who relieve the maintainer of explaining this (in favor of chasing much harder bugs, or in favor of trying to hammer down work hours to a normal level) will continue to increase.

[PhilipOakley]

@dscho wrote: just accept that most users do not really understand their system and need the explanation (that the "use links in libexec/git-core instead of many hard copies of the git executable" is already done via Windows hardlinks).

Would I be reading this right, that my suggestion of a short(ish) extra line in the 'known issues' section could be that explanation? It may only find 40% of the users but at least its a reduction in complaints... (Should I add it to my list of things to do on retiral;-)

[dscho]

@PhilipOakley sounds good ;-)

[dscho]

Addressed via git-for-windows/build-extra#229

[orgads]

What about this issue?

I see. So you have one extra executable: libexec/git-core/git.exe:

OrgadS@isr020134037 MINGW64 /c/Program Files/Git/mingw64/libexec/git-core
$ ls -li ../../bin/git.exe git-add.exe git.exe
 34902897112204028 -rwxr-xr-x 127 OrgadS 1049089 2755352 Dec 24 12:14 ../../bin/git.exe*
194499208907169829 -rwxr-xr-x   1 OrgadS 1049089 2755352 Dec 24 12:14 git.exe*
 34902897112204028 -rwxr-xr-x 127 OrgadS 1049089 2755352 Dec 24 12:14 git-add.exe*

[PhilipOakley]

I'm updating the PR (#229 above) that adds this to the Known Issues list (assuming you are commenting about the "127" factor, rather than something I haven't understood)

[orgads]

I don't refer to the factor. I refer to the additional executable libexec/git-core/git.exe, which is not a hard-link.

[dscho]

I refer to the additional executable libexec/git-core/git.exe, which is not a hard-link.

That's a bug, and if you can figure out a fix, I'll be happy to accept a PR.

[dscho]

(To clarify: this extra git.exe should also be a hardlink.)

[orgads]

First observation: make install creates all as hard links, so it's probably something in the installer...

[dscho]

It seems that the mingw-w64-git package has the hardlinks correct:

$ tar tvf mingw-w64-x86_64-git-2.21.0.1.2481c4cbe9-1-any.pkg.tar.xz
|grep 'git\.exe'
-rwxr-xr-x ContainerAdministrator/380929  40728 2019-02-26 20:48 cmd/git.exe
hrwxr-xr-x ContainerAdministrator/380929       0 2019-02-26 20:48 mingw64/bin/git.exe link to mingw64/bin/git-receive-pack.exe
hrwxr-xr-x ContainerAdministrator/380929       0 2019-02-26 20:48 mingw64/libexec/git-core/git.exe link to mingw64/bin/git-receive-pack.exe

Meaning: something must be going wrong when the installer does its magic. It does call HardlinkOrCopyGit(), or at least is supposed to, for all builtins here.

[dscho]

So the builtins are determined from the file /mingw64/share/git/builtins.txt, read here. And that file does not contain /mingw64/libexec/git-core/git.exe here.

[dscho]

That builtins.txt file is generated during installer/release.sh <version> time, and I think it probably determines which files to include by looking which files were replaced by the git-wrapper.exe in git-extra's post-install script.

You see, we got so many concerns about that "massive" filesystem usage that we started to ship at least our portable Git with the builtins replaced not by hardlinked versions of git.exe, but by the Git wrapper (which basically just re-spawns the real git.exe with the builtin's command name inserted as first parameter). And that is why git-extra's post-install script edits the files installed by mingw-w64-git.

[dscho]

Oh, and I was wrong about installer/release.sh generating that builtins file. It is https://github.com/git-for-windows/MINGW-packages/blob/01d3f0e810b7e5a37488d97c860ba3df20685412/mingw-w64-git/PKGBUILD#L154

Now, the reason why it does not include /mingw64/libexec/git-core/git.exe is obvious: that is not a builtin, if we replaced that by the Git wrapper, it would either cause an infinite loop, or it would not find the real git.exe to execute.

I guess the best idea would be to teach git-wrapper.c to detect when it was called as .../libexec/git-core/git.exe and spawn .../bin/git.exe in that case, much like /cmd/git.exe does.

Now, https://github.com/git-for-windows/MINGW-packages/blob/01d3f0e810b7e5a37488d97c860ba3df20685412/mingw-w64-git/git-wrapper.c#L696 tries to be very careful to strip only one directory level (the cmd part of C:\Program Files\Git\cmd\git.exe) in order to look whether we are in an MSYS2-based setup (essentially by looking whether <top-level-directory>\mingw64\bin exists).

To extend this to /mingw64/libexec/git-core/git.exe, we would have to use 3 instead.

And then we could teach the print-builtins target in the mingw-w64-git.mak file to also print that path.

Of course, this all would need to be tested by

1. patching MINGW-packages,
2. building and installing that mingw-w64-x86_64-git package
3. re-installing git-extra (to replace the builtins with the Git wrapper)
4. verifying that /mingw64/libexec/git-core/git.exe works as intended.

Seeing as I spent the last 38 minutes investigating it this far, I now think I should be doing more pressing things, and leave the rest to interested parties (I am looking at you, @orgads 😄 )