Regression: MinGit 2.38.0 fails to execute in Windows Nano Server

[mthalman]

• I was not able to find an open or closed issue matching what I'm seeing

Setup

• Which version of Git for Windows are you using? Is it 32-bit or 64-bit?

MinGit 2.38.0: https://github.com/git-for-windows/git/releases/download/v2.38.0.windows.1/MinGit-2.38.0-64-bit.zip

$ git --version --build-options

This fails to run.

• Which version of Windows are you running? Vista, 7, 8, 10? Is it 32-bit or 64-bit?

Windows Nano Server container (mcr.microsoft.com/windows/nanoserver:ltsc2022-amd64)

$ cmd.exe /c ver

Microsoft Windows [Version 10.0.20348.1006]

• What options did you set as part of the installation? Or did you choose the
  defaults?

# One of the following:
> type "C:\Program Files\Git\etc\install-options.txt"
> type "C:\Program Files (x86)\Git\etc\install-options.txt"
> type "%USERPROFILE%\AppData\Local\Programs\Git\etc\install-options.txt"
> type "$env:USERPROFILE\AppData\Local\Programs\Git\etc\install-options.txt"
$ cat /etc/install-options.txt

No options, used defaults.

• Any other interesting things about your environment that might be related
  to the issue you're seeing?

This is a regression. It previously worked in version 2.37.3. There was likely a new dependency that was added to MinGit that's now causing this to fail in Nano Server because that dependency is not installed. Nano Server has a subset of the files that exist in other Windows SKUs. This issue does not occur in Windows Server Core, likely because it has the necessary dependency installed.

Here's a Dockerfile which can repro the issue:

# escape=`
FROM mcr.microsoft.com/windows/nanoserver:ltsc2022-amd64

ARG MINGIT_VERSION=2.38.0

# This version works
# ARG MINGIT_VERSION=2.37.3

RUN curl -SL --output mingit.zip https://github.com/git-for-windows/git/releases/download/v%MINGIT_VERSION%.windows.1/MinGit-%MINGIT_VERSION%-64-bit.zip `
    && mkdir "C:\Program Files\MinGit" `
    && tar -oxzf mingit.zip -C "C:\Program Files\MinGit"

USER ContainerAdministrator
RUN setx /M PATH "%PATH%;C:\Program Files\MinGit\cmd"
USER ContainerUser

RUN git version

Details

• Which terminal/shell are you running Git from? e.g Bash/CMD/PowerShell/other

CMD

• What commands did you run to trigger this issue? If you can provide a
  Minimal, Complete, and Verifiable example
  this will help us understand the issue.

git version

• What did you expect to occur after running these commands?

Outputs the version

• What actually happened instead?

Execution failed with an exit code of -1073741511

• If the problem was occurring with a specific repository, can you provide the
  URL to that repository to help us with testing?

n/a

[dscho]

Here's a Dockerfile which can repro the issue:

I cannot run Docker on my computer. Is there an easy way to reproduce this? Like, something that can be integrated into a GitHub workflow?

[dscho]

It might also shed some light into the issue if you could pinpoint which snapshot introduced the problem.

[mthalman]

I cannot run Docker on my computer. Is there an easy way to reproduce this? Like, something that can be integrated into a GitHub workflow?

No, because Nano Server is only available for containers.

It might also shed some light into the issue if you could pinpoint which snapshot introduced the problem.

I'm happy to help narrow it down to a snapshot. Can you give me a starting point?

[mthalman]

I narrowed it down to the snapshot for commit ec78604. Starting with that snapshot, this behavior begins. It works fine in the previous snapshot (commit c4992d4).

While diagnosing the root cause, if you need help in understanding what DLLs actually exist (or don't exist) in Nano Server, I can provide you with that info. Just give me the path that you want to know about. Unfortunately, Windows doesn't have any documentation that indicates this information.

[mthalman]

@dscho - Any more info I can provide here to help out?

[rimrul]

I narrowed it down to the snapshot for commit ec78604. Starting with that snapshot, this behavior begins.

That sounds like the culprit might be

e56f44a or
59f74e9

and psapi.dll or bcrypt.dll.

[mthalman]

Well, I do know that Nano Server does have bcrypt.dll installed:

> docker run --rm mcr.microsoft.com/windows/nanoserver:ltsc2022 cmd /c dir C:\Windows\System32\bcrypt.dll
 Volume in drive C has no label.
 Volume Serial Number is 881B-79E7

 Directory of C:\Windows\System32

05/08/2021  01:42 AM           169,792 bcrypt.dll
               1 File(s)        169,792 bytes
               0 Dir(s)  21,252,771,840 bytes free

[mthalman]

Ah, missed your update about psapi.dll. Yes, that's the one that's missing.

> docker run --rm mcr.microsoft.com/windows/nanoserver:ltsc2022 cmd /c dir C:\Windows\System32\psapi.dll
 Volume in drive C has no label.
File Not Found
 Volume Serial Number is 881B-79E7

 Directory of C:\Windows\System32

[rimrul]

According to https://learn.microsoft.com/en-us/windows/win32/api/psapi/nf-psapi-getprocessmemoryinfo#remarks

I think we could add a fallback to K32GetProcessMemoryInfo() which was available in nanoserver in 2017 according to https://learn.microsoft.com/en-us/previous-versions/windows/desktop/legacy/mt588480(v=vs.85)

[dscho]

I had a quick look at https://github.com/microsoft/mimalloc/issues?q=is%3Aissue+is%3Aopen+nano but did not find any mention of nano server...

[dscho]

That sounds like the culprit might be

e56f44a

The thing that I do not understand about this is: we are not requiring this DLL, we go out of our way to dynamically load this function, and if it cannot be found, we simply report bogus statistics. And we only do that if mi_process_info() is called (which is not called in general, as far as I can tell).

[KevinCathcart]

on Nano server, Git.exe exits with status code -1073741511 (0xC0000139) which is STATUS_ENTRYPOINT_NOT_FOUND. (Which means some function imported from a dll does not exist).

Running it under a debugger tells me that it is GetNumaProcessorNode that is missing. This being an eager loaded import makes the exe fail at startup, before any user code has run.

The irony is that this call exists only for pre windows 7 compatibility, with the code trying to use the newer GetNumaProcessorNodeEx in preference, which it does via loadlibrary and GetProcAddress.

However Nano server has dropped the older GetNumaProcessorNode and contains only the newer GetNumaProcessorNodeEx.

If Git for Windows does not support OSes that old, I'd suggest just changing the code to call GetNumaProcessorNodeEx directly. If it does support OSes that old, then calling that call dynamically instead of statically could work (if neither call could be loaded, you can just return 1, but that can't actually happen, as any windows old enough to have neither GetNumaProcessorNode nor GetNumaProcessorNodeEx would crash on some other Numa functions called statically (that are not a problem on Nano Server), and in any case would be far too old to be supported by this project..

This might not be the only call that is broken, but I don't have any good way of checking that, since the only way I could find to even determine the missing call was to copy windbg for x64 into the container (the whole C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\windbg.exe folder) run gflags -i git.exe +SLS in the container as ContainerAdmin, and then call cdb.exe -c "g" -c "q" C:\mingit\mingw64\bin\git.exe.

---

If if github's CI runners have windbg installed (it is part of the Windows SDK, but it does not get installed as part of visual studio, while other parts of Windows SDKs are installed by it. The listing of installed software does not make it clear, and I can't check it at the moment), then it would be possible to automatically test for missing static imports (individual functions or entire DLLs) on nano-server if so desired. I could write up an explanation of how this could work later.

Hope this helps.

[dscho]

If Git for Windows does not support OSes that old, I'd suggest just changing the code to call GetNumaProcessorNodeEx directly. If it does support OSes that old, then calling that call dynamically instead of statically could work (if neither call could be loaded, you can just return 1, but that can't actually happen, as any windows old enough to have neither GetNumaProcessorNode nor GetNumaProcessorNodeEx would crash on some other Numa functions called statically (that are not a problem on Nano Server), and in any case would be far too old to be supported by this project..

I fear that this is a bit more complicated than your comment makes it sound.

Git for Windows vendored in mimalloc, and that's the code you're talking about. I do not want to deviate from mimalloc's code unnecessarily (ideally, not at all), so we'll have to work with the mimalloc project to find a solution for this.

The irony is that mimalloc already tries to load GetNumaProcessorNodeEx dynamically, and only uses that GetNumaProcessorNode that it imports statically as a fallback...

As a stop-gap solution, I think we should probably just turn the latter also into a dynamic import.