PortableGit 2.10.2.1 crashes with 0xc0000005

[orthoxerox]

• [V] I was not able to find an open or closed issue matching what I'm seeing

Setup

• Which version of Git for Windows are you using? Is it 32-bit or 64-bit?
  32-bit portable Git, tried both PortableGit-2.10.2-32-bit.7z.exe and PortableGit-2.11.0-32-bit.7z.exe

$ git --version --build-options
*crashes*

• Which version of Windows are you running? Vista, 7, 8, 10? Is it 32-bit or 64-bit?

$ cmd.exe /c ver
Microsoft Windows [Version 6.1.7601]

• What options did you set as part of the installation? Or did you choose the
  defaults?

I'm using PortableGit

• Any other interesting things about your environment that might be related
  to the issue you're seeing?

I've tried with both the default and the minified PATH:

PATH=C:\Software\PortableGit\cmd

Symantec Endpoint Protection was active, but didn't report anything.

Group policies prevent execution of programs located outside whitelisted directories (C:\Software is whilelisted).

Details

• Which terminal/shell are you running Git from? e.g Bash/CMD/PowerShell/other

git-cmd.exe

• What commands did you run to trigger this issue? If you can provide a
  Minimal, Complete, and Verifiable example
  this will help us understand the issue.

git

• What did you expect to occur after running these commands?

Git reporting its command line options.

• What actually happened instead?

Git for Windows has stopped working

From the event log:

Faulting application name: git.exe, version: 2.10.2.1, time stamp: 0x0015a000
Faulting module name: git.exe, version: 2.10.2.1, time stamp: 0x0015a000
Exception code: 0xc0000005
Fault offset: 0x0017b2aa
Faulting process id: 0x24e8
Faulting application start time: 0x01d276ee97f4afc4
Faulting application path: C:\Software\PortableGit\mingw32\bin\git.exe
Faulting module path: C:\Software\PortableGit\mingw32\bin\git.exe
Report Id: d7c8f363-e2e1-11e6-89c9-80c16ee7919c

From Report.wer:

Version=1
EventType=APPCRASH
EventTime=131298106310149037
ReportType=2
Consent=1
UploadTime=131298106310979120
ReportIdentifier=d7c8f364-e2e1-11e6-89c9-80c16ee7919c
IntegratorReportIdentifier=d7c8f363-e2e1-11e6-89c9-80c16ee7919c
Response.type=4
Sig[0].Name=Application Name
Sig[0].Value=git.exe
Sig[1].Name=Application Version
Sig[1].Value=2.10.2.1
Sig[2].Name=Application Timestamp
Sig[2].Value=0015a000
Sig[3].Name=Fault Module Name
Sig[3].Value=git.exe
Sig[4].Name=Fault Module Version
Sig[4].Value=2.10.2.1
Sig[5].Name=Fault Module Timestamp
Sig[5].Value=0015a000
Sig[6].Name=Exception Code
Sig[6].Value=c0000005
Sig[7].Name=Exception Offset
Sig[7].Value=0017b2aa
DynamicSig[1].Name=OS Version
DynamicSig[1].Value=6.1.7601.2.1.0.256.4
DynamicSig[2].Name=Locale ID
DynamicSig[2].Value=1049
DynamicSig[22].Name=Additional Information 1
DynamicSig[22].Value=0a9e
DynamicSig[23].Name=Additional Information 2
DynamicSig[23].Value=0a9e372d3b4ad19135b953a78882e789
DynamicSig[24].Name=Additional Information 3
DynamicSig[24].Value=0a9e
DynamicSig[25].Name=Additional Information 4
DynamicSig[25].Value=0a9e372d3b4ad19135b953a78882e789
UI[2]=C:\Software\PortableGit\mingw32\bin\git.exe
UI[3]=Git for Windows has stopped working
UI[4]=Windows can check online for a solution to the problem.
UI[5]=Check online for a solution and close the program
UI[6]=Check online for a solution later and close the program
UI[7]=Close the program
LoadedModule[0]=C:\Software\PortableGit\mingw32\bin\git.exe
LoadedModule[1]=C:\windows\SYSTEM32\ntdll.dll
LoadedModule[2]=C:\windows\system32\kernel32.dll
LoadedModule[3]=C:\windows\system32\KERNELBASE.dll
LoadedModule[4]=C:\windows\system32\ADVAPI32.dll
LoadedModule[5]=C:\windows\system32\msvcrt.dll
LoadedModule[6]=C:\windows\SYSTEM32\sechost.dll
LoadedModule[7]=C:\windows\system32\RPCRT4.dll
LoadedModule[8]=C:\windows\system32\USER32.dll
LoadedModule[9]=C:\windows\system32\GDI32.dll
LoadedModule[10]=C:\windows\system32\LPK.dll
LoadedModule[11]=C:\windows\system32\USP10.dll
LoadedModule[12]=C:\windows\system32\WS2_32.dll
LoadedModule[13]=C:\windows\system32\NSI.dll
LoadedModule[14]=C:\Software\PortableGit\mingw32\bin\LIBEAY32.dll
LoadedModule[15]=C:\Software\PortableGit\mingw32\bin\libgcc_s_dw2-1.dll
LoadedModule[16]=C:\Software\PortableGit\mingw32\bin\libwinpthread-1.dll
LoadedModule[17]=C:\Software\PortableGit\mingw32\bin\libiconv-2.dll
LoadedModule[18]=C:\Software\PortableGit\mingw32\bin\libintl-8.dll
LoadedModule[19]=C:\Software\PortableGit\mingw32\bin\libpcre-1.dll
LoadedModule[20]=C:\Software\PortableGit\mingw32\bin\zlib1.dll
LoadedModule[21]=C:\Software\PortableGit\mingw32\bin\libssp-0.dll
LoadedModule[22]=C:\windows\system32\IMM32.DLL
LoadedModule[23]=C:\windows\system32\MSCTF.dll
LoadedModule[24]=C:\windows\system32\CRYPTSP.dll
LoadedModule[25]=C:\windows\system32\rsaenh.dll
LoadedModule[26]=C:\windows\system32\CRYPTBASE.dll
FriendlyEventName=Stopped working
ConsentKey=APPCRASH
AppName=Git for Windows
AppPath=C:\Software\PortableGit\mingw32\bin\git.exe

Git obtained via msys2 pacman on the same machine works without errors:

git version 2.10.2
sizeof-long: 4

• If the problem was occurring with a specific repository, can you provide the
  URL to that repository to help us with testing?

No repository necessary.

[dscho]

If you can provide a Minimal, Complete, and Verifiable example this will help us understand the issue.

git

Sadly, this is not verifiable, as it does not trigger the crash here.

Do you have any stack trace or something? Does v2.11.0(3) crash also?

Git obtained via msys2 pacman

Which Git? pacman -S git? or pacman -S mingw-w64-i686-git? MSYS2's, or the one from Git for Windows' Pacman repository?

[orthoxerox]

Yes, v2.11.0(3) crashes as well.

I've got a crash dump:

*** ERROR: Symbol file could not be found.  Defaulted to export symbols for ntdll.dll - 
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(1578.2890): Access violation - code c0000005 (first/second chance not available)
eax=00000000 ebx=0038f544 ecx=00000400 edx=00000000 esi=00000002 edi=00000000
eip=77346bf4 esp=0038f4f4 ebp=0038f590 iopl=0         nv up ei pl zr na pe nc
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00000246
ntdll!KiFastSystemCallRet:
77346bf4 c3              ret
0:000> .ecxr
eax=01535e68 ebx=01535e68 ecx=01536a64 edx=0038fca4 esi=0158a4d8 edi=68cc601c
eip=01533e3a esp=0038fc90 ebp=0038fd88 iopl=0         nv up ei pl nz na po nc
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00010202

 # ChildEBP RetAddr  
WARNING: Stack unwind information not available. Following frames may be wrong.
00 0038fd88 013b127e git+0x183e3a
01 0038fe50 771def1c git+0x127e
02 0038fe5c 77363648 kernel32!BaseThreadInitThunk+0x12
03 0038fe9c 7736361b ntdll!RtlInitializeExceptionChain+0xef
04 0038feb4 00000000 ntdll!RtlInitializeExceptionChain+0xc2

I tried getting a trace with gdb, but it couldn't catch the crash.

Git obtained via pacman -S git using pacman from MSYS2 works without any errors.

[orthoxerox]

One more update. I used my existing MSYS2 installation to build Git from git-2.11.0.windows.3.tar.gz tarball. When I replaced git.exe in PortableGit\mingw32\bin\ everything worked even with a minimal PATH (to avoid potential MSYS2 dlls). The executable was much larger than in the portable version, though (8 449 290 bytes).

[dscho]

Git obtained via pacman -S git using pacman from MSYS2 works without any errors.

That is not surprising, as it is a completely different beast: it uses the MSYS2 runtime, i.e. the POSIX emulation layer based on Cygwin.

I used my existing MSYS2 installation to build Git from git-2.11.0.windows.3.tar.gz tarball.

I am not sure that this results in correct binaries. I always build with the Git for Windows SDK, which is a fork of MSYS2 with subtle, but important changes relative to MSYS2.

Another thing to try: does it work when you call git-bash.exe instead of git-cmd.exe?

[orthoxerox]

Nope, I get a segmentation fault.

[orthoxerox]

One more tidbit. I've been trying to check what post-install does, and running it step by step, and this caught my attention:

/etc/post-install/13-copy-dlls.post: line 8: git: command not found

I copied the DLLs from mingw32\bin to mingw32\libexec\git-core, but this simply means that git.exe in the latter directory now crashes just as well as the other one instead of whining about missing DLLs.

I have obtained a fresh standard VM and will see if installing git using the installer works.

[orthoxerox]

It looks like rebasing is to blame for that crash (or rather, security software). I started the installer and everything worked fine during installation until it ran autorebase and couldn't clone the repo because git crashed. DeviceLock software doesn't like rebased DLLs for some reason.

UPD: Actually, not really. It's /cmd/git.exe that crashes, not /usr/bin/git.exe. And it doesn't look like it's built from scratch by the SDK for the new version.

UPD: I found the files in MINGW-packages\mingw-w64-git, trying to build them now , but there's nothing in readme about building them for 32-bit arch. Will try to build them by hand.

UPD: the current stumbling block is

    CC git-wrapper.o
    LINK git-wrapper.exe
collect2.exe: error: ld returned 5 exit status
make: *** [git-wrapper.mak:4: git-wrapper.exe] Error 1

UPD: I have a log of the linker with -t and --verbose, but I am not sure if it can help in any way. Looks like git-wrapper.c contains some calls that are unsupported by the security software that is running on the PC.

I'll see what I can do with the support service of the security s/w.

UPD: I couldn't get the debug symbols for the git-wrapper, but that's what causes the violation exception, according to WinDBG:

 # ChildEBP RetAddr  Args to Child              
WARNING: Stack unwind information not available. Following frames may be wrong.
00 0035f8b0 77dc21e2 0035f8cc 0000000c 77db6d7a ntdll!RtlInitUnicodeStringEx+0x26
01 0035f8d4 75d18c19 0000000c 0035f918 0035f900 ntdll!LdrLoadDll+0xaf
02 0035f910 71e5c92e 00000000 00000800 0000000c KERNELBASE!FreeLibrary+0xb7
03 0035f930 71e5c897 00000002 71e7baf8 00000000 UMEngx86!RegQueryValueExW+0x36bbe
04 0035f948 71e5cbe3 00000008 71e6f204 71e744ac UMEngx86!RegQueryValueExW+0x36b27
05 0035f964 71e5c69d 71e7baf8 00000fa0 00000000 UMEngx86!RegQueryValueExW+0x36e73
06 0035f988 71e55592 00000000 a5720762 00000001 UMEngx86!RegQueryValueExW+0x3692d
07 0035f9c4 71e55567 71e20000 0035fd14 0035fa14 UMEngx86!RegQueryValueExW+0x2f822
08 0035f9d4 71e55767 71e20000 00000001 0035fd14 UMEngx86!RegQueryValueExW+0x2f7f7
09 0035fa14 71e55859 71e20000 00000001 0035fd14 UMEngx86!RegQueryValueExW+0x2f9f7
0a 0035fa28 77db8854 71e20000 00000001 0035fd14 UMEngx86!RegQueryValueExW+0x2fae9
0b 0035fa48 77dc5b3f 71e5583d 71e20000 00000001 ntdll!wcsncmp+0x58
0c 0035fb3c 77dc6071 0035fd14 7ffdf000 7ffd6000 ntdll!EtwEventRegister+0x135
0d 0035fca0 77dc5f6f 0035fd14 77d60000 7d7e5e1a ntdll!LdrQueryImageFileKeyOption+0x1d7
0e 0035fcf0 77dc34e8 0035fd14 77d60000 00000000 ntdll!LdrQueryImageFileKeyOption+0xd5
0f 0035fd00 00000000 0035fd14 77d60000 00000000 ntdll!LdrInitializeThunk+0x10

[dscho]

Symantec Endpoint Protection was active, but didn't report anything.

According to https://cygwin.com/faq/faq.html#faq.using.bloda that may be the problem. The symptom that it only starts seg-faulting as soon as the shell is involved is a strong hint: that is exactly when Git enters the MSYS2 runtime (which is a close derivative of the Cygwin runtime, hence the Cygwin FAQ).

[dscho]

I assume v2.12.2 is also crashing?

[orthoxerox]

Yes, it's crashing as well. I think I'll close this issue, since it's a defect in DL, not Git for Windows.

[orthoxerox]

I checked, and git version 2.6.0.windows.1 is not crashing. I guess I'll see what changes were done between it and 2.10 to the git-wrapper

[orthoxerox]

I've checked the diffs, and it looks like this commit is the problem: f8e286e

Only it and 29df4bb and 514335e introduced any changes between 2.10.2 and 2.6.0 and it's the one that is doing something suspicious from security PoV.

UPD: or maybe not. A program that only consists of SetConsoleCtrlHandler invocations runs without a single error. Could there be an error in memmove argument calculations?

[dscho]

f8e286e

That only sets/removes the console Ctrl+C handler. That cannot really cause a segfault, or raise any suspicions.

29df4bb

This one does not change any functionality, it simply stores the length in a separate variable instead of passing the number of bytes directly.

514335e

This one only increments atat2 (i.e. the pointer that points to the second pair of at signs when substituting a @@VARIABLENAME@@ construct) instead of adding 2 every time atat2 is referenced afterwards. So it can't be that, either.

Except. Except it also changes the guard of the memmove(): it is now only done when delta > 0... And I think there is a missing equivalent after the GetEnvironmentVariableW() for delta < 0...

I'll try some things (as time allows).

[dscho]

Oh, wait... Your report stated that the crash happened in mingw32\bin\git.exe. The git-wrapper.c code is not active in that .exe, but only in the cmd\git.exe...

So I fear we chased the wrong unicorn here.

One thing we did change at some stage the may possibly be a little problem is that we turned on the stack smasher to detect when other code plays games with our stack. That was in Git v2.6.3...

[orthoxerox]

I've copied /cmd/*, /bin/*, /git-bash.exe, git-cmd.exe from a 2.6.0 folder to a 2.10.2.1 folder and I'm still getting crashes. So it does look like it's not the wrapper.