minject.exe (x64) patching a win/arm64 binary (in a cross-compile situation)

[hjyamauchi]

Hi @daanx,

It seems that when minject.exe (x64) runs (emulated) on a windows/x64 machine against a windows/arm64 binary, it causes two redirect dlls to be loaded (mimalloc-redirect.dll and mimalloc-redirect-arm64.dll, identical arm64 binaries) though minject.exe -l lists the mimalloc-redirect.dll only.

This is encountered when we cross-compile the swift toolchain for win/arm64 on win/x64. Only the x64 minject can be run there because of the lack of the emulation in the reverse direction and there isn't an easy way to run minject-arm64.exe on win/x64.

I think the minject code (not open source, I think) would need to be changed so it consistently looks at the architecture of the target binary rather than the architecture of the machine on which it runs and makes it so that only one of the dll names is needed to be loaded at run time, as opposed to the two.

Could this be fixed (so that minject.exe (x64) running against a windows/arm64 binary to require only one mimalloc-redirect-arm64.dll)? I hope this isn't something too gnarly to fix. A workaround is to bundle both dlls. But it would be nice to be able to bundle just one redirect dll.

By the way, the native arm64 build situation just works as expected with just mimalloc-redirect-arm64.exe.

Thanks.

P.S. The following command log demonstrates the behavior I describe above.

On a Windows/x64 machine

# Take `llvm-objcopy.exe` as an example
# It is an arm64 binary
S:\SourceCache\swift>file "S:\Program Files\swift\Toolchains\0.0.0+Asserts\usr\bin\llvm-objcopy.exe"
S:\Program Files\swift\Toolchains\0.0.0+Asserts\usr\bin\llvm-objcopy.exe: PE32+ executable (console) Aarch64, for MS Windows, 6 sections

# It doesn't run on windows/x64.
S:\SourceCache\swift>"S:\Program Files\swift\Toolchains\0.0.0+Asserts\usr\bin\llvm-objcopy.exe"
This version of S:\Program Files\swift\Toolchains\0.0.0+Asserts\usr\bin\llvm-objcopy.exe is not compatible with the version of Windows you're running. Check your computer's system information and then contact the software publisher.

# Make a temporary copy
S:\SourceCache\swift>cp "S:\Program Files\swift\Toolchains\0.0.0+Asserts\usr\bin\llvm-objcopy.exe" C:\Users\hiroshi\tmp\llvm-objcopy-arm64-patched.exe

# Dump the import table
S:\SourceCache\swift>S:\SourceCache\mimalloc\bin\minject.exe -l "S:\Program Files\swift\Toolchains\0.0.0+Asserts\usr\bin\llvm-objcopy.exe"
original imported modules (14):
   0: ADVAPI32.dll
   1: ntdll.dll
   2: KERNEL32.dll
   3: MSVCP140.dll
   4: VCRUNTIME140.dll
   5: api-ms-win-crt-runtime-l1-1-0.dll
   6: api-ms-win-crt-heap-l1-1-0.dll
   7: api-ms-win-crt-stdio-l1-1-0.dll
   8: api-ms-win-crt-environment-l1-1-0.dll
   9: api-ms-win-crt-string-l1-1-0.dll
  10: api-ms-win-crt-utility-l1-1-0.dll
  11: api-ms-win-crt-convert-l1-1-0.dll
  12: api-ms-win-crt-math-l1-1-0.dll
  13: api-ms-win-crt-locale-l1-1-0.dll

# Patch it
S:\SourceCache\swift>S:\SourceCache\mimalloc\bin\minject.exe -f -i -v C:\Users\hiroshi\tmp\llvm-objcopy-arm64-patched.exe
reading 'C:\Users\hiroshi\tmp\llvm-objcopy-arm64-patched.exe'
original imported modules (14):
   0: ADVAPI32.dll
   1: ntdll.dll
   2: KERNEL32.dll
   3: MSVCP140.dll
   4: VCRUNTIME140.dll
   5: api-ms-win-crt-runtime-l1-1-0.dll
   6: api-ms-win-crt-heap-l1-1-0.dll
   7: api-ms-win-crt-stdio-l1-1-0.dll
   8: api-ms-win-crt-environment-l1-1-0.dll
   9: api-ms-win-crt-string-l1-1-0.dll
  10: api-ms-win-crt-utility-l1-1-0.dll
  11: api-ms-win-crt-convert-l1-1-0.dll
  12: api-ms-win-crt-math-l1-1-0.dll
  13: api-ms-win-crt-locale-l1-1-0.dll

inject 'mimalloc-redirect.dll'
inject 'mimalloc.dll'
leave at position  0: 'mimalloc-redirect.dll'
leave at position  0: 'mimalloc-redirect.dll'
leave at position  1: 'mimalloc.dll'

module order unchanged
wrote (intermediate) 'C:\Users\hiroshi\tmp\llvm-objcopy-arm64-patched-mi.exe' with new import order.
wrote 'C:\Users\hiroshi\tmp\llvm-objcopy-arm64-patched.exe' with new import order.

# Dump the import table after patching
S:\SourceCache\swift>S:\SourceCache\mimalloc\bin\minject.exe -l C:\Users\hiroshi\tmp\llvm-objcopy-arm64-patched.exe
original imported modules (16):
   0: mimalloc-redirect.dll
   1: mimalloc.dll
   2: ADVAPI32.dll
   3: ntdll.dll
   4: KERNEL32.dll
   5: MSVCP140.dll
   6: VCRUNTIME140.dll
   7: api-ms-win-crt-runtime-l1-1-0.dll
   8: api-ms-win-crt-heap-l1-1-0.dll
   9: api-ms-win-crt-stdio-l1-1-0.dll
  10: api-ms-win-crt-environment-l1-1-0.dll
  11: api-ms-win-crt-string-l1-1-0.dll
  12: api-ms-win-crt-utility-l1-1-0.dll
  13: api-ms-win-crt-convert-l1-1-0.dll
  14: api-ms-win-crt-math-l1-1-0.dll
  15: api-ms-win-crt-locale-l1-1-0.dll

Now copy llvm-objcopy-arm64-patched.exe along with mimalloc.dll (arm64) and mimalloc-redirect-arm64.exe to a Windows ARM64 machine.

On the Windows ARM64 machine:

# The redirect dll is an arm64 binary
C:\Users\hiroshi\tmp>file mimalloc-redirect-arm64.dll
mimalloc-redirect.dll: PE32+ executable (DLL) (GUI) Aarch64, for MS Windows, 6 sections

# mimalloc-redirect.dll doesn't exist
C:\Users\hiroshi\tmp>file mimalloc-redirect.dll
mimalloc-redirect-arm64.dll: cannot open `mimalloc-redirect-arm64.dll' (No such file or directory)

# The mimalloc override dll is an arm64 binary
C:\Users\hiroshi\tmp>file mimalloc.dll
mimalloc.dll: PE32+ executable (DLL) (GUI) Aarch64, for MS Windows, 6 sections

# It fails to run due to missing mimalloc-redirect.dll
C:\Users\hiroshi\tmp>.\llvm-objcopy-arm64-patched.exe --version                   

# Move so that we now have only mimalloc-redirect.dll
C:\Users\hiroshi\tmp>mv mimalloc-redirect-arm64.dll mimalloc-redirect.dll

# It fails to run due to missing mimalloc-redirect-arm64.dll
C:\Users\hiroshi\tmp>.\llvm-objcopy-arm64-patched.exe --version                   

# Copy so that two redirect binaries exist
C:\Users\hiroshi\tmp>cp mimalloc-redirect.dll mimalloc-redirect-arm64.dll 

# It runs successfully
C:\Users\hiroshi\tmp>.\llvm-objcopy-arm64-patched.exe --version                    
llvm-objcopy, compatible with GNU objcopy
LLVM (http://llvm.org/):
  LLVM version 19.1.5
  Optimized build with assertions.

# Run it under a debugger, which shows that two redirect dlls are loaded
C:\Users\hiroshi\tmp>windbgx .\llvm-objcopy-arm64-patched.exe --version

Microsoft (R) Windows Debugger Version 10.0.27725.1000 ARM64
Copyright (c) Microsoft Corporation. All rights reserved.

CommandLine: .\llvm-objcopy-arm64-patched.exe --version

************* Path validation summary **************
Response                         Time (ms)     Location
Deferred                                       srv*
Symbol search path is: srv*
Executable search path is: 
ModLoad: 00007ff7`c7c20000 00007ff7`c8151000   image00007ff7`c7c20000
ModLoad: 00007ffe`54140000 00007ffe`54560000   ntdll.dll
ModLoad: 00007ffe`52200000 00007ffe`52364000   C:\Windows\System32\KERNEL32.DLL
ModLoad: 00007ffe`4f930000 00007ffe`4ff5e000   C:\Windows\System32\KERNELBASE.dll
ModLoad: 00007ffe`50650000 00007ffe`5078f000   C:\Windows\System32\ADVAPI32.dll
ModLoad: 00007ffe`50e00000 00007ffe`50f48000   C:\Windows\System32\msvcrt.dll
ModLoad: 00007ffe`2a150000 00007ffe`2a16a000   C:\Users\hiroshi\tmp\mimalloc-redirect.dll           # mimalloc-redirect loaded
ModLoad: 00007ffe`23aa0000 00007ffe`23ac6000   C:\Users\hiroshi\tmp\mimalloc.dll
ModLoad: 00007ffe`51c80000 00007ffe`51d9e000   C:\Windows\System32\sechost.dll
ModLoad: 00007ffe`53cb0000 00007ffe`53ead000   C:\Windows\System32\RPCRT4.dll
ModLoad: 00007ffe`50230000 00007ffe`50458000   C:\Windows\System32\ucrtbase.dll
ModLoad: 0000024c`cf100000 0000024c`cf328000   C:\Windows\System32\ucrtbase.dll
ModLoad: 00007ffe`1e710000 00007ffe`1e72a000   C:\Users\hiroshi\tmp\mimalloc-redirect-arm64.dll   # mimalloc-redirect-arm64 loaded
ModLoad: 00007ffe`3d0f0000 00007ffe`3d248000   C:\Windows\SYSTEM32\MSVCP140.dll
ModLoad: 00007ffe`3d250000 00007ffe`3d285000   C:\Windows\SYSTEM32\VCRUNTIME140.dll
(6508.6c88): Break instruction exception - code 80000003 (first chance)
ntdll!LdrpDoDebuggerBreak+0x34:
00007ffe`542963cc d43e0000 brk         #0xF000

[daanx]

Ah - that is a bit tricky. When we build minject, we use the Windows headers that contain the structures inside a PE file (and thus the correct offsets etc. If we target a different architecture binary these need to stay the same? Maybe that is the case though. If so, we could just add an option to minject to rename the redirect dll name I guess? (so, on x64 you would invoke minject -redirectpostfix=arm64 my program.arm64.exe. to inject (arm64) mimalloc.dll and (arm64) mimalloc-redirect-arm64.dll )

[daanx]

I pushed an update to dev3 that has a new minject (v1.4) which takes the target architecture into account -- so on the arm64 exe it should use the arm64 redirect dll. Let me know if it works for you.