[x64] inject into different-architecture child: x64 to WOW64, WOW64 to x64

[derekbruening]

From bruen...@google.com on June 12, 2012 09:52:13

this was PR 254193

xref issue #20 : handle execve of different-architecture child: 32 to 64, 64 to 32
xref issue #147 /PR 408318: client interactions with cross-arch execve

xref PR 240257: support 32-bit clients on WOW64? how mix 32 and 64 bit code?
xref issue #49 /PR 253431
xref issue #381 /PR 215423
(xref PR 249112: [x64] inject into child process)
(xref issue #142 /PR 251677: [x64] early inject into child process)

---

old notes which may not all apply:

For deciding which DR to inject, probably should just have 64-bit config
take precedence: if 32-bit config is also present, too bad, it gets
ignored.

For the child-doesn't-equal parent cases we'll have to have the other
architecture's bootstrap code to inject into the child (or for 64-to-32
fool the loader into loading the 32-bit lib if go early and run 64-bit
bootstrap code): will require a cross-compiled asm routine, or gencode, for
inject-into-thread.

Perhaps worth reconsidering what we really want to bother to support here
(PR 240257).

One plan is to never support 32-bit DR following into 64-bit child.
Long-term we'll only support 64-bit-DR in WOW64 following children (PR
253431). Only when a client hasn't been ported will we support 32-bit DR
in a WOW64 process,
So if dr64 is in a process that launches a wow64 child foo.exe where
foo.exe has only a wow64 config, our stance is that we won't inject into
the child (since we don't bother to support cross-arch child injection) and
you'll have to rely on AppInit.

Original issue: http://code.google.com/p/dynamorio/issues/detail?id=803

[derekbruening]

From bruen...@google.com on June 12, 2012 10:28:19

** TODO change existing injection to map from parent but take over later

one challenge of cross-arch injection is generating 64-bit code from 32-bit
DR. the earliest injection method (issue #234) makes this easier b/c there's
much less generated code: it maps the library in from the parent and has
only a small sequence of generated code that its hook targets. a proposal
is to implement cross-arch injection by switching the existing (late)
injection to map up front and just use a later hook point. this will then
make this same injection code easier to make work cross-arch.

64-bit parent with 32-bit child has the complication of a delayed hook for
32-bit ntdll.

xref issue #234

[derekbruening]

From bruen...@google.com on June 21, 2012 09:00:26

turns out that they changed wow64: on vista+ 32-bit ntdll is loaded by the kernel up front (in NtCreateUserProcess). however, a delayed hook is needed for xp/2003 b/c it is wow64.dll (loaded by 64-bit ntdll) who loads 32-bit ntdll there.

[derekbruening]

This was blocking usage in https://groups.google.com/g/dynamorio-users/c/2cAuAhIBR8Y

[derekbruening]

Multi-arch encoding #1684 would help by allowing one arch to build IR for another, although here we need across bitwidths, which is more challenging.

[derekbruening]

There are a number of complexities for trying to inject into a 64-bit child process from a 32-bit parent:

One big one is that we have to adjust all the child process queries to use NtWow64* variants, and handle 64-bit pointer sizes and other field differences.

There's no NtWow64MapViewOfSection64 but we can map into the low 2G.

PEB query: NtWow64QueryInformationProcess64
Image entry query: NtWow64ReadVirtualMemory64

Allocate gencode: NtWow64AllocateVirtualMemory64 and NtWow64QueryVirtualMemory64
Write gencode: NtWow64WriteVirtualMemory64

But for the hook, and gencode -w, we need NtWow64ProtectVirtualMemory64 --
but it doesn't exist!!!

For the hook: do we give up on inserting our hook at image entry (or any
earlier place) and fall back on thread injection and change the thread
context? Or, we could use switch_modes_and_call() to invoke the ntdll64
NtProtectVirtualMemory (have to add support for 5 args).

For our gencode: will our gencode run afoul of W^X policies? Should we
make it static asm code instead inside our library, with all
target-process-address-dependent instructions indirected through .data?
With no hook to restore, the gencode simplifies to pushing a retaddr of the
app start point and calling dynamorio_earliest_init_takeover().

[derekbruening]

Actually NtWow64QueryVirtualMemory64 was removed in win10 so we're in the same boat there as for Protect.

[derekbruening]

I have this implemented now, for everything except XP64 parent32-child64 which I don't think is worth my time.

Pasting in my PR #4653 commit message as documentation since it highlights a few issues that I hit, in particular with a hook at the thread start point and with assumptions for parent64-child32 that the thread start is ntdll32!RtlUserThreadStart:

Adds a long-missing feature: following into a Windows child process of
a different bitwidth.

Switches injection from DR and from drinjectlib (including drrun and
drinject) to use -early_inject_map. This was most easily done by turning
on -early_inject by default as well. However, the -early_inject_location
default is INJECT_LOCATION_ThreadStart, a new "early" injection location
which is the same late takeover point as with thread injection (we could
also use _ImageEntry, which is only very slightly later, but that fails for
.NET and other applications). Switching all injection over to
map-from-the-parent simplifies cross-arch following, as well as making it
easier to shift the takeover point to an earlier spot in the future. This
is a step toward #607 by switching drinjectlib to use map injection; the
takeover point, as mentioned, is still the thread start.

Placing a hook at the thread start causes some stability issues, so instead
of the usual hook for -early_inject_map, for INJECT_LOCATION_ThreadStart we
set the thread context, like thread injection does. The gencode still
restores the hook as a nop, for simplicity. For parent64 child32, we can't
easily locate the thread start, so we assume it's
ntdll32!RtlUserThreadStart (which is also a fallback if anything fails in
other cases; the final fallback is a hook at the image entry, which works
nearly everywhere but not for .NET where the image entry is not reached).

Adds an -inject_x64 option to inject a 64-bit DR lib into a 32-bit
child from a 64-bit parent, but this option is only sketched out and
is not fully supported yet: #49 covers adding tests and official
support.

Adds library swapping code to find the other-bitwidth library, which
assumes a parallel directory structure. Add a new fatal error if the
library for a child is not found.

To support generating code for all 3 child-parent cases (same-same,
32-64, and 64-32), and in particular for 32-64, switches the small
gencode sequence for -early_inject_map from using IR to using raw
bytes. A multi-arch encoder (#1684) would help but we would need
cross-bitwidth support there, which is not on the horizon. Fixes what
look like bugs in the original gencode generation along the way
(s/pc/cur_local_pos/ and s/local_code_buf/remote_code_buf/): it's not
clear how it worked before.

Adds support for several system calls from a 32-bit parent to a 64-bit
child where the desired NtWow64* system call does not exist. We use
switch_modes_and_call() for NtProtectVirtualMemory and
NtQueryVirtualMemory.

Changes all types in the injection code to handle 64-bit addresses in
32-bit code. Adds UNICODE_STRING_32 and
RTL_USER_PROCESS_PARAMETERS_32 for handling 32-bit structures from
64-bit parents. Similarly, adds RTL_USER_PROCESS_PARAMETERS_64 and
PROCESS_BASIC_INFORMATION64.

Adds get_process_imgname_cmdline() capability for 64-bit remote from 32-bit.

Adds get_remote_proc_address() and uses it to look up
dynamorio_earliest_init_takeover() in a child DR.

Finds the remote ntdll base via a remote query memory walk plus remote
image header parsing. This requires adding a switch_modes_and_call()
version of NtQueryVirtualMemory (also mentioned above), which needs
64-bit args: so we refactor switch_modes_and_call() to take in a
struct of all 64-bit fields for the args.

Fixes a few bugs in other routines to properly get the image name and
image entry for 32-bit children of 64-bit parents.

Updates environment variable propagation code to handle a 32-bit
parent and a 64-bit child. Updates a 64-bit parent and 32-bit child
to insert the variables into the 32-bit PEB (64-bit does no good),
which requires finding the 32-bit PEB. This is done via the 32-bit
TEB, using a hack due to what seems like a kernel bug where it has the
TebBaseAddress 0x2000 too low.

Makes environment variable propagation failures fatal and visible,
unlike previously where errors would just result in silently letting
the child run natively. Turns some other prior soft errors into fatal
errors on child takeover.

Moves environment variable propagation to post-CreateUserProcess
instead of waiting for ResumeThread, which avoids having to get the
thread context (for which we have no other-bitwidth support) to figure
out whether it's the first thread in the process or not. We bail on
propagation for pre-Vista where we'd have to wait for ResumeThred.

Generalizes the other-bitwidth Visual Studio toolchain environment
variable setting for use in a new build-and-test other-bitwidth test
which builds dynamorio and the large_options client (to ensure options
are propagated to children; and it has convenient init and exit time
prints) for the other bitwidth, arranges parallel lib dirs, and runs
the other client.

[derekbruening]

I'm documenting some of the errors seen when hooking the thread start (RtlUserThreadStart);

2021-01-05T04:29:20.6113344Z 22: <Application D:\a\dynamorio\dynamorio\build_debug-internal-32\suite\tests\bin\win32.callback.exe (3924).  Internal Error: DynamoRIO debug check failure: ..\core\win32\os.c:1750 doing_detach || ((size == (size_t)ALIGN_FORWARD( ostd->stack_top - (ptr_int_t)ostd->stack_base, PAGE_SIZE) || (size == PAGE_SIZE + (size_t)(ostd->stack_top - ostd->stack_base) && is_wow64_process(NT_CURRENT_PROCESS) 

2021-01-05T04:29:33.7219760Z 33: <Application D:\a\dynamorio\dynamorio\build_debug-internal-32\suite\tests\bin\win32.reload.exe (4780).  Internal Error: DynamoRIO debug check failure: ..\core\win32\syscall.c:2023 !exitproc || check_sole_thread()
2021-01-05T04:29:33.7220987Z 33: (Error occurred @6582 frags)

2021-01-05T04:28:57.0155414Z <Application D:\a\dynamorio\dynamorio\build_debug-internal-64\suite\tests\bin\win32.nativeterminate.exe (6128).  Internal Error: DynamoRIO debug check failure: ..\core\win32\syscall.c:2023 !exitproc || check_sole_thread()

2021-01-05T04:29:04.1714209Z 34: <Application D:\a\dynamorio\dynamorio\build_debug-internal-64\suite\tests\bin\win32.reload.exe (3816).  Internal Error: DynamoRIO debug check failure: ..\core\module_list.c:304 vmvector_overlap(loaded_module_areas, base, base + view_size)

These asserts are all related to thread exit. Is the problem that an early thread hits the hook? But wouldn't the first one there trigger init? Is it a race? Is someone reading that code? Not clear exactly what the issue is. But we're avoiding the hook now in any case.