App running +rwx code at DR injection time crashes #5198

eranzim · 2021-11-09T21:34:05Z

Describe the bug
I tried running my code with winafl and DynamoRIO in several ways (including both drrun.exe and afl-fuzz.exe), all failed with c0000005 (access violation). I saw it recommended somewhere to run with drrun.exe but without winafl, to see if that works, and that also gives an access violation, even when I removed everything from my test exe and left just "return 0":
C:\Code\MyResearch\DynamoRIO-Windows-8.0.18936\bin64\drrun.exe -- C:\Code\Test\TestExe\x64\Release\TestExe.exe

I can see the crash in windows' event log, brought below.
I also tried to check where that offset in ntdll is, and got to LdrGetDllHandleByName - the exception is when it tries to write to its out param.

To Reproduce
C:\Code\MyResearch\DynamoRIO-Windows-8.0.18936\bin64\drrun.exe -- C:\Code\Test\TestExe\x64\Release\TestExe.exe
TestExe.exe is a 64-bit exe compiled in Release with Visual Studio 2019 16.11.5, which contains a _tmain function that only returns 0.

Running TestExe.exe directly doesn't crash.
Running with -debug still crashes, see output below.

Expected behavior
Don't crash.

Screenshots or Pasted Text
In eventvwr (Windows Logs > Application), I can see:

Faulting application name: TestExe.exe, version: 0.0.0.0, time stamp: 0x618a6e15
Faulting module name: ntdll.dll, version: 10.0.19041.1288, time stamp: 0xa280d1d6
Exception code: 0xc0000005
Fault offset: 0x0000000000076ffb
Faulting process id: 0x7764
Faulting application start time: 0x01d7d568252e029e
Faulting application path: C:\Code\Test\TestExe\x64\Release\TestExe.exe
Faulting module path: C:\WINDOWS\SYSTEM32\ntdll.dll
Report Id: 5cd5b69b-1c62-473d-aea0-da04415907ab
Faulting package full name: 
Faulting package-relative application ID:

And also:

Fault bucket 2193024489411161897, type 4
Event Name: APPCRASH
Response: Not available
Cab Id: 0

Problem signature:
P1: TestExe.exe
P2: 0.0.0.0
P3: 618a6e15
P4: ntdll.dll
P5: 10.0.19041.1288
P6: a280d1d6
P7: c0000005
P8: 0000000000076ffb
P9: 
P10: 

Attached files:
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER8848.tmp.dmp
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER8888.tmp.WERInternalMetadata.xml
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER8898.tmp.xml
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER88A6.tmp.csv
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER88D6.tmp.txt

These files may be available here:
\\?\C:\ProgramData\Microsoft\Windows\WER\ReportArchive\AppCrash_TestExe.exe_765ed3d798174e1ba1abff5141a4da78ab5fa1_8fc52057_1b47e7a3-ee3c-4f7b-8675-84bf386b4201

Analysis symbol: 
Rechecking for solution: 0
Report Id: 5cd5b69b-1c62-473d-aea0-da04415907ab
Report Status: 268435456
Hashed bucket: 318aba5cafebc57efe6f30276b15af29
Cab Guid: 0

Output of running with debug:

# C:\Code\MyResearch\DynamoRIO-Windows-8.0.18936\bin64\drrun.exe -debug -- C:\Code\Test\TestExe\x64\Release\TestExe.exe
<Starting application C:\Code\Test\TestExe\x64\Release\TestExe.exe (29100)>
<Running on newer-than-this-build "Microsoft Windows 10-2009 x64">
<Early threads found>
<Initial options = -no_dynamic_options -code_api -probe_api -stack_size 56K -max_elide_jmp 0 -max_elide_call 0 -no_inline_ignored_syscalls -native_exec_default_list '' -no_native_exec_managed_code -no_indcall2direct >
<intercept_syscall_wrapper: not hooking NtCreateUserProcess due to conflict @0x00007ffec8bee655>
<intercept_syscall_wrapper: not hooking NtTerminateProcess due to conflict @0x00007ffec8bed2e5>
<intercept_syscall_wrapper: not hooking NtTerminateThread due to conflict @0x00007ffec8bed7c5>
<intercept_syscall_wrapper: not hooking NtMapViewOfSection due to conflict @0x00007ffec8bed265>
<intercept_syscall_wrapper: not hooking NtOpenFile due to conflict @0x00007ffec8bed3c5>
<Stopping application C:\Code\Test\TestExe\x64\Release\TestExe.exe (29100)>
<CURIOSITY : reached_image_entry_yet() || standalone_library || ( (((void)(((dynamo_options.checklevel >= (1)) && !(!((OPTION_IS_STRING_thin_client)) || (((&options_lock)->num_readers > 0) || self_owns_write_lock(&options_lock)))) ? (d_r_internal_error("D:\\a\\dynamorio\\dynamorio\\core\\win32\\os.c", 1231, "!((OPTION_IS_STRING_thin_client)) || READWRITE_LOCK_HELD(&options_lock)"), 0) : 0)), dynamo_options.thin_client)) || dr_api_entry || (!((dynamo_options.client_lib)[0] == '\0')) in file D:\a\dynamorio\dynamorio\core\win32\os.c line 1231
version 8.0.18936, custom build
-no_dynamic_options -code_api -probe_api -stack_size 56K -max_elide_jmp 0 -max_elide_call 0 -no_inline_ignored_syscalls -native_exec_default_list '' -no_native_exec_managed_code -no_indcall2direct
0x000001634dde4288 0x000001634ddc4800
0x00000000153928ed 0xccccccccccccc300
C:\Code\MyResearch\DynamoRIO-Windows-8.0.18936\lib64\debug\dynamorio.dll=0x0000000015000000>

Versions
Using latest DynamoRIO release (DynamoRIO-Windows-8.0.18936.zip), Windows 10 21H1 (OS build 19043.1320). 64-bit versions for everything.

Additional context
I have SEP, but even when I disabled it - the crash still happened.

The text was updated successfully, but these errors were encountered:

derekbruening · 2021-11-10T01:30:48Z

Failure up front in every app usually points at incompatibilities with security software injecting code into every process.
When you disabled SEP, were the messages like <intercept_syscall_wrapper: not hooking NtCreateUserProcess due to conflict @0x00007ffec8bee655> still there? It is possible that "disabling" from the system tray or whatnot it doesn't really disable its injection into every process. Do you still see SEP libraries inside the process if you launch your TestExe.exe without drrun? Running smc -stop may actually disable it (it did on old versions) as a test to see whether it is the problem.

derekbruening · 2021-11-10T01:31:41Z

Xref DynamoRIO/drmemory#1665 (comment)

eranzim · 2021-11-10T07:58:16Z

Interesting - I disabled SEP from the tray and also ran sfc -stop which made even the tray icon disappear - but I still get those messages, and sysfer.dll (SEP's dll) is still loaded into both drrun.exe and TestExe.exe. Can't find sysfer.dll in Sysinternals' autoruns (running as admin), might be one of its services loading it. Annoying as it is, I might need to reinstall DynamoRIO on a clean VM and try there. Will update.

eranzim · 2021-11-10T13:12:00Z

I installed a new Win 10 Pro x64 VM, clean install, barely installed any software on it, disabled Windows Defender protection - and I'm still getting the same crash. This time the debug output doesn't contain those messages:

# C:\Work\DynamoRIO-Windows-8.0.18936\bin64\drrun.exe -debug -- C:\Work\TestExe.exe
<Starting application C:\Work\TestExe.exe (9604)>
<Running on newer-than-this-build "Microsoft Windows 10-2009 x64">
<Initial options = -no_dynamic_options -code_api -probe_api -stack_size 56K -max_elide_jmp 0 -max_elide_call 0 -no_inline_ignored_syscalls -native_exec_default_list '' -no_native_exec_managed_code -no_indcall2direct >
<Stopping application C:\Work\TestExe.exe (9604)>
<CURIOSITY : reached_image_entry_yet() || standalone_library || ( (((void)(((dynamo_options.checklevel >= (1)) && !(!((OPTION_IS_STRING_thin_client)) || (((&options_lock)->num_readers > 0) || self_owns_write_lock(&options_lock)))) ? (d_r_internal_error("D:\\a\\dynamorio\\dynamorio\\core\\win32\\os.c", 1231, "!((OPTION_IS_STRING_thin_client)) || READWRITE_LOCK_HELD(&options_lock)"), 0) : 0)), dynamo_options.thin_client)) || dr_api_entry || (!((dynamo_options.client_lib)[0] == '\0')) in file D:\a\dynamorio\dynamorio\core\win32\os.c line 1231
version 8.0.18936, custom build
-no_dynamic_options -code_api -probe_api -stack_size 56K -max_elide_jmp 0 -max_elide_call 0 -no_inline_ignored_syscalls -native_exec_default_list '' -no_native_exec_managed_code -no_indcall2direct
0x000002909e3e4288 0x000002909e3c4800
0x00000000153928ed 0xccccccccccccc300
C:\Work\DynamoRIO-Windows-8.0.18936\lib64\debug\dynamorio.dll=0x0000000015000000>

I thought maybe the TestExe project had some non-standard flags or something, so I created a new clean project, added a single Main.c file with this code:

#include <Windows.h>
#include <tchar.h>

INT _tmain(INT nArgc, PCTSTR apszArgv[])
{
	return 0;
}

It crashes just the same, with this similar albeit slightly different debug output:

# C:\Work\DynamoRIO-Windows-8.0.18936\bin64\drrun.exe -debug -- C:\Work\Harness\x64\Release\Harness.exe
<Starting application C:\Work\Harness\x64\Release\Harness.exe (4916)>
<Running on newer-than-this-build "Microsoft Windows 10-2009 x64">
<Early threads found>
<Initial options = -no_dynamic_options -code_api -probe_api -stack_size 56K -max_elide_jmp 0 -max_elide_call 0 -no_inline_ignored_syscalls -native_exec_default_list '' -no_native_exec_managed_code -no_indcall2direct >
<Stopping application C:\Work\Harness\x64\Release\Harness.exe (4916)>
<CURIOSITY : reached_image_entry_yet() || standalone_library || ( (((void)(((dynamo_options.checklevel >= (1)) && !(!((OPTION_IS_STRING_thin_client)) || (((&options_lock)->num_readers > 0) || self_owns_write_lock(&options_lock)))) ? (d_r_internal_error("D:\\a\\dynamorio\\dynamorio\\core\\win32\\os.c", 1231, "!((OPTION_IS_STRING_thin_client)) || READWRITE_LOCK_HELD(&options_lock)"), 0) : 0)), dynamo_options.thin_client)) || dr_api_entry || (!((dynamo_options.client_lib)[0] == '\0')) in file D:\a\dynamorio\dynamorio\core\win32\os.c line 1231
version 8.0.18936, custom build
-no_dynamic_options -code_api -probe_api -stack_size 56K -max_elide_jmp 0 -max_elide_call 0 -no_inline_ignored_syscalls -native_exec_default_list '' -no_native_exec_managed_code -no_indcall2direct
0x000001e6ac6a4288 0x000001e6ac684800
0x00000000153928ed 0xccccccccccccc300
C:\Work\DynamoRIO-Windows-8.0.18936\lib64\debug\dynamorio.dll=0x0000000015000000>

derekbruening · 2021-11-10T17:37:34Z

Hmm.

Does notepad run under drrun?
Does using an older version of DR make a difference?
Does running through an intermediary and passing -early_inject_location 5 work? E.g.: path\to\bin64\drrun.exe -debug -early_inject_location 5 -- path\to\bin64\create_process.exe C:\Work\TestExe.exe
Does passing -no_early_inject_map make a difference?
The Stopping message implies that DR thinks this is an app crash. Could you run with -debug -loglevel 4 and attach all of the log files?

eranzim · 2021-11-10T22:47:02Z

Hmm.

Does notepad run under drrun?

Yes, no crash.

Does using an older version of DR make a difference?

Will check tomorrow. Any specific version to check with?

Does running through an intermediary and passing -early_inject_location 5 work? E.g.: path\to\bin64\drrun.exe -debug -early_inject_location 5 -- path\to\bin64\create_process.exe C:\Work\TestExe.exe

No, it crashes - same crash (dll and offset), only this time the crash is in create_process.exe instead of my exe.

Does passing -no_early_inject_map make a difference?

Same crash.

The Stopping message implies that DR thinks this is an app crash. Could you run with -debug -loglevel 4 and attach all of the log files?

# C:\Work\DynamoRIO-Windows-8.0.18936\bin64\drrun.exe -debug -loglevel 4 -- C:\Work\Harness\x64\Release\Harness.exe
<log dir=C:\Work\DynamoRIO-Windows-8.0.18936\logs\Harness.exe.9100.00000000>
<Starting application C:\Work\Harness\x64\Release\Harness.exe (9100)>
<Running on newer-than-this-build "Microsoft Windows 10-2009 x64">
<Early threads found>
<Initial options = -no_dynamic_options -loglevel 4 -code_api -probe_api -stack_size 56K -max_elide_jmp 0 -max_elide_call 0 -no_inline_ignored_syscalls -native_exec_default_list '' -no_native_exec_managed_code -no_indcall2direct >
<curiosity: rex.w on OPSZ_6_irex10_short4!>
<Stopping application C:\Work\Harness\x64\Release\Harness.exe (9100)>
<CURIOSITY : reached_image_entry_yet() || standalone_library || ( (((void)(((dynamo_options.checklevel >= (1)) && !(!((OPTION_IS_STRING_thin_client)) || (((&options_lock)->num_readers > 0) || self_owns_write_lock(&options_lock)))) ? (d_r_internal_error("D:\\a\\dynamorio\\dynamorio\\core\\win32\\os.c", 1231, "!((OPTION_IS_STRING_thin_client)) || READWRITE_LOCK_HELD(&options_lock)"), 0) : 0)), dynamo_options.thin_client)) || dr_api_entry || (!((dynamo_options.client_lib)[0] == '\0')) in file D:\a\dynamorio\dynamorio\core\win32\os.c line 1231
version 8.0.18936, custom build
-no_dynamic_options -loglevel 4 -code_api -probe_api -stack_size 56K -max_elide_jmp 0 -max_elide_call 0 -no_inline_ignored_syscalls -native_exec_default_list '' -no_native_exec_managed_code -no_indcall2direct
0x00000272bb664288 0x00000272bb644800
0x00000000153928ed 0xccccccccccccc300
C:\Work\DynamoRIO-Windows-8.0.18936\lib64\debug\dynamorio.dll=0x0000000015000000>

Harness.exe.9100.00000000.zip

eranzim · 2021-11-11T11:43:54Z

Tried it with other versions as well (DynamoRIO-Windows-8.0.18929.zip, DynamoRIO-Windows-8.0.18816.zip), same result (same crash).

derekbruening · 2021-11-11T16:40:20Z

From the log file, DR injects and the app code it starts with is this non-library generated code:

interp: start_pc = 0x00000272bb350010
check_thread_vm_area: pc = 0x00000272bb350010
new shared vm area: 0x00000272bb350000-0x00000272bb351000 ---- alloc
checking thread vmareas against executable_areas
prepend_entry_to_fraglist: putting fragment @0x00000272bb350010 (shared) on vmarea 0x00000272bb350000-0x00000272bb351000
check_thread_vm_area: check_stop = 0x00000272bb351000
  0x00000272bb350010  9c                   pushf  %rsp -> %rsp 0xfffffff8(%rsp)[8byte]
        reads flag before writing it!
          reads OF prior to writing it!
  0x00000272bb350011  50                   push   %rax %rsp -> %rsp 0xfffffff8(%rsp)[8byte]
  0x00000272bb350012  51                   push   %rcx %rsp -> %rsp 0xfffffff8(%rsp)[8byte]
  0x00000272bb350013  52                   push   %rdx %rsp -> %rsp 0xfffffff8(%rsp)[8byte]
  0x00000272bb350014  53                   push   %rbx %rsp -> %rsp 0xfffffff8(%rsp)[8byte]
  0x00000272bb350015  55                   push   %rbp %rsp -> %rsp 0xfffffff8(%rsp)[8byte]
  0x00000272bb350016  56                   push   %rsi %rsp -> %rsp 0xfffffff8(%rsp)[8byte]
  0x00000272bb350017  57                   push   %rdi %rsp -> %rsp 0xfffffff8(%rsp)[8byte]
  0x00000272bb350018  41 50                push   %r8 %rsp -> %rsp 0xfffffff8(%rsp)[8byte]
  0x00000272bb35001a  41 51                push   %r9 %rsp -> %rsp 0xfffffff8(%rsp)[8byte]
  0x00000272bb35001c  41 52                push   %r10 %rsp -> %rsp 0xfffffff8(%rsp)[8byte]
  0x00000272bb35001e  41 53                push   %r11 %rsp -> %rsp 0xfffffff8(%rsp)[8byte]
  0x00000272bb350020  41 54                push   %r12 %rsp -> %rsp 0xfffffff8(%rsp)[8byte]
  0x00000272bb350022  41 55                push   %r13 %rsp -> %rsp 0xfffffff8(%rsp)[8byte]
  0x00000272bb350024  41 56                push   %r14 %rsp -> %rsp 0xfffffff8(%rsp)[8byte]
  0x00000272bb350026  41 57                push   %r15 %rsp -> %rsp 0xfffffff8(%rsp)[8byte]
  0x00000272bb350028  48 83 ec 28          sub    $0x0000000000000028 %rsp -> %rsp
  0x00000272bb35002c  4c 8d 05 d5 ff ff ff lea    <rel> 0x00000272bb350008 -> %r8
  0x00000272bb350033  33 d2                xor    %edx %edx -> %edx
  0x00000272bb350035  48 8d 0d cc 00 00 00 lea    <rel> 0x00000272bb350108 -> %rcx
  0x00000272bb35003c  ff 15 c6 ff ff ff    call   <rel> 0x00000272bb350008[8byte] %rsp -> %rsp 0xfffffff8(%rsp)[8byte]
mbr exit target = 0x00007ff66d87f9c0
end_pc = 0x00000272bb350042
Fragment 3, tag 0x00000272bb350010, flags 0x1000030, shared, size 112:
        []

That code calls LdrGetDllHandleByName and sets the output param to point to an address in the same allocation as its code, 0x00000272bb350008:

Fragment 4, tag 0x00007ffa59d86fa0, flags 0x1000630, shared, size 60:
        [ntdll.dll!LdrGetDllHandleByName]

That's what crashes, as you said:

priv_mcontext_t @0x00000272bb644800
        xax = 0x0000000000000000
        xbx = 0x0000000000000000
        xcx = 0x00000272bb350108
        xdx = 0x0000000000000000
        xsi = 0x0000000000000000
        xdi = 0x0000000000000000
        xbp = 0x0000000000000000
        xsp = 0x000000b0c41ff958
        r8  = 0x00000272bb350008

ASYNCH intercepted exception in thread 3972 at pc 0x00007ff66d8848d0
        exception code = 0x00000272c0000005, ExceptionFlags=0x0000000000000000
        record=0x0000000000000000, params=2
        PC 0x00007ff66d8848d0 tried to write address 0x00000272bb350008

Normally, DR's initial app code for default options (-early_inject_location == INJECT_LOCATION_ThreadStart) is RtlUserThreadStart. It gets the target through a few methods but it starts with NtGetContextThread.

Sometimes DR has to hook to take over, though it prefers calling NtSetContextThread. Its hook restore code is always run, though, and that blindly sets the target location to +rx.

So it looks like you have something in these apps that is inserting generated code mixed with data that must be writable and executable, and DR is marking that code non-writable, and it then crashes trying to write.

Looking at the loaded libraries: probably it is apphelp.dll with some compatibility layer? And that's why other people have not hit this problem and it is limited to certain machines and certain apps?

If you run drrun -late -- <yourapp> does that work? I believe that bypasses the mark-rx code in DR causing the problem here.

A proposed fix is to add the prior page prot bits to earliest_args_t and use them in the protect call here: https://github.com/DynamoRIO/dynamorio/blob/master/core/win32/os.c#L9081
The args are written here so the prior prot value has to be made available here:
https://github.com/DynamoRIO/dynamorio/blob/master/core/win32/inject.c#L1212

eranzim · 2021-11-11T18:43:46Z

Running with -late seems to work :)
If you fix this and want me to test the fix, let me know.
Thanks!

Instead of blindly marking the inject hook page as +rx, we remember its actual prior protections and restore those. This avoids crashes in cases where the app's initial code is actually writable. Tested manually by the filer of #5198. It is difficult to create an automated test. Fixes #5198

derekbruening · 2021-11-11T23:32:47Z

Running with -late seems to work :) If you fix this and want me to test the fix, let me know. Thanks!

Could you test with #5203?

eranzim · 2021-11-13T21:04:12Z

Can you provide me with a compiled package, or instruct me how to build it myself?

derekbruening · 2021-11-14T02:47:49Z

Can you provide me with a compiled package, or instruct me how to build it myself?

https://github.com/DynamoRIO/dynamorio/releases/tag/cronbuild-8.0.18942-1

eranzim · 2021-11-14T08:42:56Z

That solved my crash :)
Thanks!

Instead of blindly marking the inject hook page as +rx, we remember its actual prior protections and restore those. This avoids crashes in cases where the app's initial code is actually writable. Tested manually by the filer of #5198. It is difficult to create an automated test. Fixes #5198

derekbruening · 2021-11-14T12:21:36Z

Thank you for testing the fix

Updates DR to 3b4d7485 which fixes DynamoRIO/dynamorio#4958, eliminating basic blocks coming from the DR library. This also pulls in a fix for an injection crash from app compatibility layers: DynamoRIO/dynamorio#5198. Removes the workaround in the umbra_client_insert_app_to_shadow test that is now no longer needed. Issue: DynamoRIO/dynamorio#4958, DynamoRIO/dynamorio#5198

johnfxgalea added the OpSys-Windows label Nov 9, 2021

eranzim mentioned this issue Nov 11, 2021

drrun gets exception c0000005 googleprojectzero/winafl#250

Open

eranzim mentioned this issue Nov 11, 2021

ASSERT FAILURE: C:\Code\MyResearch\winafl\winafl.c:969: (0) (invalid option) googleprojectzero/winafl#340

Closed

derekbruening mentioned this issue Nov 11, 2021

Release DynamoRIO v9.0.0 #5145

Closed

21 tasks

derekbruening mentioned this issue Nov 11, 2021

i#5198: Preserve actual inject hook page protections #5203

Merged

derekbruening self-assigned this Nov 13, 2021

derekbruening changed the title ~~Executable crashes with access violation when run with drrun.exe~~ App running +rwx code at DR injection time crashes Nov 13, 2021

derekbruening closed this as completed in #5203 Nov 14, 2021

derekbruening mentioned this issue Nov 23, 2021

DRi#4958: Update DR to eliminate DR blocks DynamoRIO/drmemory#2439

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

App running +rwx code at DR injection time crashes #5198

App running +rwx code at DR injection time crashes #5198

eranzim commented Nov 9, 2021

derekbruening commented Nov 10, 2021

derekbruening commented Nov 10, 2021

eranzim commented Nov 10, 2021

eranzim commented Nov 10, 2021

derekbruening commented Nov 10, 2021

eranzim commented Nov 10, 2021

eranzim commented Nov 11, 2021

derekbruening commented Nov 11, 2021

eranzim commented Nov 11, 2021

derekbruening commented Nov 11, 2021

eranzim commented Nov 13, 2021

derekbruening commented Nov 14, 2021

eranzim commented Nov 14, 2021

derekbruening commented Nov 14, 2021

App running +rwx code at DR injection time crashes #5198

App running +rwx code at DR injection time crashes #5198

Comments

eranzim commented Nov 9, 2021

derekbruening commented Nov 10, 2021

derekbruening commented Nov 10, 2021

eranzim commented Nov 10, 2021

eranzim commented Nov 10, 2021

derekbruening commented Nov 10, 2021

eranzim commented Nov 10, 2021

eranzim commented Nov 11, 2021

derekbruening commented Nov 11, 2021

eranzim commented Nov 11, 2021

derekbruening commented Nov 11, 2021

eranzim commented Nov 13, 2021

derekbruening commented Nov 14, 2021

eranzim commented Nov 14, 2021

derekbruening commented Nov 14, 2021