Skip to content

Latest commit

 

History

History
113 lines (94 loc) · 6.89 KB

README.md

File metadata and controls

113 lines (94 loc) · 6.89 KB
Raw
English (en-US) 简体中文 (zh-CN)

Avoid Deadlocking on The Heap When Updating Threads

Why does Detours may deadlock when updating threads?

The original Detours uses the CRT heap (via new/delete), and if another thread that also uses this heap and is holding the heap lock is suspended while updating threads, then Detours will deadlock when it accesses the heap.

Raymond Chen discussed the problem that CRT heap deadlocks when suspending threads in blog "The Old New Thing" article "Are there alternatives to _lock and _unlock in Visual Studio 2015?" is the same scenario, and also mentions Detours, here quotes the original text and will not go into details:

Furthermore, you would be best served to take the heap lock (Heap­Lock) before suspending the thread, because the Detours library will allocate memory during thread suspension.

Demo of Detours Deadlock

SlimDetours provides Demo: DeadLock to demonstrate the occurrence of a deadlock in Detours and the resolution in SlimDetours.

One of the threads (HeapUserThread) keeps calling malloc/free (equivalent to new/delete):

 while (!g_bStop)
 {
     p = malloc(4);
     if (p != NULL)
     {
         free(p);
     }
 }

Another thread (SetHookThread) uses Detours or SlimDetours constantly hook and unhook:

while (!g_bStop)
{
    hr = HookTransactionBegin(g_eEngineType);
    if (FAILED(hr))
    {
        break;
    }
    if (g_eEngineType == EngineMicrosoftDetours)
    {
        hr = HRESULT_FROM_WIN32(DetourUpdateThread((HANDLE)lpThreadParameter));
        if (FAILED(hr))
        {
            break;
        }
    }
    hr = HookAttach(g_eEngineType, EnableHook, (PVOID*)&g_pfnEqualRect, Hooked_EqualRect);
    if (FAILED(hr))
    {
        HookTransactionAbort(g_eEngineType);
        break;
    }
    hr = HookTransactionCommit(g_eEngineType);
    if (FAILED(hr))
    {
        break;
    }

    EnableHook = !EnableHook;
}

Note

SlimDetours updates threads automatically (see 🔗 TechWiki: Update Threads Automatically When Applying Inline Hooks), so there is no such function as DetourUpdateThread.

Execute these 2 threads at the same time for 10 seconds, then send a stop signal (g_bStop = TRUE;) and wait 10 seconds again, if it times out, there is a high probability that a deadlock is occurred, a breakpoint will be triggered, and you can observe the call stack of these 2 threads in the debugger for confirmation. For example, if you specify to run this demo with Detours "Demo.exe -Run DeadLock -Engine=MSDetours", the following call stack will see a heap deadlock:

Worker Thread	Demo.exe!HeapUserThread	Demo.exe!heap_alloc_dbg_internal
    [External Code]
    Demo.exe!heap_alloc_dbg_internal(const unsigned __int64 size, const int block_use, const char * const file_name, const int line_number) Line 359
    Demo.exe!heap_alloc_dbg(const unsigned __int64 size, const int block_use, const char * const file_name, const int line_number) Line 450
    Demo.exe!_malloc_dbg(unsigned __int64 size, int block_use, const char * file_name, int line_number) Line 496
    Demo.exe!malloc(unsigned __int64 size) Line 27
    Demo.exe!HeapUserThread(void * lpThreadParameter) Line 29
    [External Code]

Worker Thread	Demo.exe!SetHookThread	Demo.exe!__acrt_lock
    [External Code]
    Demo.exe!__acrt_lock(__acrt_lock_id _Lock) Line 55
    Demo.exe!heap_alloc_dbg_internal(const unsigned __int64 size, const int block_use, const char * const file_name, const int line_number) Line 309
    Demo.exe!heap_alloc_dbg(const unsigned __int64 size, const int block_use, const char * const file_name, const int line_number) Line 450
    Demo.exe!_malloc_dbg(unsigned __int64 size, int block_use, const char * file_name, int line_number) Line 496
    Demo.exe!malloc(unsigned __int64 size) Line 27
    [External Code]
    Demo.exe!DetourDetach(void * * ppPointer, void * pDetour) Line 2392
    Demo.exe!HookAttach(_DEMO_ENGINE_TYPE EngineType, int Enable, void * * ppPointer, void * pDetour) Line 140
    Demo.exe!SetHookThread(void * lpThreadParameter) Line 65
    [External Code]

Use SlimDetours run this demo "Demo.exe -Run DeadLock -Engine=SlimDetours" will pass successfully.

How did other hooking libraries avoid this problem?

mhook uses Virtual­Alloc to allocate memory pages instead of Heap­Alloc to allocate heap memory, which is a solution mentioned at the end of the above article.

Both of MinHook and SlimDetours created a new private heap for internal use, which avoids this problem and saves memory usage:

_detour_memory_heap = RtlCreateHeap(HEAP_NO_SERIALIZE | HEAP_GROWABLE, NULL, 0, 0, NULL, NULL);

Note

Detours already has a transaction mechanism, and SlimDetours' new feature "Delay Hook" also uses SRW locks, so this heap does not need serialized access.

MinHook creates in its initialization function MH_Initialize, and SlimDetours creates in one-time initialization in the first called memory allocation function, so there is no and no need for a separate initialization function.


This work is licensed under Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0).

Ratin <ratin@knsoft.org>
China national certified senior system architect
ReactOS contributor