Add DetourFinishHelperProcess export

[DrusTheAxe]

Add DetourFinishHelperProcess export to help with DetourIsHelperProcess() and avoid stack overflow.

Per https://github.com/microsoft/Detours/wiki/Using-Detours

...If the DLL may be used in mixed 32-bit and 64-bit environments, then the DllMain function must call the DetourIsHelperProcess API. The DLL must export the DetourFinishHelperProcess API as export Ordinal 1, which will be called by rundll32.exe to perform the helper tasks.

DetourFinishHelperProcess implies this is only relevant when cross bit-ness across process creation

When creating a 32-bit target process from a 64-bit parent process or creating a 64-bit target process from a 32-bit parent process, the DetourCreateProcessWithDllEx API must create a temporary helper process. . It loads a copy of the user-supplied DLL into the helper process using the rundll32.exe mechanism. Rundll32.exe will call DLL's Ordinal 1 export function. The DetourFinishHelperProcess API must be set as the DLL's Ordinal 1 export function.

Sample code and most related references don't make this clear but the samples do implement this albeit obscurely buried in build logic e.g. https://github.com/microsoft/Detours/blob/main/samples/simple/Makefile line 36+ (notice the /export:DetourFinishHelperProcess,@1,NONAME)

$(BIND)\simple$(DETOURS_BITS).dll $(BIND)\simple$(DETOURS_BITS).lib: \
        $(OBJD)\simple.obj $(OBJD)\simple.res $(DEPS)
    cl /LD $(CFLAGS) /Fe$(@R).dll /Fd$(@R).pdb \
        $(OBJD)\simple.obj $(OBJD)\simple.res \
        /link $(LINKFLAGS) /subsystem:console \
        /export:DetourFinishHelperProcess,@1,NONAME \
        /export:TimedSleepEx \
        $(LIBS)

We don't use DetourCreateProcessWithDllEx but have seen a seemingly similar issue SVCHOSTGROUP_*_STACK_OVERFLOW_c0000005_Microsoft.WindowsAppRuntime.dll!DetourEnumerateModules where DllMain() in Microsoft.WindowsAppRuntime.dll called DetourIsHelperProcess() called DetourFindPayloadEx() called DetourEnumerateModules() and boom.

Unclear why this would happen despite reviewing Detours' implementation in detail but

A similar reported issue is a crash when combining

• unpackaged process
• Microsoft.WindowsAppRuntime.dll
• OS Dynamic Dependencies

Unclear why that combination causes grief but it does, and there's enough variables in the mix this export may be the fix. At a minimum it's's a bug to call DetourIsHelperProcess() without the export so worst case we're less wrong, best case it cures the strange crash. Fingers crossed.

https://task.ms/53686166

[DrusTheAxe]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).