Concurrency runtime operator new variants in /MTd lead to many mismatch errors

[derekbruening]

From bruen...@google.com on May 16, 2013 13:41:52

While investigating issue #1233 I tried to make a smaller example that uses the
Concurrency runtime added in VS2010. I took the example code here: http://msdn.microsoft.com/en-us/library/vstudio/ee513829(v=vs.100).aspx building /MTd has a lot of these:

Error #1: INVALID HEAP ARGUMENT: allocated with malloc, freed with operator delete
#0 replace_operator_delete [d:\derek\drmemory\git\src\common\alloc_replace.c:2408]
#1 Concurrency::SchedulerPolicy::~SchedulerPolicy [f:\dd\vctools\crt_bld\self_x86\crt\src\schedulerpolicybase.cpp:155]
#2 Concurrency::details::ResourceManager::CreateSchedulerProxy [f:\dd\vctools\crt_bld\self_x86\crt\src\resourcemanager.cpp:2561]
#3 Concurrency::details::ResourceManager::RegisterScheduler [f:\dd\vctools\crt_bld\self_x86\crt\src\resourcemanager.cpp:1096]
#4 Concurrency::details::SchedulerBase::Initialize [f:\dd\vctools\crt_bld\self_x86\crt\src\schedulerbase.cpp:420]
#5 Concurrency::details::SchedulerBase::GetDefaultScheduler [f:\dd\vctools\crt_bld\self_x86\crt\src\schedulerbase.cpp:530]
#6 Concurrency::details::SchedulerBase::CreateContextFromDefaultScheduler [f:\dd\vctools\crt_bld\self_x86\crt\src\schedulerbase.cpp:444]
#7 Concurrency::details::SchedulerBase::CurrentContext [f:\dd\vctools\crt_bld\self_x86\crt\src\schedulerbase.h:260]
#8 Concurrency::details::_TaskCollection::_TaskCollection [f:\dd\vctools\crt_bld\self_x86\crt\src\taskcollection.cpp:538]
#9 Concurrency::task_group::task_group [c:\program files (x86)\microsoft visual studio 10.0\vc\include\ppl.h:454]
#10 perform_task [d:\derek\dr\test\i1233-concurrency\scheduler\scheduler-instance.cpp:14]
#11 current_scheduler [d:\derek\dr\test\i1233-concurrency\scheduler\scheduler-instance.cpp:27]
#12 wmain [d:\derek\dr\test\i1233-concurrency\scheduler\scheduler-instance.cpp:106]
Note: @0:00:00.929 in thread 2404
Note: memory was allocated here:
Note: # 0 replace_malloc [d:\derek\drmemory\git\src\common\alloc_replace.c:2233]
Note: # 1 operator new [f:\dd\vctools\crt_bld\self_x86\crt\src\dbgnew.cpp:55]
Note: # 2 operator new [f:\dd\vctools\crt_bld\self_x86\crt\src\concrtinternal.h:67]
Note: # 3 Concurrency::SchedulerPolicy::SchedulerPolicy [f:\dd\vctools\crt_bld\self_x86\crt\src\schedulerpolicybase.cpp:126]
Note: # 4 Concurrency::details::SchedulerBase::GetPolicy [f:\dd\vctools\crt_bld\self_x86\crt\src\schedulerbase.cpp:944]
Note: # 5 Concurrency::details::SchedulerResourceManagement::GetPolicy [f:\dd\vctools\crt_bld\self_x86\crt\src\schedulerresourcemanagement.cpp:67]
Note: # 6 Concurrency::details::ResourceManager::CreateSchedulerProxy [f:\dd\vctools\crt_bld\self_x86\crt\src\resourcemanager.cpp:2554]
Note: # 7 Concurrency::details::ResourceManager::RegisterScheduler [f:\dd\vctools\crt_bld\self_x86\crt\src\resourcemanager.cpp:1096]
Note: # 8 Concurrency::details::SchedulerBase::Initialize [f:\dd\vctools\crt_bld\self_x86\crt\src\schedulerbase.cpp:420]
Note: # 9 Concurrency::details::SchedulerBase::GetDefaultScheduler [f:\dd\vctools\crt_bld\self_x86\crt\src\schedulerbase.cpp:530]
Note: #10 Concurrency::details::SchedulerBase::CreateContextFromDefaultScheduler [f:\dd\vctools\crt_bld\self_x86\crt\src\schedulerbase.cpp:444]
Note: #11 Concurrency::details::SchedulerBase::CurrentContext [f:\dd\vctools\crt_bld\self_x86\crt\src\schedulerbase.h:260]
Note: #12 Concurrency::details::_TaskCollection::_TaskCollection [f:\dd\vctools\crt_bld\self_x86\crt\src\taskcollection.cpp:538]
Note: #13 Concurrency::task_group::task_group [c:\program files (x86)\microsoft visual studio 10.0\vc\include\ppl.h:454]
Note: #14 perform_task [d:\derek\dr\test\i1233-concurrency\scheduler\scheduler-instance.cpp:14]
Note: #15 current_scheduler [d:\derek\dr\test\i1233-concurrency\scheduler\scheduler-instance.cpp:27]
Note: #16 wmain [d:\derek\dr\test\i1233-concurrency\scheduler\scheduler-instance.cpp:106]

0:000> x scheduler_instance!operator new*
0012fb30 scheduler_instance!operator new (unsigned int, HeapManager, int)
000cfc90 scheduler_instance!operator new (unsigned int, ConcRTNewMoniker, char, int)
000eaf50 scheduler_instance!operator new (unsigned int, int, char_, int)
000e0560 scheduler_instance!operator new (unsigned int)
000ce1d0 scheduler_instance!operator new (unsigned int, std::DebugHeapTag_t, char_, int)
000c81a0 scheduler_instance!operator new (unsigned int, void_)
000eafd0 scheduler_instance!operator new[](unsigned int, int, char*, int)
00143a60 scheduler_instance!operator new[](unsigned int)
000ce280 scheduler_instance!operator new[](unsigned int, std::_DebugHeapTag_t*, char*, int)

operator new in scheduler-instance.exe @0x0033fc90 generic type=10 => drsyms res=0, 4 args
WARNING: unknown 3+-arg overload of operator new in scheduler-instance.exe @0x0033fc90
operator new is not straight-line so type is unknown
WARNING: unable to determine type of operator new in scheduler-instance.exe @0x0033fc90

/MDd works b/c we feel comfortable assuming it's non-placement there.

0:000> U 000cfc90 L20
scheduler_instance!operator new [f:\dd\vctools\crt_bld\self_x86\crt\src\concrtinternal.h @ 66]:
000cfc90 8bff mov edi,edi
000cfc92 55 push ebp
000cfc93 8bec mov ebp,esp
000cfc95 8b4514 mov eax,[ebp+0x14]
000cfc98 50 push eax
000cfc99 8b4d10 mov ecx,[ebp+0x10]
000cfc9c 51 push ecx
000cfc9d 6a02 push 0x2
000cfc9f 8b5508 mov edx,[ebp+0x8]
000cfca2 52 push edx
000cfca3 e8a8b20100 call scheduler_instance!operator new (000eaf50)
000cfca8 83c410 add esp,0x10
000cfcab 5d pop ebp
000cfcac c3 ret

so issue #1006 is wrong: there is a downside to simply not intercepting a few
operator news. we'll report a mismatch if we are intercepting the
corresponding operator deletes! this is a big negative for replacing as
opposed to wrapping: distinguishing placement new.

Original issue: http://code.google.com/p/drmemory/issues/detail?id=1239

[derekbruening]

From bruen...@google.com on May 16, 2013 21:13:30

**** TODO identify all the msvc operators?

solution: we must have syms to hit this (else won't intercept any of these
operators), so can we use the syms to identify these as being libc? we can
then ensure we intercept all the libc operators.

WARNING: unknown 3+-arg overload of operator new in scheduler-instance.exe @0x0131e1d0
WARNING: unknown 3+-arg overload of operator new[] in scheduler-instance.exe @0x0131e280
WARNING: unknown 3+-arg overload of operator new in scheduler-instance.exe @0x0133af50
WARNING: unknown 3+-arg overload of operator new[] in scheduler-instance.exe @0x0133afd0
WARNING: unknown 3+-arg overload of operator new in scheduler-instance.exe @0x0131fc90
WARNING: unknown 3+-arg overload of operator new in scheduler-instance.exe @0x0137fb30
WARNING: unknown 3+-arg overload of operator delete in scheduler-instance.exe @0x0131e2a0
WARNING: unknown 3+-arg overload of operator delete[] in scheduler-instance.exe @0x0131e2c0
WARNING: unknown 3+-arg overload of operator delete in scheduler-instance.exe @0x013211e0
=>
std::_DebugHeapTag_t, int,char*,int, _ConcRTNewMoniker, _HeapManager

**** TODO still have mismatches, but different, once replacing all

Error #1: INVALID HEAP ARGUMENT: allocated with operator new, freed with operator delete[]

0 replace_operator_delete_array [d:\derek\drmemory\git\src\common\alloc_replace.c:2428](0x73975400 <drmemoryl

ib.dll+0x175400) modid:3

1 Concurrency::details::SchedulerProxy::GrantAllocation [f:\dd\vctools\crt_bld\self_x86\crt\src\schedulerproxy

.cpp:652](0x003b39b4 <scheduler-instance.exe+0x639b4) modid:1

2 Concurrency::details::ResourceManager::PerformAllocation [f:\dd\vctools\crt_bld\self_x86\crt\src\resourceman

ager.cpp:1909](0x0038963d <scheduler-instance.exe+0x3963d) modid:1

3 Concurrency::details::ResourceManager::RequestInitialVirtualProcessors [f:\dd\vctools\crt_bld\self_x86\crt\s

rc\resourcemanager.cpp:1147](0x0038790e <scheduler-instance.exe+0x3790e) modid:1

4 Concurrency::details::SchedulerProxy::RequestInitialVirtualProcessors [f:\dd\vctools\crt_bld\self_x86\crt\sr

c\schedulerproxy.cpp:156](0x003b293d <scheduler-instance.exe+0x6293d) modid:1

5 Concurrency::details::SchedulerBase::Initialize [f:\dd\vctools\crt_bld\self_x86\crt\src\schedulerbase.cpp:42

1](0x00364ffa <scheduler-instance.exe+0x14ffa) modid:1

6 Concurrency::details::SchedulerBase::GetDefaultScheduler [f:\dd\vctools\crt_bld\self_x86\crt\src\schedulerba

se.cpp:530](0x0036535f <scheduler-instance.exe+0x1535f) modid:1

7 Concurrency::details::SchedulerBase::CreateContextFromDefaultScheduler [f:\dd\vctools\crt_bld\self_x86\crt\s

rc\schedulerbase.cpp:444](0x0036514d <scheduler-instance.exe+0x1514d) modid:1

8 Concurrency::details::SchedulerBase::CurrentContext [f:\dd\vctools\crt_bld\self_x86\crt\src\schedulerbase.h:

260](0x0035fdd7 <scheduler-instance.exe+0xfdd7) modid:1

9 Concurrency::details::_TaskCollection::_TaskCollection [f:\dd\vctools\crt_bld\self_x86\crt\src\taskcollectio

n.cpp:538](0x003609eb <scheduler-instance.exe+0x109eb) modid:1
#10 Concurrency::task_group::task_group [c:\program files (x86)\microsoft visual studio 10.0\vc\include\ppl.h:45
4](0x00351e3f <scheduler-instance.exe+0x1e3f) modid:1
#11 perform_task [d:\derek\dr\test\i1233-concurrency\scheduler\scheduler-instance.cpp:14](0x00351a66
<scheduler-instance.exe+0x1a66) modid:1
#12 current_scheduler [d:\derek\dr\test\i1233-concurrency\scheduler\scheduler-instance.cpp:27](0x00351aea
<scheduler-instance.exe+0x1aea) modid:1
#13 wmain [d:\derek\dr\test\i1233-concurrency\scheduler\scheduler-instance.cpp:106](0x00351d12
<scheduler-instance.exe+0x1d12) modid:1
Note: @0:00:01.495 in thread 8868
Note: memory was allocated here:
Note: # 0 replace_operator_new [d:\derek\drmemory\git\src\common\alloc_replace.c:2360](0x73974370 <drmemorylib.
dll+0x174370) modid:3
Note: # 1 Concurrency::details::SchedulerProxy::GrantAllocation [f:\dd\vctools\crt_bld\self_x86\crt\src\schedule
rproxy.cpp:573](0x003b3736 <scheduler-instance.exe+0x63736) modid:1
Note: # 2 Concurrency::details::ResourceManager::PerformAllocation [f:\dd\vctools\crt_bld\self_x86\crt\src\resou
rcemanager.cpp:1909](0x0038963d <scheduler-instance.exe+0x3963d) modid:1
Note: # 3 Concurrency::details::ResourceManager::RequestInitialVirtualProcessors [f:\dd\vctools\crt_bld\self_x86
\crt\src\resourcemanager.cpp:1147](0x0038790e <scheduler-instance.exe+0x3790e) modid:1
Note: # 4 Concurrency::details::SchedulerProxy::RequestInitialVirtualProcessors [f:\dd\vctools\crt_bld\self_x86
crt\src\schedulerproxy.cpp:156](0x003b293d <scheduler-instance.exe+0x6293d) modid:1
Note: # 5 Concurrency::details::SchedulerBase::Initialize [f:\dd\vctools\crt_bld\self_x86\crt\src\schedulerbase.cpp:421](0x00364ffa <scheduler-instance.exe+0x14ffa) modid:1
Note: # 6 Concurrency::details::SchedulerBase::GetDefaultScheduler [f:\dd\vctools\crt_bld\self_x86\crt\src\schedulerbase.cpp:530](0x0036535f <scheduler-instance.exe+0x1535f) modid:1
Note: # 7 Concurrency::details::SchedulerBase::CreateContextFromDefaultScheduler [f:\dd\vctools\crt_bld\self_x86\crt\src\schedulerbase.cpp:444](0x0036514d <scheduler-instance.exe+0x1514d) modid:1
Note: # 8 Concurrency::details::SchedulerBase::CurrentContext [f:\dd\vctools\crt_bld\self_x86\crt\src\schedulerbase.h:260](0x0035fdd7 <scheduler-instance.exe+0xfdd7) modid:1
Note: # 9 Concurrency::details::_TaskCollection::_TaskCollection [f:\dd\vctools\crt_bld\self_x86\crt\src\taskcollection.cpp:538](0x003609eb <scheduler-instance.exe+0x109eb) modid:1
Note: #10 Concurrency::task_group::task_group [c:\program files (x86)\microsoft visual studio 10.0\vc\include\ppl.h:454](0x00351e3f <scheduler-instance.exe+0x1e3f) modid:1
Note: #11 perform_task [d:\derek\dr\test\i1233-concurrency\scheduler\scheduler-instance.cpp:14](0x00351a66 <scheduler-instance.exe+0x1a66) modid:1

code looks good:
IVirtualProcessorRoot** vprocArray = _concrt_new IVirtualProcessorRoot *[vprocCount];
delete [] vprocArray;

but that new call is:
0:001> U 0x003b3736-5
scheduler_instance!Concurrency::details::SchedulerProxy::GrantAllocation+0x241 [f:\dd\vctools\crt_bld\self_x86\crt\src\schedulerproxy.cpp @ 573]:
003b3731 e85ac5faff call scheduler_instance!operator new (0035fc90)
0:001> x scheduler_instance!operator new*
0035fc90 scheduler_instance!operator new (unsigned int, _ConcRTNewMoniker, char*, int)

it seems that the _ConcRTNewMoniker operators don't have [] and thus we
must disable mismatches for them (but only them: so a new concept to
disable individually and not for the whole set).

**** TODO what about non-libc?

but what about the bigger picture: the app defines a bunch of operators and
we only replace a few? disable mismatch detection if we can't replace them
all? that would disable detection on chrome.dll, or several chrome dlls for component build

[derekbruening]

From derek.br...@gmail.com on May 17, 2013 07:46:13

This issue was closed by revision r1383 .

Status: Fixed