odata webapi core: sehexception when using $apply=aggregate #10426

mysticdotnet · 2018-05-31T08:25:56Z

When i use ef core as orm i have a sehexception

Exception message: sehexception
Steps to reproduce
Steps are described in this issue: 1413

Further technical details
EF Core version: (2.1.0)
Database Provider: (Microsoft.EntityFrameworkCore.SqlServer)
Operating system: Windows 10
IDE: (Visual Studio 2017 15.7.2)

jkotas · 2018-06-02T17:52:09Z

Repro steps:

Clone https://github.com/OData/WebApi
cd WebApi\samples\AspNetCoreODataSample.Web
dotnet run
Attach windbg
Navigate to localhost:5000/efcore/Movies?$apply=groupby((ReleaseDate))

Result: Dirty AV or invalid instruction

ntdll!KiUserExceptionDispatch+0x2e [minkernel\ntos\rtl\amd64\trampoln.asm @ 745] 
System_Private_CoreLib+0xa3042c
0x4005f40
Microsoft_AspNetCore_OData!Microsoft.AspNet.OData.Query.Expressions.GroupByWrapper.GetHashCode()+0x105 [C:\repro\WebApi\src\Microsoft.AspNet.OData.Shared\Query\Expressions\DynamicTypeWrapper.cs @ 80] 
System_Private_CoreLib!System.Collections.Generic.ObjectEqualityComparer`1[[System.__Canon, System.Private.CoreLib]].GetHashCode(System.__Canon)+0x1e
System_Linq!System.Linq.Lookup`2[[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib]].InternalGetHashCode(System.__Canon)+0x31

Repros on both .NET Core 2.0 and .NET Core 2.1.

jkotas · 2018-06-02T18:30:40Z

Running this on checked runtime is hitting asserts in the JIT:

Assert failure(PID 14036 [0x000036d4], Thread: 17860 [0x45c4]): Assertion failed '((regMask & emitThisGCrefRegs) && (ins == INS_add)) || ((regMask & emitThisByrefRegs) && (ins == INS_add || ins == INS_sub))' in 'Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpParser`1[FrameAdapter][Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.FrameAdapter]:ParseHeaders(struct,struct,byref,byref,byref):bool:this' (IL size 499)

I have not connected this assert to the root cause of the crash. All signs so far are pointing to that this is a code bug.

cc @dotnet/jit-contrib

mysticdotnet · 2018-06-02T18:33:09Z

@jkotas thank you very much for your help

BruceForstall · 2018-07-09T23:36:42Z

I had to navigate to http://localhost:5913/efcore/Movies?$apply=groupby((ReleaseDate))

BruceForstall · 2018-07-10T22:46:17Z

I reproed this with a daily build of dotnet CLI + Debug dotnet/coreclr components (built from the same commit hash as the daily build). It always seems to repro in Microsoft_AspNetCore_OData!Microsoft.AspNet.OData.Query.Expressions.GroupByWrapper.GetHashCode() calling v.GetHashCode() which appears to be garbage:

0x00000003`00000011
Microsoft_AspNetCore_OData!Microsoft.AspNet.OData.Query.Expressions.GroupByWrapper.GetHashCode()+0x105
System_Private_CoreLib!System.Collections.Generic.ObjectEqualityComparer`1[[System.__Canon, System.Private.CoreLib]].GetHashCode(System.__Canon)+0x43
System_Linq!System.Linq.Lookup`2[[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib]].InternalGetHashCode(System.__Canon)+0x31
System_Linq!System.Linq.Lookup`2[[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib]].GetGrouping(System.__Canon, Boolean)+0x28
System_Linq!System.Linq.Lookup`2[[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib]].Create[[System.__Canon, System.Private.CoreLib]](System.Collections.Generic.IEnumerable`1<System.__Canon>, System.Func`2<System.__Canon,System.__Canon>, System.Func`2<System.__Canon,System.__Canon>, System.Collections.Generic.IEqualityComparer`1<System.__Canon>)+0xad
System_Linq!System.Linq.GroupedEnumerable`3[[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib]].GetEnumerator()+0x34
System_Linq!System.Linq.Enumerable+SelectEnumerableIterator`2[[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib]].MoveNext()+0x38
Microsoft_EntityFrameworkCore!Microsoft.EntityFrameworkCore.Query.Internal.LinqOperatorProvider+ExceptionInterceptor`1+EnumeratorExceptionInterceptor[[System.__Canon, System.Private.CoreLib]].MoveNext()+0x52
...

The code here to call v.GetHashCode() (in JitDump: CALLV ind int System.Object.GetHashCode) is:

00007fff`5c0f42a3 488b4dc8        mov     rcx,qword ptr [rbp-38h]
00007fff`5c0f42a7 488b45c8        mov     rax,qword ptr [rbp-38h]
00007fff`5c0f42ab 488b00          mov     rax,qword ptr [rax]
00007fff`5c0f42ae 488b4048        mov     rax,qword ptr [rax+48h]
00007fff`5c0f42b2 ff5010          call    qword ptr [rax+10h]

That doesn't look unusual, but the last indirection give garbage:

0:026> dq @rbp-38 l1
00000092`adffb7c8  00000092`adffb8d0
0:026> dq 92`adffb8d0 l1
00000092`adffb8d0  00000215`bd49d348
0:026> dq 215`bd49d348+48 l1
00000215`bd49d390  00000215`bd49d0e8
0:026> dq 215`bd49d0e8+10 l1
00000215`bd49d0f8  00000003`00000011

(another run; different data)
Here, we're calling 00000003`00000007:

0:018> dq @rbp-38
0000000d`a7e3b978  0000000d`a7e3ba80 000001da`2a2715a0
0000000d`a7e3b988  0000000d`a7e3ba80 00000001`00000001
0000000d`a7e3b998  00000000`6f7c16ce 000001da`2a271530
0000000d`a7e3b9a8  000001da`2a2710b0 0000000d`a7e3b9f0
0000000d`a7e3b9b8  00007fff`b582dfa3 000001da`2a271530
0000000d`a7e3b9c8  000001da`2a271530 00000000`00000f54
0000000d`a7e3b9d8  00000000`00000f54 00000000`00000000
0000000d`a7e3b9e8  000001da`2a18f858 0000000d`a7e3bb30
0:018> dq d`a7e3ba80
0000000d`a7e3ba80  000001da`2a270e88 0000000d`a7e3bb30
0000000d`a7e3ba90  000001da`2a270d68 00007fff`5ca49928
0000000d`a7e3baa0  00000000`00000000 000001da`2a18f858
0000000d`a7e3bab0  000001da`2a2710b0 00007fff`ef426dcd
0000000d`a7e3bac0  000001da`2a270c28 000001db`35c11a30
0000000d`a7e3bad0  00000000`00000f54 00000000`00000f54
0000000d`a7e3bae0  0000000d`a7e3bac0 00000000`00000000
0000000d`a7e3baf0  000001da`2a270c28 00007fff`5ca49928
0:018> !do 1da`2a270e88
Name:        System.Func`2[[AspNetCoreODataSample.Web.Models.Movie, AspNetCoreODataSample.Web],[AspNetCoreODataSample.Web.Models.Movie, AspNetCoreODataSample.Web]]
MethodTable: 00007fff5ca42190
EEClass:     00007fffb46a6cb0
Size:        64(0x40) bytes
File:        C:\Program Files\dotnet\shared\Microsoft.NETCore.App\3.0.0-preview1-26710-03\System.Private.CoreLib.dll
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
00007fffb5ab1a68  40001ee        8        System.Object  0 instance 0000000000000000 _target
00007fffb5ab1a68  40001ef       10        System.Object  0 instance 000001da2a268458 _methodBase
00007fffb5ab2038  40001f0       18        System.IntPtr  1 instance     7fff57bf67d0 _methodPtr
00007fffb5ab2038  40001f1       20        System.IntPtr  1 instance                0 _methodPtrAux
00007fffb5ab1a68  40001fb       28        System.Object  0 instance 0000000000000000 _invocationList
00007fffb5ab2038  40001fc       30        System.IntPtr  1 instance                0 _invocationCount
0:018> dq 1da`2a270e88
000001da`2a270e88  00007fff`5ca42190 00000000`00000000
000001da`2a270e98  000001da`2a268458 00007fff`57bf67d0
000001da`2a270ea8  00000000`00000000 00000000`00000000
000001da`2a270eb8  00000000`00000000 00000000`00000000
000001da`2a270ec8  00007fff`5ca45908 000001da`2a270c28
000001da`2a270ed8  000001da`2a270d68 000001da`2a270e88
000001da`2a270ee8  00000000`00000000 00010000`00000000
000001da`2a270ef8  00007fff`b5ae0120 00000000`00000000
0:018> dq 1da`2a270e88+48 l1
000001da`2a270ed0  000001da`2a270c28
0:018> !do 1da`2a270c28
Name:        System.Linq.Enumerable+SelectManySingleSelectorIterator`2[[Microsoft.EntityFrameworkCore.Storage.Internal.InMemoryTableSnapshot, Microsoft.EntityFrameworkCore.InMemory],[AspNetCoreODataSample.Web.Models.Movie, AspNetCoreODataSample.Web]]
MethodTable: 00007fff5ca48148
EEClass:     00007fff576ce4c8
Size:        64(0x40) bytes
File:        C:\Program Files\dotnet\shared\Microsoft.NETCore.App\3.0.0-preview1-26710-03\System.Linq.dll
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
00007fffb5ab3908  400001a       10         System.Int32  1 instance                7 _threadId
00007fffb5ab3908  400001b       14         System.Int32  1 instance                3 _state
00007fffb5a73ac8  400001c        8       System.__Canon  0 instance 000001da2a18f858 _current
00007fffb46379e0  400003b       18 ...Private.CoreLib]]  0 instance 000001da2a2708c8 _source
0000000000000000  400003c       20                       0 instance 000001da2a270888 _selector
00007fffb5a80db8  400003d       28 ...Private.CoreLib]]  0 instance 000001da2a271210 _sourceEnumerator
00007fffb5a80db8  400003e       30 ...Private.CoreLib]]  0 instance 000001da2a2712a0 _subEnumerator
0:018> dq 1da`2a270c28
000001da`2a270c28  00007fff`5ca48148 000001da`2a18f858
000001da`2a270c38  00000003`00000007 000001da`2a2708c8
000001da`2a270c48  000001da`2a270888 000001da`2a271210
000001da`2a270c58  000001da`2a2712a0 00010000`00000000
000001da`2a270c68  00007fff`b5ae0120 00000000`00000000
000001da`2a270c78  00000000`00000000 000001da`2a267438
000001da`2a270c88  000001da`2a270cb0 000001da`2a270cf0
000001da`2a270c98  000001da`2a267738 00000000`00000007

Maybe there's missing indirections, via the MethodTable for this object?

0:018> !DumpMT -MD 00007fff5ca48148
EEClass:         00007fff576ce4c8
Module:          00007fff572cd438
Name:            System.Linq.Enumerable+SelectManySingleSelectorIterator`2[[Microsoft.EntityFrameworkCore.Storage.Internal.InMemoryTableSnapshot, Microsoft.EntityFrameworkCore.InMemory],[AspNetCoreODataSample.Web.Models.Movie, AspNetCoreODataSample.Web]]
mdToken:         0000000002000019
File:            C:\Program Files\dotnet\shared\Microsoft.NETCore.App\3.0.0-preview1-26710-03\System.Linq.dll
BaseSize:        0x40
ComponentSize:   0x0
Slots in VTable: 18
Number of IFaces in IFaceMap: 6
--------------------------------------
MethodDesc Table
           Entry       MethodDesc    JIT Name
00007fffb5288080 00007fffb48772d8 PreJIT System.Object.ToString()
00007fffb52880f0 00007fffb4877308 PreJIT System.Object.Equals(System.Object)
00007fffb5288240 00007fffb48773a8 PreJIT System.Object.GetHashCode()
00007fffb5288280 00007fffb4877418 PreJIT System.Object.Finalize()
00007fffef419da0 00007fff57345d70 PreJIT System.Linq.Enumerable+Iterator`1[[System.__Canon, System.Private.CoreLib]].get_Current()
00007fff572367f0 00007fff576db568   NONE System.Linq.Enumerable+SelectManySingleSelectorIterator`2[[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib]].Clone()
00007fffef41e710 00007fff576db598 PreJIT System.Linq.Enumerable+SelectManySingleSelectorIterator`2[[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib]].Dispose()
00007fffef419dd0 00007fff57345e00 PreJIT System.Linq.Enumerable+Iterator`1[[System.__Canon, System.Private.CoreLib]].GetEnumerator()
00007fffef41e8a0 00007fff576db5f8 PreJIT System.Linq.Enumerable+SelectManySingleSelectorIterator`2[[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib]].MoveNext()
00007fff57215f50 00007fff57345e60   NONE System.Linq.Enumerable+Iterator`1[[System.__Canon, System.Private.CoreLib]].Select(System.Func`2)
00007fffef419e80 00007fff57345ea8 PreJIT System.Linq.Enumerable+Iterator`1[[System.__Canon, System.Private.CoreLib]].Where(System.Func`2)
00007fffef419ee0 00007fff57345ed8 PreJIT System.Linq.Enumerable+Iterator`1[[System.__Canon, System.Private.CoreLib]].System.Collections.IEnumerator.get_Current()
00007fffef419ef0 00007fff57345f18 PreJIT System.Linq.Enumerable+Iterator`1[[System.__Canon, System.Private.CoreLib]].System.Collections.IEnumerable.GetEnumerator()
00007fff57215f70 00007fff57345f58   NONE System.Linq.Enumerable+Iterator`1[[System.__Canon, System.Private.CoreLib]].System.Collections.IEnumerator.Reset()
00007fffef41e770 00007fff576db5c8 PreJIT System.Linq.Enumerable+SelectManySingleSelectorIterator`2[[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib]].GetCount(Boolean)
00007fffef41ea00 00007fff576db628 PreJIT System.Linq.Enumerable+SelectManySingleSelectorIterator`2[[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib]].ToArray()
00007fffef41ec50 00007fff576db658 PreJIT System.Linq.Enumerable+SelectManySingleSelectorIterator`2[[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib]].ToList()
00007fff572367e8 00007fff576db538   NONE System.Linq.Enumerable+SelectManySingleSelectorIterator`2[[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib]]..ctor(System.Collections.Generic.IEnumerable`1, System.Func`2>)

In the JIT, the IR for the control (call address) expression is:

N001 (  3,  2) [000204] ------------       t204 =    LCL_VAR   ref    V03 loc2
                                                 /--*  t204   ref
N002 (  4,  3) [000205] ------------       t205 = *  LEA(b+0)  byref
                                                 /--*  t205   byref
N003 (  7,  5) [000206] ------------       t206 = *  IND       long
                                                 /--*  t206   long
N004 (  8,  6) [000207] ------------       t207 = *  LEA(b+72) long
                                                 /--*  t207   long
N005 ( 11,  8) [000208] ------------       t208 = *  IND       long
                                                 /--*  t208   long
N006 ( 12,  9) [000209] ------------       t209 = *  LEA(b+16) long
                                                 /--*  t209   long
N007 ( 15, 11) [000210] ------------       t210 = *  IND       long

BruceForstall · 2018-07-10T22:48:30Z

The source code is:

public override int GetHashCode()
        {
            EnsureValues();
            long hash = 1870403278L; //Arbitrary number from Anonymous Type GetHashCode implementation
            foreach (var v in this.Values.Values)
            {
                hash = (hash * -1521134295L) + (v == null ? 0 : v.GetHashCode());
            }

            return (int)hash;
        }

BruceForstall · 2018-07-10T22:49:06Z

@jkotas Could you take a look? Maybe you could more easily spot if there's a problem with the object model here someplace.

jkotas · 2018-07-19T21:54:52Z

The crash is caused by invalid IL generated via Reflection.Emit.

Method Name:  DynamicClass.lambda_method(System.Runtime.CompilerServices.Closure, AspNetCoreODataSample.Web.Models.Movie)

IL_0000: newobj 6000002 Microsoft.AspNet.OData.Query.Expressions.GroupByWrapper..ctor()
IL_0005: dup 
IL_0006: newobj 6000003 Microsoft.AspNet.OData.Query.Expressions.AggregationPropertyContainer+LastInChain..ctor()
IL_000b: dup 
IL_000c: ldstr 70000004 "ReleaseDate"
IL_0011: callvirt 6000005 
IL_0016: dup 
IL_0017: ldarg.1 
// This returns System.DateTimeOffset
IL_0018: callvirt 6000006 AspNetCoreODataSample.Web.Models.Movie.get_ReleaseDate()
// This passes it to method that object reference - bad type mismatch!!!!
IL_001d: callvirt 6000007 Microsoft.AspNet.OData.Query.Expressions.PropertyContainer+NamedProperty`1[[System.__Canon, System.Private.CoreLib]].set_Value(System.__Canon) IL_0022: callvirt 6000008 Microsoft.AspNet.OData.Query.Expressions.GroupByWrapper.set_GroupByContainer(Microsoft.AspNet.OData.Query.Expressions.AggregationPropertyContainer)
IL_0027: ret

The method with invalid IL is created at this callstack:

system_private_corelib!System.Reflection.Emit.DynamicMethod.GetMethodDescriptor()+0x10a
system_private_corelib!System.Reflection.Emit.DynamicMethod.CreateDelegate(System.Type, System.Object)+0x2a
DynamicClass.lambda_method(System.Runtime.CompilerServices.Closure, Microsoft.EntityFrameworkCore.Query.QueryContext)+0x1a2
microsoft_entityframeworkcore!Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.Execute[[System.Int32, System.Private.CoreLib]](System.Linq.Expressions.Expression)+0x7456ac12
microsoft_entityframeworkcore!Microsoft.EntityFrameworkCore.Query.Internal.EntityQueryProvider.Execute[[System.Int32, System.Private.CoreLib]](System.Linq.Expressions.Expression)+0x74562e78
remotion_linq!Remotion.Linq.QueryableBase`1[[System.__Canon, System.Private.CoreLib]].System.Collections.IEnumerable.GetEnumerator()+0x21
newtonsoft_json!Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeList(Newtonsoft.Json.JsonWriter, System.Collections.IEnumerable, Newtonsoft.Json.Serialization.JsonArrayContract, Newtonsoft.Json.Serialization.JsonProperty, Newtonsoft.Json.Serialization.JsonContainerContract, Newtonsoft.Json.Serialization.JsonProperty)+0xe5
newtonsoft_json!Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeValue(Newtonsoft.Json.JsonWriter, System.Object, Newtonsoft.Json.Serialization.JsonContract, Newtonsoft.Json.Serialization.JsonProperty, Newtonsoft.Json.Serialization.JsonContainerContract, Newtonsoft.Json.Serialization.JsonProperty)+0x18d
newtonsoft_json!Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.Serialize(Newtonsoft.Json.JsonWriter, System.Object, System.Type)+0xf1
newtonsoft_json!Newtonsoft.Json.JsonSerializer.SerializeInternal(Newtonsoft.Json.JsonWriter, System.Object, System.Type)+0x468
newtonsoft_json!Newtonsoft.Json.JsonSerializer.Serialize(Newtonsoft.Json.JsonWriter, System.Object)+0x14
microsoft_aspnetcore_mvc_formatters_json!Microsoft.AspNetCore.Mvc.Formatters.JsonOutputFormatter.WriteObject(System.IO.TextWriter, System.Object)+0x53
microsoft_aspnetcore_mvc_formatters_json!Microsoft.AspNetCore.Mvc.Formatters.JsonOutputFormatter+<WriteResponseBodyAsync>d__11.MoveNext()+0xf6
system_private_corelib!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[System.__Canon, System.Private.CoreLib]](System.__Canon ByRef)+0xffffffff`ab1048d9
microsoft_aspnetcore_mvc_formatters_json!Microsoft.AspNetCore.Mvc.Formatters.JsonOutputFormatter.WriteResponseBodyAsync(Microsoft.AspNetCore.Mvc.Formatters.OutputFormatterWriteContext, System.Text.Encoding)+0x65
microsoft_aspnetcore_mvc_core!Microsoft.AspNetCore.Mvc.Formatters.TextOutputFormatter.WriteAsync(Microsoft.AspNetCore.Mvc.Formatters.OutputFormatterWriteContext)+0x160

jkotas · 2018-07-19T21:56:49Z

This issue was moved to dotnet/efcore#12733

RussKeldorph assigned BruceForstall Jun 12, 2018

jkotas closed this as completed Jul 19, 2018

msftgits transferred this issue from dotnet/coreclr Jan 31, 2020

msftgits added this to the 3.0 milestone Jan 31, 2020

ghost locked as resolved and limited conversation to collaborators Dec 16, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

odata webapi core: sehexception when using $apply=aggregate #10426

odata webapi core: sehexception when using $apply=aggregate #10426

mysticdotnet commented May 31, 2018

jkotas commented Jun 2, 2018

jkotas commented Jun 2, 2018

mysticdotnet commented Jun 2, 2018

BruceForstall commented Jul 9, 2018

BruceForstall commented Jul 10, 2018

BruceForstall commented Jul 10, 2018

BruceForstall commented Jul 10, 2018

jkotas commented Jul 19, 2018

jkotas commented Jul 19, 2018

odata webapi core: sehexception when using $apply=aggregate #10426

odata webapi core: sehexception when using $apply=aggregate #10426

Comments

mysticdotnet commented May 31, 2018

jkotas commented Jun 2, 2018

jkotas commented Jun 2, 2018

mysticdotnet commented Jun 2, 2018

BruceForstall commented Jul 9, 2018

BruceForstall commented Jul 10, 2018

BruceForstall commented Jul 10, 2018

BruceForstall commented Jul 10, 2018

jkotas commented Jul 19, 2018

jkotas commented Jul 19, 2018