Infinite loop in compiled query cache

[kierenj]

After a while of usage, my ASP.NET Core API gets stuck at 100% CPU and needs to be restarted. By the way I've found my Azure App Service stuck at 100% CPU after 24+ hours, I'm just guessing it's infinite ;)

I would appreciate any pointers or tips as to what I can do/give to help diagnose further.

It's hard to tell exactly where the loop is occurring, but when I break execution, the call stack looks very much like this:

http://imgur.com/a/oKst0

(Lots of recursive ExpressionEqualityComparer.GetHashCode under CompiledQueryCache.GetOrAddQueryCore<>)

Edit: Bonus - in trying to repro this, I see an exception when calling those endpoints below. I don't know why this would occur sometimes, and other times the code gets stuck in a loop.:

fail: Microsoft.AspNetCore.Server.Kestrel[13]
      Connection id "0HL11STOA0872": An unhandled exception was thrown by the application.
System.ArgumentException: Expression of type 'Microsoft.EntityFrameworkCore.Query.ExpressionVisitors.Internal.ValueBufferShaper' cannot be used for parameter of type 'Microsoft.EntityFrameworkCore.Query.ExpressionVisitors.Internal.IShaper`1[MyProj.EntityModel.WidgetManufacturer]' of method 'System.Collections.Generic.IEnumerable`1[Microsoft.EntityFrameworkCore.Query.EntityQueryModelVisitor+TransparentIdentifier`2[Microsoft.EntityFrameworkCore.Storage.ValueBuffer,System.Collections.Generic.IEnumerable`1[MyProj.EntityModel.WidgetManufacturer]]] _GroupJoin[ValueBuffer,WidgetManufacturer,Nullable`1,TransparentIdentifier`2](Microsoft.EntityFrameworkCore.Query.RelationalQueryContext, System.Collections.Generic.IEnumerable`1[Microsoft.EntityFrameworkCore.Storage.ValueBuffer], Microsoft.EntityFrameworkCore.Query.ExpressionVisitors.Internal.IShaper`1[Microsoft.EntityFrameworkCore.Storage.ValueBuffer], Microsoft.EntityFrameworkCore.Query.ExpressionVisitors.Internal.IShaper`1[MyProj.EntityModel.WidgetManufacturer], System.Func`2[MyProj.EntityModel.WidgetManufacturer,System.Nullable`1[System.Guid]], System.Func`3[Microsoft.EntityFrameworkCore.Storage.ValueBuffer,System.Collections.Generic.IEnumerable`1[MyProj.EntityModel.WidgetManufacturer],Microsoft.EntityFrameworkCore.Query.EntityQueryModelVisitor+TransparentIdentifier`2[Microsoft.EntityFrameworkCore.Storage.ValueBuffer,System.Collections.Generic.IEnumerable`1[MyProj.EntityModel.WidgetManufacturer]]], Microsoft.EntityFrameworkCore.Query.Internal.GroupJoinInclude, Microsoft.EntityFrameworkCore.Query.Internal.GroupJoinInclude)'
Parameter name: arguments[3]
   at System.Dynamic.Utils.ExpressionUtils.ValidateOneArgument(MethodBase method, ExpressionType nodeKind, Expression arguments, ParameterInfo pi, String methodParamName, String argumentParamName, Int32 index)
   at System.Dynamic.Utils.ExpressionUtils.ValidateArgumentTypes(MethodBase method, ExpressionType nodeKind, ReadOnlyCollection`1& arguments, String methodParamName)
   at System.Linq.Expressions.Expression.Call(Expression instance, MethodInfo method, IEnumerable`1 arguments)
   at System.Linq.Expressions.MethodCallExpressionN.Rewrite(Expression instance, IList`1 args)
   at System.Linq.Expressions.ExpressionVisitor.VisitMethodCall(MethodCallExpression node)
   at System.Linq.Expressions.MethodCallExpression.Accept(ExpressionVisitor visitor)
   at Microsoft.EntityFrameworkCore.Query.ExpressionVisitors.ExpressionVisitorBase.Visit(Expression node)
   at System.Linq.Expressions.ExpressionVisitor.VisitAndConvert[T](ReadOnlyCollection`1 nodes, String callerName)
   at Microsoft.EntityFrameworkCore.Query.RelationalQueryModelVisitor.QuerySourceUpdater.VisitMethodCall(MethodCallExpression methodCallExpression)
   at System.Linq.Expressions.MethodCallExpression.Accept(ExpressionVisitor visitor)
   at Microsoft.EntityFrameworkCore.Query.ExpressionVisitors.ExpressionVisitorBase.Visit(Expression node)
   at Microsoft.EntityFrameworkCore.Query.EntityQueryModelVisitor.VisitMainFromClause(MainFromClause fromClause, QueryModel queryModel)
   at Remotion.Linq.QueryModelVisitorBase.VisitQueryModel(QueryModel queryModel)
   at Microsoft.EntityFrameworkCore.Query.EntityQueryModelVisitor.VisitQueryModel(QueryModel queryModel)
   at Microsoft.EntityFrameworkCore.Query.RelationalQueryModelVisitor.VisitQueryModel(QueryModel queryModel)
   at Microsoft.EntityFrameworkCore.Query.Internal.SqlServerQueryModelVisitor.VisitQueryModel(QueryModel queryModel)
   at Microsoft.EntityFrameworkCore.Query.EntityQueryModelVisitor.CreateQueryExecutor[TResult](QueryModel queryModel)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.CompileQueryCore[TResult](Expression query, INodeTypeProvider nodeTypeProvider, IDatabase database, ILogger logger, Type contextType)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.<>c__DisplayClass15_0`1.<Execute>b__0()
   at Microsoft.EntityFrameworkCore.Query.Internal.CompiledQueryCache.GetOrAddQueryCore[TFunc](Object cacheKey, Func`1 compiler)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.Execute[TResult](Expression query)
   at Remotion.Linq.QueryableBase`1.System.Collections.IEnumerable.GetEnumerator()

Steps to reproduce

Same model as #6937, #6938, #7102.

The two queries sitting somewhere in the call stacks are (note - I can't be sure it's these ones in particular causing itI've confirmed these were the methods causing my problem by re-ordering the LINQ clauses and not having the problem since):

            var identities = _context.Set<WidgetManufacturedIdentity>();
            var result = identities.Where(i => i.Manufacturer.Id == id && !i.Manufacturer.Retired.HasValue)
                .Select(i => i.RangeName)
                .Where(r => r != null && r != "")
                .Distinct()
                .Select(r => new { name = r });
            return Json(result); // for ref, it's a controller method

and

            var identities = _context.Set<WidgetManufacturedIdentity>();
            var result = identities.Where(i => i.Manufacturer.Id == id && !i.Manufacturer.Retired.HasValue)
                .Select(i => i.BrandName)
                .Where(b => b != null && b != "")
                .Distinct()
                .Select(b => new { name = b });
            return Json(result); // for ref, it's a controller method

Further technical details

EF Core version: 1.2.0-preview1-22878
Database Provider: Microsoft.EntityFrameworkCore.SqlServer
Operating system: Window 10.0.14393
IDE: VS2015, commandline, Azure App Service

[divega]

@anpete please investigate and see what releases this applies to.

[anpete]

Fixed in rel/1.1.1

[Eilon]

This patch bug is approved. Please use the normal code review process w/ a PR and make sure the fix is in the correct branch, then close the bug and mark it as done.

[divega]

@anpete closing again since the fix is in and approved for the patch.

[maumar]

verified