EntityFramework's dbContext started throwing an EntityCommandCompilationException

[ysmoradi]

Suddenly every usage of ef started throwing the following exception:

System.Data.Entity.Core.EntityCommandCompilationException: An error occurred while preparing the command definition. See the inner exception for details.\r\n ---> System.InvalidOperationException: Operations that change non-concurrent collections must have exclusive access. A concurrent update was performed on this collection and corrupted its state. The collection's state is no longer correct.\r\n   at System.Collections.Generic.Dictionary`2.FindEntry(TKey key)\r\n   at System.Data.Entity.Core.Query.InternalTrees.BitVec.ArrayPool.GetBag(Int32 length)\r\n   at System.Data.Entity.Core.Query.InternalTrees.BitVec.ArrayPool.GetArray(Int32 length)\r\n   at System.Data.Entity.Core.Query.InternalTrees.BitVec..ctor(Int32 length, Boolean defaultValue)\r\n   at System.Data.Entity.Core.Query.InternalTrees.Command.CreateVarVec()\r\n   at System.Data.Entity.Core.Query.InternalTrees.Table..ctor(Command command, TableMD tableMetadata, Int32 tableId)\r\n   at System.Data.Entity.Core.Query.InternalTrees.Command.CreateTableInstance(TableMD tableMetadata)\r\n   at System.Data.Entity.Core.Query.PlanCompiler.ITreeGenerator.Visit(DbScanExpression e)\r\n   at System.Data.Entity.Core.Query.PlanCompiler.ITreeGenerator.VisitBoundExpression(DbExpression boundExpression, Var& boundVar)\r\n   at System.Data.Entity.Core.Query.PlanCompiler.ITreeGenerator.EnterExpressionBinding(DbExpressionBinding binding)\r\n   at System.Data.Entity.Core.Query.PlanCompiler.ITreeGenerator.Visit(DbFilterExpression e)\r\n   at System.Data.Entity.Core.Query.PlanCompiler.ITreeGenerator.VisitBoundExpression(DbExpression boundExpression, Var& boundVar)\r\n   at System.Data.Entity.Core.Query.PlanCompiler.ITreeGenerator.EnterExpressionBinding(DbExpressionBinding binding)\r\n   at System.Data.Entity.Core.Query.PlanCompiler.ITreeGenerator.GenerateStandardProject(DbProjectExpression e)\r\n   at System.Data.Entity.Core.Query.PlanCompiler.ITreeGenerator.Visit(DbLimitExpression expression)\r\n   at System.Data.Entity.Core.Query.PlanCompiler.ITreeGenerator..ctor(DbQueryCommandTree ctree, DiscriminatorMap discriminatorMap)\r\n   at System.Data.Entity.Core.Query.PlanCompiler.PlanCompiler.Initialize()\r\n   at System.Data.Entity.Core.Query.PlanCompiler.PlanCompiler.Compile(List`1& providerCommands, ColumnMap& resultColumnMap, Int32& columnCount, Set`1& entitySets)\r\n   at System.Data.Entity.Core.EntityClient.Internal.EntityCommandDefinition..ctor(DbProviderFactory storeProviderFactory, DbCommandTree commandTree, DbInterceptionContext interceptionContext, IDbDependencyResolver resolver, BridgeDataReaderFactory bridgeDataReaderFactory, ColumnMapFactory columnMapFactory)\r\n   --- End of inner exception stack trace ---\r\n   at System.Data.Entity.Core.EntityClient.Internal.EntityCommandDefinition..ctor(DbProviderFactory storeProviderFactory, DbCommandTree commandTree, DbInterceptionContext interceptionContext, IDbDependencyResolver resolver, BridgeDataReaderFactory bridgeDataReaderFactory, ColumnMapFactory columnMapFactory)\r\n   at System.Data.Entity.Core.EntityClient.Internal.EntityProviderServices.CreateDbCommandDefinition(DbProviderManifest providerManifest, DbCommandTree commandTree, DbInterceptionContext interceptionContext)\r\n   at System.Data.Entity.Core.Objects.Internal.ObjectQueryExecutionPlanFactory.CreateCommandDefinition(ObjectContext context, DbQueryCommandTree tree)\r\n   at System.Data.Entity.Core.Objects.Internal.ObjectQueryExecutionPlanFactory.Prepare(ObjectContext context, DbQueryCommandTree tree, Type elementType, MergeOption mergeOption, Boolean streaming, Span span, IEnumerable`1 compiledQueryParameters, AliasGenerator aliasGenerator)\r\n   at System.Data.Entity.Core.Objects.ELinq.ELinqQueryState.GetExecutionPlan(Nullable`1 forMergeOption)\r\n   at System.Data.Entity.Core.Objects.ObjectQuery`1.<>c__DisplayClass43_0.<GetResultsAsync>b__1()\r\n   at System.Data.Entity.Core.Objects.ObjectContext.ExecuteInTransactionAsync[T](Func`1 func, IDbExecutionStrategy executionStrategy, Boolean startLocalTransaction, Boolean releaseConnectionOnSuccess, CancellationToken cancellationToken)\r\n   at System.Data.Entity.SqlServer.DefaultSqlExecutionStrategy.ExecuteAsyncImplementation[TResult](Func`1 func)\r\n   at System.Data.Entity.Core.Objects.ObjectQuery`1.GetResultsAsync(Nullable`1 forMergeOption, IDbExecutionStrategy executionStrategy, CancellationToken cancellationToken)\r\n   at System.Data.Entity.Internal.LazyAsyncEnumerator`1.FirstMoveNextAsync(CancellationToken cancellationToken)\r\n   at System.Data.Entity.Infrastructure.IDbAsyncEnumerableExtensions.SingleOrDefaultAsync[TSource](IDbAsyncEnumerable`1 source, CancellationToken cancellationToken)\r\n   at XFront.ServerCodes.Implementations.XFrontDeviceIdStaffIdCompanyIdManagementMiddleware.ManageStaffIdDeviceIdCompanyId(IOwinContext context)\r\n   at XFront.ServerCodes.Implementations.XFrontDeviceIdStaffIdCompanyIdManagementMiddleware.Invoke(IOwinContext context)\r\n   at Microsoft.Owin.Mapping.MapMiddleware.Invoke(IDictionary`2 environment)\r\n   at Microsoft.Owin.Mapping.MapMiddleware.Invoke(IDictionary`2 environment)\r\n   at Microsoft.Owin.Mapping.MapMiddleware.Invoke(IDictionary`2 environment)\r\n   at Microsoft.Owin.Mapping.MapMiddleware.Invoke(IDictionary`2 environment)\r\n   at Microsoft.Owin.Mapping.MapMiddleware.Invoke(IDictionary`2 environment)\r\n   at Microsoft.Owin.Mapping.MapMiddleware.Invoke(IDictionary`2 environment)\r\n   at Microsoft.Owin.Mapping.MapWhenMiddleware.Invoke(IDictionary`2 environment)\r\n   at Microsoft.AspNetCore.Owin.WebSocketAcceptAdapter.<>c__DisplayClass6_0.<<AdaptWebSockets>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.Owin.Security.Infrastructure.AuthenticationMiddleware`1.Invoke(IOwinContext context)\r\n   at Microsoft.AspNetCore.Owin.WebSocketAcceptAdapter.<>c__DisplayClass6_0.<<AdaptWebSockets>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Bit.OwinCore.Middlewares.AspNetCoreExceptionHandlerMiddleware.Invoke(HttpContext context)",
  "ExceptionAdditionalMessage": "Request-Execution-Exception",
  "ExceptionType": "System.Data.Entity.Core.EntityCommandCompilationException",
  "ExceptionTypeAssemblyQualifiedName": "System.Data.Entity.Core.EntityCommandCompilationException, EntityFramework, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"
}

Steps to reproduce

N/A

Further technical details

EF version: 6.3.0-preview8-19405-04
Database Provider: EntityFramework.SqlServer
Operating system: Microsoft Windows NT 10.0.17763.0 X64
IDE: VS 16.2.2
.NET Core 3.0.0-preview8-28405-07

Restarting IIS/App resolved the issue.

[kosinsky]

I'm seeing the same issue too for EF 6.3 on .NET Classic. The issue looks transient race condition as a result it's difficult to get repro :(

[preston-m-price]

Looks like unsafe dictionary access: https://github.com/aspnet/EntityFramework6/blob/7d7d5fe22240bebaae76352828165822aa99940a/src/EntityFramework/Core/Query/InternalTrees/VarVec.cs#L873

[bricelam]

Introduced by PR #430? cc @cjpearson

[ajcvickers]

Triage: @bricelam will revert the PR for 6.3 Ask Mode. We will re-consider it for 6.4.

[Pilchie]

Ask Mode Template

Description

Revert an optimization PR that caused regressions from shipping EF.

Customer Impact

Found by a few customers using EF6.

Regression?

Yes, regression from EF 6.2, introduced in June.

Risk

Low – it’s a straight revert of the original PR to get back to shipped behaviour

[elizatarasila]

When will this be deployed?

[bricelam]

Daily builds are available to test now. Use version 6.3.0-rc2-*