Operations that change non-concurrent collections must have exclusive access

[Ro3A]

I ran into the exact same issue as the one mentioned on the IdentityServer github:

IdentityServer/IdentityServer4#2453

This occured when I switched from .net core 2.1.1 to 2.1.2. As soon as I reverted back, the issue went away. I'm not using IdentityServer so this must be a .net core issue specific to 2.1.2. I didn't see an open issue over here yet so figured I'd create one.

InvalidOperationException: Operations that change non-concurrent collections must have exclusive access. A concurrent update was performed on this collection and corrupted its state. The collection's state is no longer correct. System.Collections.Generic.Dictionary<TKey, TValue>.FindEntry(TKey key) System.Collections.Generic.Dictionary<TKey, TValue>.TryGetValue(TKey key, out TValue value) Microsoft.EntityFrameworkCore.Metadata.Internal.Model.GetDisplayName(Type type) Microsoft.EntityFrameworkCore.Metadata.Internal.Model.FindEntityType(Type type) Microsoft.EntityFrameworkCore.ModelExtensions.FindEntityType(IModel model, Type type) Microsoft.EntityFrameworkCore.Query.Internal.IncludeCompiler+IncludeLoadTreeNodeBase.Compile(QueryCompilationContext queryCompilationContext, QueryModel queryModel, bool trackingQuery, bool asyncQuery, ref int collectionIncludeId, QuerySourceReferenceExpression targetQuerySourceReferenceExpression) Microsoft.EntityFrameworkCore.Query.Internal.IncludeCompiler+IncludeLoadTreeNode.CompileCollectionInclude(QueryCompilationContext queryCompilationContext, Expression targetExpression, Expression entityParameter, bool trackingQuery, bool asyncQuery, ref int collectionIncludeId) Microsoft.EntityFrameworkCore.Query.Internal.IncludeCompiler+IncludeLoadTreeNodeBase.Compile(QueryCompilationContext queryCompilationContext, QueryModel queryModel, bool trackingQuery, bool asyncQuery, ref int collectionIncludeId, QuerySourceReferenceExpression targetQuerySourceReferenceExpression) Microsoft.EntityFrameworkCore.Query.Internal.IncludeCompiler+IncludeLoadTree.Compile(QueryCompilationContext queryCompilationContext, QueryModel queryModel, bool trackingQuery, bool asyncQuery, ref int collectionIncludeId) Microsoft.EntityFrameworkCore.Query.Internal.IncludeCompiler.CompileIncludes(QueryModel queryModel, bool trackingQuery, bool asyncQuery) Microsoft.EntityFrameworkCore.Query.EntityQueryModelVisitor.OptimizeQueryModel(QueryModel queryModel, bool asyncQuery) Microsoft.EntityFrameworkCore.Query.RelationalQueryModelVisitor.OptimizeQueryModel(QueryModel queryModel, bool asyncQuery) Microsoft.EntityFrameworkCore.Query.EntityQueryModelVisitor.CreateAsyncQueryExecutor(QueryModel queryModel) Microsoft.EntityFrameworkCore.Storage.Database.CompileAsyncQuery(QueryModel queryModel) Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.CompileAsyncQueryCore(Expression query, IQueryModelGenerator queryModelGenerator, IDatabase database) Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler+<>c__DisplayClass22_0.b__0() Microsoft.EntityFrameworkCore.Query.Internal.CompiledQueryCache.GetOrAddQueryCore(object cacheKey, Func<Func<QueryContext, TFunc>> compiler) Microsoft.EntityFrameworkCore.Query.Internal.CompiledQueryCache.GetOrAddAsyncQuery(object cacheKey, Func<Func<QueryContext, IAsyncEnumerable>> compiler) Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.CompileAsyncQuery(Expression query) Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.ExecuteAsync(Expression query) Microsoft.EntityFrameworkCore.Query.Internal.EntityQueryProvider.ExecuteAsync(Expression expression) Microsoft.EntityFrameworkCore.Query.Internal.EntityQueryable.System.Collections.Generic.IAsyncEnumerable.GetEnumerator() System.Linq.AsyncEnumerable.Aggregate_<TSource, TAccumulate, TResult>(IAsyncEnumerable source, TAccumulate seed, Func<TAccumulate, TSource, TAccumulate> accumulator, Func<TAccumulate, TResult> resultSelector, CancellationToken cancellationToken)

Here's the offending code, in my case:

var result = await _someDbContext.SomeTable
.Where(x => x.Active.HasValue && x.Active.Value == true)
.Select(x => new SomeModel
{
Name = x.Name,
Id = x.Id
}).ToListAsync();

[benaadams]

This is an intentional exception that is being thrown by the Dictionary in newer .NET Core https://github.com/dotnet/corefx/issues/28123 which indicates the dictionary is being incorrectly modified concurrently by multiple threads (it wouldn't have been detected previously and could lead to infinite loops)

[benaadams]

It would suggest that

Microsoft.EntityFrameworkCore.Metadata.Internal.Model.GetDisplayName(Type type)

can be called by multiple threads on the same instance; one would fail on TryGetValue and add a new item to the Dictionary while another was calling TryGetValue; which would lead to this issue.

Either it should be a ConcurrentDictionary or a lock used around the access+add

[fschlaef]

Comment moved to #12682

[benaadams]

@fschlaef that stack trace looks to be a different issue this time in re-motion Relinq?

It suggests NodeTypeProviders.Register is being called concurrently with NodeTypeProviders.GetNodeType which causes and issue as the Dictionary is neither a ConcurrentDictionary nor used under lock

Which is called via Microsoft.EntityFrameworkCore.Query.Internal.QueryModelGenerator.ParseQuery

[jwcao]

The first time I saw the issue in IdentityServer/IdentityServer4#2453 was on .Net Core 2.1.0 and it happened again on 2.1.1.

[benaadams]

Added an issue to corefx so its easier to find information about this exception in search https://github.com/dotnet/corefx/issues/31186

[smitpatel]

@fschlaef - Your issue looks similar to #12682

[fschlaef]

@smitpatel You are right, as @benaadams thought this is a different issue. I moved my comment there to avoid pollution.

[Ro3A]

Just to be sure, since there was a lot of movement around this across a few different github threads, this is not something I need to change in my code but is in fact an issue in core right?

[benaadams]

this is not something I need to change in my code

@Ro3A no; this looks like an EF issue

[divega]

Once we have the fix we will consider whether we need to include it in a patch.

[AndriySvyryd]

Justification: This is a regression without a good workaround (other than just using one thread)
Risk: Low, fix just replaces the dictionary implementation, no functional changes are made

[ajcvickers]

Description

A non-thread safe dictionary can be accessed from multiple threads when using an EF model.

Customer Impact

2.1: Occasional exceptions due to this corefx change: https://github.com/dotnet/corefx/issues/28123
Pre 2.0: Presumably even rarer failures due to the unsafe access, but nothing ever reported.

Regression?

 Technically, no. Effectively, yes, since 2.1 apps are now more likely to fail than 2.0 apps.

Risk

Very low; use ConcurrentDictionary instead,.

[vivmishra]

Approved for 2.1.4

[ajcvickers]

@natemcmaster @ryanbrandenburg Are we okay to start checking things in to a release branch for 2.1.4, or should we hold off for a while?

[natemcmaster]

Once #12888 is in, go for it.

[tomchovanec]

Is there any temporary workround? Restarting AppPool whenever whis exception occures is not a good solution, I have this exception on every query execution. (Microsoft.EntityFrameworkCore 2.1.1)

[natemcmaster]

@tomchovanec you could try using a nightly build which has this fix, such as https://dotnet.myget.org/feed/aspnetcore-dev/package/nuget/Microsoft.EntityFrameworkCore.SqlServer/2.1.3-rtm-30962.

[patriksvensson]

@natemcmaster Been hitting this as well lately. When is 2.1.3 with the fix expected to be released?

[natemcmaster]

@patriksvensson next week.

[tomchovanec]

@natemcmaster Thank you, nightly build could help temporary. I was looking for some ungly hack because it's not possible to compile sources or add them to the solution that contains 60+ projects while many other developers works on it.

[johnnyoshika]

I ran into this problem as well (which required an AppPool restart to fix). To get around this problem, I'd like to upgrade EF Core. The NuGet package I'm loading is referenced like this:

<PackageReference Include="Microsoft.AspNetCore.App" />

According to this page https://docs.microsoft.com/en-us/aspnet/core/migration/20_21?view=aspnetcore-2.1, version attribute is no longer recommended. Excerpt:

Remove the "Version" attribute on the package reference to Microsoft.AspNetCore.App. Projects which use do not need to set the version. The version will be implied by the target framework and selected to best match the way ASP.NET Core 2.1 works. (See below for more information.)

A couple of questions:

• How do I know which version of EF Core it's loading?
• How do I upgrade the EF Core version if I'm loading Microsoft.AspNetCore.App?

Thanks in advance!

[stefankip]

I'm still running into this with EFCore 3.1.4.

Code:

someItems.AsParallel().ForAll(x =>
{
    var existingItem = currentItems.SingleOrDefault(y => y.Id== x.Id && y.Status == x.Status);
    if (existingItem != null)
    {
        x.Item= existingItem.Item;
        context.Entry(existingItem).State = EntityState.Unchanged;
    }
    else
    {
        context.Items.Add(x); // -----> this is where the exception is thrown
    }
});

Exception:

InvalidOperationException: Operations that change non-concurrent collections must have exclusive access. A concurrent update was performed on this collection and corrupted its state. The collection's state is no longer correct.

Stacktrace:

   at System.ThrowHelper.ThrowInvalidOperationException_ConcurrentOperationsNotSupported()
   at System.Collections.Generic.Dictionary`2.FindEntry(TKey key)
   at System.Collections.Generic.Dictionary`2.TryGetValue(TKey key, TValue& value)
   at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.EntityReferenceMap.TryGet(Object entity, IEntityType entityType, InternalEntityEntry& entry, Boolean throwOnNonUniqueness)
   at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.TryGetEntry(Object entity, IEntityType entityType, Boolean throwOnTypeMismatch)
   at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.GetOrCreateEntry(Object entity, IEntityType entityType)
   at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.EntityEntryGraphIterator.TraverseGraph[TState](EntityEntryGraphNode`1 node, Func`2 handleNode)
   at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.EntityGraphAttacher.AttachGraph(InternalEntityEntry rootEntry, EntityState targetState, EntityState storeGeneratedWithKeySetTargetState, Boolean forceStateWhenUnknownKey)
   at Microsoft.EntityFrameworkCore.DbContext.SetEntityState(InternalEntityEntry entry, EntityState entityState)
   at Microsoft.EntityFrameworkCore.DbContext.SetEntityState[TEntity](TEntity entity, EntityState entityState)
   at Microsoft.EntityFrameworkCore.DbContext.Add[TEntity](TEntity entity)
   at Microsoft.EntityFrameworkCore.Internal.InternalDbSet`1.Add(TEntity entity)
   at <MyCode>
   at System.Linq.Parallel.ForAllOperator`1.ForAllEnumerator`1.MoveNext(TInput& currentElement, Int32& currentKey)
   at System.Linq.Parallel.ForAllSpoolingTask`2.SpoolingWork()
   at System.Linq.Parallel.SpoolingTaskBase.Work()

[roji]

@stefankip your problem isn't really related to this issue: performing concurrent operations on a DbContext - as you're doing with AsParallel - isn't supported (and there isn't much to be gained from doing so). You'll need to do a simple, non-parallel foreach. See #18148 for a similar discussion on this.

[stefankip]

Yeah I already change it to foreach, but the forall was performing a lot faster, because the someItems collection is a List.
I thought the call to DbContext.Add() was a simple Add(), it's not performing the underlying SQL calls yet right?

[roji]

That's right - in almost all cases, Add doesn't do any database traffic. The exception is if you have value generation such as HiLo which contacts the database (in which case you once again cannot parallelize, since a context uses only a single database connection).

[stefankip]

So if I'm only doing stuff in-memory at this point, why can't I use parallelization? Using ForAll performs much better on large lists than foreach.

[roji]

Operations being in-memory only doesn't mean they're thread-safe: .NET Dictionary itself is in-memory only, and does not allow concurrent updates, since doing so safely would hurt performance for everyone not needing that. The discussion in #18148 has some more details.