Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Process crash with "Internal CLR error" (0x80131506) in Roslyn CI #45557

Closed
tmat opened this issue Dec 3, 2020 · 27 comments · Fixed by #45818 or #46059
Closed

Process crash with "Internal CLR error" (0x80131506) in Roslyn CI #45557

tmat opened this issue Dec 3, 2020 · 27 comments · Fixed by #45818 or #46059
Assignees
Labels
area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI bug runtime-coreclr specific to the CoreCLR runtime tenet-reliability Reliability/stability related issue (stress, load problems, etc.)
Milestone

Comments

@tmat
Copy link
Member

tmat commented Dec 3, 2020

Description

Roslyn CI failed with the following error:

https://dev.azure.com/dnceng/public/_build/results?buildId=906787&view=logs&j=7817cc08-46f6-529a-b675-9c2d2b016c0b&t=0fd245be-2f58-5546-b778-7c0bac845221&l=68

The active test run was aborted. Reason: Test host process crashed : Fatal error. Internal CLR error. (0x80131506)
   at System.Collections.Concurrent.ConcurrentDictionary`2[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]]..ctor(Int32, Int32, Boolean, System.Collections.Generic.IEqualityComparer`1<System.__Canon>)
   at System.Collections.Concurrent.ConcurrentDictionary`2[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]]..ctor()
   at System.RuntimeType.CreateInstanceDefaultCtor(Boolean, Boolean, Boolean, Boolean)
   at System.Activator.CreateInstance[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]]()
   at System.Threading.LazyInitializer.EnsureInitializedCore[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](System.__Canon ByRef)
   at Microsoft.CodeAnalysis.Compilation.MarkImportDirectiveAsUsed(Microsoft.CodeAnalysis.SyntaxTree, Int32)
   at Microsoft.CodeAnalysis.Compilation.MarkImportDirectiveAsUsed(Microsoft.CodeAnalysis.SyntaxNode)
   at Microsoft.CodeAnalysis.CSharp.Imports.LookupSymbolInAliases(Microsoft.CodeAnalysis.CSharp.Binder, Microsoft.CodeAnalysis.CSharp.LookupResult, System.String, Int32, Roslyn.Utilities.ConsList`1<Microsoft.CodeAnalysis.CSharp.Symbols.TypeSymbol>, Microsoft.CodeAnalysis.CSharp.LookupOptions, Boolean, System.Collections.Generic.HashSet`1<Microsoft.CodeAnalysis.DiagnosticInfo> ByRef)
   at Microsoft.CodeAnalysis.CSharp.InContainerBinder.LookupSymbolsInSingleBinder(Microsoft.CodeAnalysis.CSharp.LookupResult, System.String, Int32, Roslyn.Utilities.ConsList`1<Microsoft.CodeAnalysis.CSharp.Symbols.TypeSymbol>, Microsoft.CodeAnalysis.CSharp.LookupOptions, Microsoft.CodeAnalysis.CSharp.Binder, Boolean, System.Collections.Generic.HashSet`1<Microsoft.CodeAnalysis.DiagnosticInfo> ByRef)
   at Microsoft.CodeAnalysis.CSharp.Binder.LookupSymbolsInternal(Microsoft.CodeAnalysis.CSharp.LookupResult, System.String, Int32, Roslyn.Utilities.ConsList`1<Microsoft.CodeAnalysis.CSharp.Symbols.TypeSymbol>, Microsoft.CodeAnalysis.CSharp.LookupOptions, Boolean, System.Collections.Generic.HashSet`1<Microsoft.CodeAnalysis.DiagnosticInfo> ByRef)
   at Microsoft.CodeAnalysis.CSharp.Binder.BindNonGenericSimpleNamespaceOrTypeOrAliasSymbol(Microsoft.CodeAnalysis.CSharp.Syntax.IdentifierNameSyntax, Microsoft.CodeAnalysis.DiagnosticBag, Roslyn.Utilities.ConsList`1<Microsoft.CodeAnalysis.CSharp.Symbols.TypeSymbol>, Boolean, Microsoft.CodeAnalysis.CSharp.Symbols.NamespaceOrTypeSymbol)
   at Microsoft.CodeAnalysis.CSharp.Binder.BindNamespaceOrTypeOrAliasSymbol(Microsoft.CodeAnalysis.CSharp.Syntax.ExpressionSyntax, Microsoft.CodeAnalysis.DiagnosticBag, Roslyn.Utilities.ConsList`1<Microsoft.CodeAnalysis.CSharp.Symbols.TypeSymbol>, Boolean)
   at Microsoft.CodeAnalysis.CSharp.Binder.BindTypeOrAlias(Microsoft.CodeAnalysis.CSharp.Syntax.ExpressionSyntax, Microsoft.CodeAnalysis.DiagnosticBag, Roslyn.Utilities.ConsList`1<Microsoft.CodeAnalysis.CSharp.Symbols.TypeSymbol>, Boolean)
   at Microsoft.CodeAnalysis.CSharp.Binder.BindTypeOrAliasOrConstraintKeyword(Microsoft.CodeAnalysis.CSharp.Syntax.TypeSyntax, Microsoft.CodeAnalysis.DiagnosticBag, ConstraintContextualKeyword ByRef)
   at Microsoft.CodeAnalysis.CSharp.Binder.BindTypeParameterConstraints(Microsoft.CodeAnalysis.CSharp.Syntax.TypeParameterSyntax, Microsoft.CodeAnalysis.CSharp.Syntax.TypeParameterConstraintClauseSyntax, Boolean, Microsoft.CodeAnalysis.DiagnosticBag)
   at Microsoft.CodeAnalysis.CSharp.Binder.BindTypeParameterConstraintClauses(Microsoft.CodeAnalysis.CSharp.Symbol, System.Collections.Immutable.ImmutableArray`1<Microsoft.CodeAnalysis.CSharp.Symbols.TypeParameterSymbol>, Microsoft.CodeAnalysis.CSharp.Syntax.TypeParameterListSyntax, Microsoft.CodeAnalysis.SyntaxList`1<Microsoft.CodeAnalysis.CSharp.Syntax.TypeParameterConstraintClauseSyntax>, Microsoft.CodeAnalysis.DiagnosticBag, Boolean, Boolean)
   at Microsoft.CodeAnalysis.CSharp.Symbols.ConstraintsHelper.MakeTypeParameterConstraintKinds(Microsoft.CodeAnalysis.CSharp.Symbols.MethodSymbol, Microsoft.CodeAnalysis.CSharp.Binder, System.Collections.Immutable.ImmutableArray`1<Microsoft.CodeAnalysis.CSharp.Symbols.TypeParameterSymbol>, Microsoft.CodeAnalysis.CSharp.Syntax.TypeParameterListSyntax, Microsoft.CodeAnalysis.SyntaxList`1<Microsoft.CodeAnalysis.CSharp.Syntax.TypeParameterConstraintClauseSyntax>)
   at Microsoft.CodeAnalysis.CSharp.Symbols.SourceOrdinaryMethodSymbol.GetTypeParameterConstraintKinds()
   at Microsoft.CodeAnalysis.CSharp.Symbols.SourceOrdinaryMethodSymbol.GetTypeParameterConstraintTypes()
   at Microsoft.CodeAnalysis.CSharp.Symbols.SourceMethodTypeParameterSymbol.ResolveBounds(Roslyn.Utilities.ConsList`1<Microsoft.CodeAnalysis.CSharp.Symbols.TypeParameterSymbol>, Microsoft.CodeAnalysis.DiagnosticBag)
   at Microsoft.CodeAnalysis.CSharp.Symbols.SourceTypeParameterSymbolBase.GetBounds(Roslyn.Utilities.ConsList`1<Microsoft.CodeAnalysis.CSharp.Symbols.TypeParameterSymbol>)
   at Microsoft.CodeAnalysis.CSharp.Symbols.SourceTypeParameterSymbolBase.GetConstraintTypes(Roslyn.Utilities.ConsList`1<Microsoft.CodeAnalysis.CSharp.Symbols.TypeParameterSymbol>)
   at Microsoft.CodeAnalysis.CSharp.Symbols.TypeParameterSymbol.EnsureAllConstraintsAreResolved(System.Collections.Immutable.ImmutableArray`1<Microsoft.CodeAnalysis.CSharp.Symbols.TypeParameterSymbol>)
   at Microsoft.CodeAnalysis.CSharp.Symbols.VarianceSafety.CheckTypeParametersVarianceSafety(System.Collections.Immutable.ImmutableArray`1<Microsoft.CodeAnalysis.CSharp.Symbols.TypeParameterSymbol>, Microsoft.CodeAnalysis.CSharp.Symbols.MethodSymbol, Microsoft.CodeAnalysis.DiagnosticBag)
   at Microsoft.CodeAnalysis.CSharp.Symbols.VarianceSafety.CheckMethodVarianceSafety(Microsoft.CodeAnalysis.CSharp.Symbols.MethodSymbol, LocationProvider`1<Microsoft.CodeAnalysis.CSharp.Symbols.MethodSymbol>, Microsoft.CodeAnalysis.DiagnosticBag)
   at Microsoft.CodeAnalysis.CSharp.Symbols.VarianceSafety.CheckMethodVarianceSafety(Microsoft.CodeAnalysis.CSharp.Symbols.MethodSymbol, Microsoft.CodeAnalysis.DiagnosticBag)
   at Microsoft.CodeAnalysis.CSharp.Symbols.VarianceSafety.CheckInterfaceVarianceSafety(Microsoft.CodeAnalysis.CSharp.Symbols.NamedTypeSymbol, Microsoft.CodeAnalysis.DiagnosticBag)
   at Microsoft.CodeAnalysis.CSharp.Symbols.SourceMemberContainerTypeSymbol.GetSynthesizedExplicitImplementations(System.Threading.CancellationToken)
   at Microsoft.CodeAnalysis.CSharp.Symbols.SourceMemberContainerTypeSymbol.ForceComplete(Microsoft.CodeAnalysis.SourceLocation, System.Threading.CancellationToken)
   at Microsoft.CodeAnalysis.CSharp.Symbol.ForceCompleteMemberByLocation(Microsoft.CodeAnalysis.SourceLocation, Microsoft.CodeAnalysis.CSharp.Symbol, System.Threading.CancellationToken)
   at Roslyn.Utilities.RoslynParallel+<>c__DisplayClass1_0.<For>g__errorHandlingBody|0(Int32)
   at System.Threading.Tasks.Parallel+<>c__DisplayClass19_0`1[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].<ForWorker>b__1(System.Threading.Tasks.RangeWorker ByRef, Int32, Boolean ByRef)
   at System.Threading.Tasks.TaskReplicator+Replica.Execute()
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(System.Threading.Thread, System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread)
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
Command: test "F:\workspace\_work\1\s\artifacts\bin\Microsoft.CodeAnalysis.CSharp.Symbol.UnitTests\Release\net5.0\Microsoft.CodeAnalysis.CSharp.Symbol.UnitTests.dll" --filter <PrivateImplementationDetails>+__StaticArrayInitTypeSize=10|<PrivateImplementationDetails>+__StaticArrayInitTypeSize=100|<PrivateImplementationDetails>+__StaticArrayInitTypeSize=104|<PrivateImplementationDetails>+__StaticArrayInitTypeSize=108|<PrivateImplementationDetails>+__StaticArrayInitTypeSize=12|<PrivateImplementationDetails>+__StaticArrayInitTypeSize=120|<PrivateImplementationDetails>+__StaticArrayInitTypeSize=132|<PrivateImplementationDetails>+__StaticArrayInitTypeSize=136|<PrivateImplementationDetails>+__StaticArrayInitTypeSize=144|<PrivateImplementationDetails>+__StaticArrayInitTypeSize=16|<PrivateImplementationDetails>+__StaticArrayInitTypeSize=176|<PrivateImplementationDetails>+__StaticArrayInitTypeSize=1764|<PrivateImplementationDetails>+__StaticArrayInitTypeSize=20|<PrivateImplementationDetails>+__StaticArrayInitTypeSize=224|<PrivateImplementationDetails>+__StaticArrayInitTypeSize=24|<PrivateImplementationDetails>+__StaticArrayInitTypeSize=28|<PrivateImplementationDetails>+__StaticArrayInitTypeSize=32|<PrivateImplementationDetails>+__StaticArrayInitTypeSize=36|<PrivateImplementationDetails>+__StaticArrayInitTypeSize=40|<PrivateImplementationDetails>+__StaticArrayInitTypeSize=44|<PrivateImplementationDetails>+__StaticArrayInitTypeSize=48|<PrivateImplementationDetails>+__StaticArrayInitTypeSize=5|<PrivateImplementationDetails>+__StaticArrayInitTypeSize=52|<PrivateImplementationDetails>+__StaticArrayInitTypeSize=56|<PrivateImplementationDetails>+__StaticArrayInitTypeSize=60|<PrivateImplementationDetails>+__StaticArrayInitTypeSize=64|<PrivateImplementationDetails>+__StaticArrayInitTypeSize=68|<PrivateImplementationDetails>+__StaticArrayInitTypeSize=72|<PrivateImplementationDetails>+__StaticArrayInitTypeSize=76|<PrivateImplementationDetails>+__StaticArrayInitTypeSize=80|<PrivateImplementationDetails>+__StaticArrayInitTypeSize=820|<PrivateImplementationDetails>+__StaticArrayInitTypeSize=84|<PrivateImplementationDetails>+__StaticArrayInitTypeSize=88|<PrivateImplementationDetails>+__StaticArrayInitTypeSize=92|<PrivateImplementationDetails>+__StaticArrayInitTypeSize=96|Microsoft.CodeAnalysis.CSharp.ForEachStatementInfoTests|Microsoft.CodeAnalysis.CSharp.QueryClauseInfoTests|Microsoft.CodeAnalysis.CSharp.TypeInfoTests|Microsoft.CodeAnalysis.CSharp.UnitTests.AccessTests|Microsoft.CodeAnalysis.CSharp.UnitTests.BadSymbolReference|Microsoft.CodeAnalysis.CSharp.UnitTests.BaseClassTests|Microsoft.CodeAnalysis.CSharp.UnitTests.ClsComplianceTests|Microsoft.CodeAnalysis.CSharp.UnitTests.CompilationAPITests|Microsoft.CodeAnalysis.CSharp.UnitTests.CompilationAPITests+EvolvingTestReference|Microsoft.CodeAnalysis.CSharp.UnitTests.CompilationCreationTests|Microsoft.CodeAnalysis.CSharp.UnitTests.CompilationCreationTests+DummyReferenceResolver|Microsoft.CodeAnalysis.CSharp.UnitTests.CompilationCreationTests+Resolver|Microsoft.CodeAnalysis.CSharp.UnitTests.CompilationErrorTests|Microsoft.CodeAnalysis.CSharp.UnitTests.CompletionTests|Microsoft.CodeAnalysis.CSharp.UnitTests.ConstructorDocumentationCommentTests|Microsoft.CodeAnalysis.CSharp.UnitTests.ContainerTests|Microsoft.CodeAnalysis.CSharp.UnitTests.CrefTests|Microsoft.CodeAnalysis.CSharp.UnitTests.CrossLanguageTests|Microsoft.CodeAnalysis.CSharp.UnitTests.CSharpCompilationOptionsTests|Microsoft.CodeAnalysis.CSharp.UnitTests.CSharpCompilationOptionsTests+MetadataReferenceResolverWithEquality|Microsoft.CodeAnalysis.CSharp.UnitTests.CustomModifierCopyTests|Microsoft.CodeAnalysis.CSharp.UnitTests.Declarations.SourcePlusMetadataTests|Microsoft.CodeAnalysis.CSharp.UnitTests.DeclarationTests|Microsoft.CodeAnalysis.CSharp.UnitTests.DeclarationTests+CountedSyntaxTree|Microsoft.CodeAnalysis.CSharp.UnitTests.DeclarationTests+CountedSyntaxTree+Reference|Microsoft.CodeAnalysis.CSharp.UnitTests.DelegateTests|Microsoft.CodeAnalysis.CSharp.UnitTests.DestructorDocumentationCommentTests|Microsoft.CodeAnalysis.CSharp.UnitTests.DocumentationCommentCompilerTests|Microsoft.CodeAnalysis.CSharp.UnitTests.DocumentationCommentCompilerTests+DocumentationCommentAdder|Microsoft.CodeAnalysis.CSharp.UnitTests.DocumentationCommentIDTests|Microsoft.CodeAnalysis.CSharp.UnitTests.DocumentationModeTests|Microsoft.CodeAnalysis.CSharp.UnitTests.EventDocumentationCommentTests|Microsoft.CodeAnalysis.CSharp.UnitTests.ExtendedPartialMethodsTests|Microsoft.CodeAnalysis.CSharp.UnitTests.ExternAliasTests|Microsoft.CodeAnalysis.CSharp.UnitTests.FieldDocumentationCommentTests|Microsoft.CodeAnalysis.CSharp.UnitTests.FieldTests|Microsoft.CodeAnalysis.CSharp.UnitTests.FunctionPointerTypeSymbolTests|Microsoft.CodeAnalysis.CSharp.UnitTests.FunctionPointerTypeSymbolTests+Equality|Microsoft.CodeAnalysis.CSharp.UnitTests.GetSemanticInfoTests|Microsoft.CodeAnalysis.CSharp.UnitTests.GetUnusedImportDirectivesTests|Microsoft.CodeAnalysis.CSharp.UnitTests.ImplicitClassTests|Microsoft.CodeAnalysis.CSharp.UnitTests.IncludeTests|Microsoft.CodeAnalysis.CSharp.UnitTests.IndexedProperties_BindingTests|Microsoft.CodeAnalysis.CSharp.UnitTests.IndexedTypeParameterTests|Microsoft.CodeAnalysis.CSharp.UnitTests.LoadDirectiveTests|Microsoft.CodeAnalysis.CSharp.UnitTests.LocationTests|Microsoft.CodeAnalysis.CSharp.UnitTests.LookupSymbolsInfoTests|Microsoft.CodeAnalysis.CSharp.UnitTests.MetadataMemberTests|Microsoft.CodeAnalysis.CSharp.UnitTests.MetadataTypeTests|Microsoft.CodeAnalysis.CSharp.UnitTests.MethodDocumentationCommentTests|Microsoft.CodeAnalysis.CSharp.UnitTests.MethodEqualityTests|Microsoft.CodeAnalysis.CSharp.UnitTests.MethodTests --framework net5.0 --logger "xunit;LogFilePath=F:\workspace\_work\1\s\artifacts\TestResults\Release\Microsoft.CodeAnalysis.CSharp.Symbol.UnitTests.dll.1_net5.0_x86.xml"
xUnit output log: F:\workspace\_work\1\s\artifacts\log\Release\xUnitFailure-Microsoft.CodeAnalysis.CSharp.Symbol.UnitTests.dll.1.log

CLR version 5.0.100 on Windows.

@Dotnet-GitSync-Bot Dotnet-GitSync-Bot added area-Microsoft.CSharp untriaged New issue has not been triaged by the area owner labels Dec 3, 2020
@ghost
Copy link

ghost commented Dec 3, 2020

Tagging subscribers to this area: @cston
See info in area-owners.md if you want to be subscribed.

Issue Details

Description

Roslyn CI failed with the following error:

https://dev.azure.com/dnceng/public/_build/results?buildId=906787&view=logs&j=7817cc08-46f6-529a-b675-9c2d2b016c0b&t=0fd245be-2f58-5546-b778-7c0bac845221&l=68

The active test run was aborted. Reason: Test host process crashed : Fatal error. Internal CLR error. (0x80131506)
   at System.Collections.Concurrent.ConcurrentDictionary`2[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]]..ctor(Int32, Int32, Boolean, System.Collections.Generic.IEqualityComparer`1<System.__Canon>)
   at System.Collections.Concurrent.ConcurrentDictionary`2[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]]..ctor()
   at System.RuntimeType.CreateInstanceDefaultCtor(Boolean, Boolean, Boolean, Boolean)
   at System.Activator.CreateInstance[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]]()
   at System.Threading.LazyInitializer.EnsureInitializedCore[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](System.__Canon ByRef)
   at Microsoft.CodeAnalysis.Compilation.MarkImportDirectiveAsUsed(Microsoft.CodeAnalysis.SyntaxTree, Int32)
   at Microsoft.CodeAnalysis.Compilation.MarkImportDirectiveAsUsed(Microsoft.CodeAnalysis.SyntaxNode)
   at Microsoft.CodeAnalysis.CSharp.Imports.LookupSymbolInAliases(Microsoft.CodeAnalysis.CSharp.Binder, Microsoft.CodeAnalysis.CSharp.LookupResult, System.String, Int32, Roslyn.Utilities.ConsList`1<Microsoft.CodeAnalysis.CSharp.Symbols.TypeSymbol>, Microsoft.CodeAnalysis.CSharp.LookupOptions, Boolean, System.Collections.Generic.HashSet`1<Microsoft.CodeAnalysis.DiagnosticInfo> ByRef)
   at Microsoft.CodeAnalysis.CSharp.InContainerBinder.LookupSymbolsInSingleBinder(Microsoft.CodeAnalysis.CSharp.LookupResult, System.String, Int32, Roslyn.Utilities.ConsList`1<Microsoft.CodeAnalysis.CSharp.Symbols.TypeSymbol>, Microsoft.CodeAnalysis.CSharp.LookupOptions, Microsoft.CodeAnalysis.CSharp.Binder, Boolean, System.Collections.Generic.HashSet`1<Microsoft.CodeAnalysis.DiagnosticInfo> ByRef)
   at Microsoft.CodeAnalysis.CSharp.Binder.LookupSymbolsInternal(Microsoft.CodeAnalysis.CSharp.LookupResult, System.String, Int32, Roslyn.Utilities.ConsList`1<Microsoft.CodeAnalysis.CSharp.Symbols.TypeSymbol>, Microsoft.CodeAnalysis.CSharp.LookupOptions, Boolean, System.Collections.Generic.HashSet`1<Microsoft.CodeAnalysis.DiagnosticInfo> ByRef)
   at Microsoft.CodeAnalysis.CSharp.Binder.BindNonGenericSimpleNamespaceOrTypeOrAliasSymbol(Microsoft.CodeAnalysis.CSharp.Syntax.IdentifierNameSyntax, Microsoft.CodeAnalysis.DiagnosticBag, Roslyn.Utilities.ConsList`1<Microsoft.CodeAnalysis.CSharp.Symbols.TypeSymbol>, Boolean, Microsoft.CodeAnalysis.CSharp.Symbols.NamespaceOrTypeSymbol)
   at Microsoft.CodeAnalysis.CSharp.Binder.BindNamespaceOrTypeOrAliasSymbol(Microsoft.CodeAnalysis.CSharp.Syntax.ExpressionSyntax, Microsoft.CodeAnalysis.DiagnosticBag, Roslyn.Utilities.ConsList`1<Microsoft.CodeAnalysis.CSharp.Symbols.TypeSymbol>, Boolean)
   at Microsoft.CodeAnalysis.CSharp.Binder.BindTypeOrAlias(Microsoft.CodeAnalysis.CSharp.Syntax.ExpressionSyntax, Microsoft.CodeAnalysis.DiagnosticBag, Roslyn.Utilities.ConsList`1<Microsoft.CodeAnalysis.CSharp.Symbols.TypeSymbol>, Boolean)
   at Microsoft.CodeAnalysis.CSharp.Binder.BindTypeOrAliasOrConstraintKeyword(Microsoft.CodeAnalysis.CSharp.Syntax.TypeSyntax, Microsoft.CodeAnalysis.DiagnosticBag, ConstraintContextualKeyword ByRef)
   at Microsoft.CodeAnalysis.CSharp.Binder.BindTypeParameterConstraints(Microsoft.CodeAnalysis.CSharp.Syntax.TypeParameterSyntax, Microsoft.CodeAnalysis.CSharp.Syntax.TypeParameterConstraintClauseSyntax, Boolean, Microsoft.CodeAnalysis.DiagnosticBag)
   at Microsoft.CodeAnalysis.CSharp.Binder.BindTypeParameterConstraintClauses(Microsoft.CodeAnalysis.CSharp.Symbol, System.Collections.Immutable.ImmutableArray`1<Microsoft.CodeAnalysis.CSharp.Symbols.TypeParameterSymbol>, Microsoft.CodeAnalysis.CSharp.Syntax.TypeParameterListSyntax, Microsoft.CodeAnalysis.SyntaxList`1<Microsoft.CodeAnalysis.CSharp.Syntax.TypeParameterConstraintClauseSyntax>, Microsoft.CodeAnalysis.DiagnosticBag, Boolean, Boolean)
   at Microsoft.CodeAnalysis.CSharp.Symbols.ConstraintsHelper.MakeTypeParameterConstraintKinds(Microsoft.CodeAnalysis.CSharp.Symbols.MethodSymbol, Microsoft.CodeAnalysis.CSharp.Binder, System.Collections.Immutable.ImmutableArray`1<Microsoft.CodeAnalysis.CSharp.Symbols.TypeParameterSymbol>, Microsoft.CodeAnalysis.CSharp.Syntax.TypeParameterListSyntax, Microsoft.CodeAnalysis.SyntaxList`1<Microsoft.CodeAnalysis.CSharp.Syntax.TypeParameterConstraintClauseSyntax>)
   at Microsoft.CodeAnalysis.CSharp.Symbols.SourceOrdinaryMethodSymbol.GetTypeParameterConstraintKinds()
   at Microsoft.CodeAnalysis.CSharp.Symbols.SourceOrdinaryMethodSymbol.GetTypeParameterConstraintTypes()
   at Microsoft.CodeAnalysis.CSharp.Symbols.SourceMethodTypeParameterSymbol.ResolveBounds(Roslyn.Utilities.ConsList`1<Microsoft.CodeAnalysis.CSharp.Symbols.TypeParameterSymbol>, Microsoft.CodeAnalysis.DiagnosticBag)
   at Microsoft.CodeAnalysis.CSharp.Symbols.SourceTypeParameterSymbolBase.GetBounds(Roslyn.Utilities.ConsList`1<Microsoft.CodeAnalysis.CSharp.Symbols.TypeParameterSymbol>)
   at Microsoft.CodeAnalysis.CSharp.Symbols.SourceTypeParameterSymbolBase.GetConstraintTypes(Roslyn.Utilities.ConsList`1<Microsoft.CodeAnalysis.CSharp.Symbols.TypeParameterSymbol>)
   at Microsoft.CodeAnalysis.CSharp.Symbols.TypeParameterSymbol.EnsureAllConstraintsAreResolved(System.Collections.Immutable.ImmutableArray`1<Microsoft.CodeAnalysis.CSharp.Symbols.TypeParameterSymbol>)
   at Microsoft.CodeAnalysis.CSharp.Symbols.VarianceSafety.CheckTypeParametersVarianceSafety(System.Collections.Immutable.ImmutableArray`1<Microsoft.CodeAnalysis.CSharp.Symbols.TypeParameterSymbol>, Microsoft.CodeAnalysis.CSharp.Symbols.MethodSymbol, Microsoft.CodeAnalysis.DiagnosticBag)
   at Microsoft.CodeAnalysis.CSharp.Symbols.VarianceSafety.CheckMethodVarianceSafety(Microsoft.CodeAnalysis.CSharp.Symbols.MethodSymbol, LocationProvider`1<Microsoft.CodeAnalysis.CSharp.Symbols.MethodSymbol>, Microsoft.CodeAnalysis.DiagnosticBag)
   at Microsoft.CodeAnalysis.CSharp.Symbols.VarianceSafety.CheckMethodVarianceSafety(Microsoft.CodeAnalysis.CSharp.Symbols.MethodSymbol, Microsoft.CodeAnalysis.DiagnosticBag)
   at Microsoft.CodeAnalysis.CSharp.Symbols.VarianceSafety.CheckInterfaceVarianceSafety(Microsoft.CodeAnalysis.CSharp.Symbols.NamedTypeSymbol, Microsoft.CodeAnalysis.DiagnosticBag)
   at Microsoft.CodeAnalysis.CSharp.Symbols.SourceMemberContainerTypeSymbol.GetSynthesizedExplicitImplementations(System.Threading.CancellationToken)
   at Microsoft.CodeAnalysis.CSharp.Symbols.SourceMemberContainerTypeSymbol.ForceComplete(Microsoft.CodeAnalysis.SourceLocation, System.Threading.CancellationToken)
   at Microsoft.CodeAnalysis.CSharp.Symbol.ForceCompleteMemberByLocation(Microsoft.CodeAnalysis.SourceLocation, Microsoft.CodeAnalysis.CSharp.Symbol, System.Threading.CancellationToken)
   at Roslyn.Utilities.RoslynParallel+<>c__DisplayClass1_0.<For>g__errorHandlingBody|0(Int32)
   at System.Threading.Tasks.Parallel+<>c__DisplayClass19_0`1[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].<ForWorker>b__1(System.Threading.Tasks.RangeWorker ByRef, Int32, Boolean ByRef)
   at System.Threading.Tasks.TaskReplicator+Replica.Execute()
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(System.Threading.Thread, System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread)
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
Command: test "F:\workspace\_work\1\s\artifacts\bin\Microsoft.CodeAnalysis.CSharp.Symbol.UnitTests\Release\net5.0\Microsoft.CodeAnalysis.CSharp.Symbol.UnitTests.dll" --filter <PrivateImplementationDetails>+__StaticArrayInitTypeSize=10|<PrivateImplementationDetails>+__StaticArrayInitTypeSize=100|<PrivateImplementationDetails>+__StaticArrayInitTypeSize=104|<PrivateImplementationDetails>+__StaticArrayInitTypeSize=108|<PrivateImplementationDetails>+__StaticArrayInitTypeSize=12|<PrivateImplementationDetails>+__StaticArrayInitTypeSize=120|<PrivateImplementationDetails>+__StaticArrayInitTypeSize=132|<PrivateImplementationDetails>+__StaticArrayInitTypeSize=136|<PrivateImplementationDetails>+__StaticArrayInitTypeSize=144|<PrivateImplementationDetails>+__StaticArrayInitTypeSize=16|<PrivateImplementationDetails>+__StaticArrayInitTypeSize=176|<PrivateImplementationDetails>+__StaticArrayInitTypeSize=1764|<PrivateImplementationDetails>+__StaticArrayInitTypeSize=20|<PrivateImplementationDetails>+__StaticArrayInitTypeSize=224|<PrivateImplementationDetails>+__StaticArrayInitTypeSize=24|<PrivateImplementationDetails>+__StaticArrayInitTypeSize=28|<PrivateImplementationDetails>+__StaticArrayInitTypeSize=32|<PrivateImplementationDetails>+__StaticArrayInitTypeSize=36|<PrivateImplementationDetails>+__StaticArrayInitTypeSize=40|<PrivateImplementationDetails>+__StaticArrayInitTypeSize=44|<PrivateImplementationDetails>+__StaticArrayInitTypeSize=48|<PrivateImplementationDetails>+__StaticArrayInitTypeSize=5|<PrivateImplementationDetails>+__StaticArrayInitTypeSize=52|<PrivateImplementationDetails>+__StaticArrayInitTypeSize=56|<PrivateImplementationDetails>+__StaticArrayInitTypeSize=60|<PrivateImplementationDetails>+__StaticArrayInitTypeSize=64|<PrivateImplementationDetails>+__StaticArrayInitTypeSize=68|<PrivateImplementationDetails>+__StaticArrayInitTypeSize=72|<PrivateImplementationDetails>+__StaticArrayInitTypeSize=76|<PrivateImplementationDetails>+__StaticArrayInitTypeSize=80|<PrivateImplementationDetails>+__StaticArrayInitTypeSize=820|<PrivateImplementationDetails>+__StaticArrayInitTypeSize=84|<PrivateImplementationDetails>+__StaticArrayInitTypeSize=88|<PrivateImplementationDetails>+__StaticArrayInitTypeSize=92|<PrivateImplementationDetails>+__StaticArrayInitTypeSize=96|Microsoft.CodeAnalysis.CSharp.ForEachStatementInfoTests|Microsoft.CodeAnalysis.CSharp.QueryClauseInfoTests|Microsoft.CodeAnalysis.CSharp.TypeInfoTests|Microsoft.CodeAnalysis.CSharp.UnitTests.AccessTests|Microsoft.CodeAnalysis.CSharp.UnitTests.BadSymbolReference|Microsoft.CodeAnalysis.CSharp.UnitTests.BaseClassTests|Microsoft.CodeAnalysis.CSharp.UnitTests.ClsComplianceTests|Microsoft.CodeAnalysis.CSharp.UnitTests.CompilationAPITests|Microsoft.CodeAnalysis.CSharp.UnitTests.CompilationAPITests+EvolvingTestReference|Microsoft.CodeAnalysis.CSharp.UnitTests.CompilationCreationTests|Microsoft.CodeAnalysis.CSharp.UnitTests.CompilationCreationTests+DummyReferenceResolver|Microsoft.CodeAnalysis.CSharp.UnitTests.CompilationCreationTests+Resolver|Microsoft.CodeAnalysis.CSharp.UnitTests.CompilationErrorTests|Microsoft.CodeAnalysis.CSharp.UnitTests.CompletionTests|Microsoft.CodeAnalysis.CSharp.UnitTests.ConstructorDocumentationCommentTests|Microsoft.CodeAnalysis.CSharp.UnitTests.ContainerTests|Microsoft.CodeAnalysis.CSharp.UnitTests.CrefTests|Microsoft.CodeAnalysis.CSharp.UnitTests.CrossLanguageTests|Microsoft.CodeAnalysis.CSharp.UnitTests.CSharpCompilationOptionsTests|Microsoft.CodeAnalysis.CSharp.UnitTests.CSharpCompilationOptionsTests+MetadataReferenceResolverWithEquality|Microsoft.CodeAnalysis.CSharp.UnitTests.CustomModifierCopyTests|Microsoft.CodeAnalysis.CSharp.UnitTests.Declarations.SourcePlusMetadataTests|Microsoft.CodeAnalysis.CSharp.UnitTests.DeclarationTests|Microsoft.CodeAnalysis.CSharp.UnitTests.DeclarationTests+CountedSyntaxTree|Microsoft.CodeAnalysis.CSharp.UnitTests.DeclarationTests+CountedSyntaxTree+Reference|Microsoft.CodeAnalysis.CSharp.UnitTests.DelegateTests|Microsoft.CodeAnalysis.CSharp.UnitTests.DestructorDocumentationCommentTests|Microsoft.CodeAnalysis.CSharp.UnitTests.DocumentationCommentCompilerTests|Microsoft.CodeAnalysis.CSharp.UnitTests.DocumentationCommentCompilerTests+DocumentationCommentAdder|Microsoft.CodeAnalysis.CSharp.UnitTests.DocumentationCommentIDTests|Microsoft.CodeAnalysis.CSharp.UnitTests.DocumentationModeTests|Microsoft.CodeAnalysis.CSharp.UnitTests.EventDocumentationCommentTests|Microsoft.CodeAnalysis.CSharp.UnitTests.ExtendedPartialMethodsTests|Microsoft.CodeAnalysis.CSharp.UnitTests.ExternAliasTests|Microsoft.CodeAnalysis.CSharp.UnitTests.FieldDocumentationCommentTests|Microsoft.CodeAnalysis.CSharp.UnitTests.FieldTests|Microsoft.CodeAnalysis.CSharp.UnitTests.FunctionPointerTypeSymbolTests|Microsoft.CodeAnalysis.CSharp.UnitTests.FunctionPointerTypeSymbolTests+Equality|Microsoft.CodeAnalysis.CSharp.UnitTests.GetSemanticInfoTests|Microsoft.CodeAnalysis.CSharp.UnitTests.GetUnusedImportDirectivesTests|Microsoft.CodeAnalysis.CSharp.UnitTests.ImplicitClassTests|Microsoft.CodeAnalysis.CSharp.UnitTests.IncludeTests|Microsoft.CodeAnalysis.CSharp.UnitTests.IndexedProperties_BindingTests|Microsoft.CodeAnalysis.CSharp.UnitTests.IndexedTypeParameterTests|Microsoft.CodeAnalysis.CSharp.UnitTests.LoadDirectiveTests|Microsoft.CodeAnalysis.CSharp.UnitTests.LocationTests|Microsoft.CodeAnalysis.CSharp.UnitTests.LookupSymbolsInfoTests|Microsoft.CodeAnalysis.CSharp.UnitTests.MetadataMemberTests|Microsoft.CodeAnalysis.CSharp.UnitTests.MetadataTypeTests|Microsoft.CodeAnalysis.CSharp.UnitTests.MethodDocumentationCommentTests|Microsoft.CodeAnalysis.CSharp.UnitTests.MethodEqualityTests|Microsoft.CodeAnalysis.CSharp.UnitTests.MethodTests --framework net5.0 --logger "xunit;LogFilePath=F:\workspace\_work\1\s\artifacts\TestResults\Release\Microsoft.CodeAnalysis.CSharp.Symbol.UnitTests.dll.1_net5.0_x86.xml"
xUnit output log: F:\workspace\_work\1\s\artifacts\log\Release\xUnitFailure-Microsoft.CodeAnalysis.CSharp.Symbol.UnitTests.dll.1.log

CLR version 5.0.100 on Windows.

Author: tmat
Assignees: -
Labels:

area-Microsoft.CSharp, untriaged

Milestone: -

@tmat tmat added runtime-coreclr specific to the CoreCLR runtime and removed area-Microsoft.CSharp labels Dec 3, 2020
@danmoseley
Copy link
Member

Is a dump possible @tmat? Or is there any repro?

@tmat
Copy link
Member Author

tmat commented Dec 3, 2020

@jkotas
Copy link
Member

jkotas commented Dec 3, 2020

Crash during GC:

00 000000c5`744fc078 00007ff8`2eed9fee coreclr!MethodTable::GetLoaderAllocatorObjectForGC+0x13 [F:\workspace\_work\1\s\src\coreclr\src\vm\methodtable.cpp @ 9414] 
01 000000c5`744fc080 00007ff8`2ee18299 coreclr!WKS::gc_heap::mark_object_simple+0x153cde [F:\workspace\_work\1\s\src\coreclr\src\gc\gc.cpp @ 19435] 
02 000000c5`744fc0d0 00007ff8`2ee3bcd6 coreclr!WKS::gc_heap::mark_through_cards_for_segments+0x4e9 [F:\workspace\_work\1\s\src\coreclr\src\gc\gc.cpp @ 30261] 
03 000000c5`744fc220 00007ff8`2ee39223 coreclr!WKS::gc_heap::mark_phase+0x17a6 [F:\workspace\_work\1\s\src\coreclr\src\gc\gc.cpp @ 20839] 
04 000000c5`744fd820 00007ff8`2ee38377 coreclr!WKS::gc_heap::gc1+0x1c3 [F:\workspace\_work\1\s\src\coreclr\src\gc\gc.cpp @ 16694] 
05 000000c5`744fd8f0 00007ff8`2ee04266 coreclr!WKS::gc_heap::garbage_collect+0xa67 [F:\workspace\_work\1\s\src\coreclr\src\gc\gc.cpp @ 18280] 
06 000000c5`744fd9d0 00007ff8`2ee176d5 coreclr!WKS::GCHeap::GarbageCollectGeneration+0x256 [F:\workspace\_work\1\s\src\coreclr\src\gc\gc.cpp @ 37751] 
07 000000c5`744fda60 00007ff8`2ee19ff7 coreclr!WKS::gc_heap::allocate_more_space+0x1c5 [F:\workspace\_work\1\s\src\coreclr\src\gc\gc.cpp @ 14486] 
08 000000c5`744fdae0 00007ff8`2eda81ef coreclr!WKS::GCHeap::Alloc+0x87 [F:\workspace\_work\1\s\src\coreclr\src\gc\gc.cpp @ 36745] 
09 000000c5`744fdb50 00007ff7`d2d9d38c coreclr!JIT_New+0x1ff [F:\workspace\_work\1\s\src\coreclr\src\vm\jithelpers.cpp @ 2318] 
0a 000000c5`744fddd0 00007ff7`d2252080 System_Collections_Concurrent!System.Collections.Concurrent.ConcurrentDictionary`2[[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib]]..ctor(Int32, Int32, Boolean, System.Collections.Generic.IEqualityComparer`1<System.__Canon>)+0xffffffff`a4f30cbc [/_/src/libraries/System.Collections.Concurrent/src/System/Collections/Concurrent/ConcurrentDictionary.cs @ 213] 

Due to corrupted GC heap:

0:028> !verifyheap
object 000002b88987c0b8: bad member 000002B88A0632E0 at 000002B88987C100
Last good object: 000002B88987C078.
object 000002b88987fba0: bad member 000002B88A0632E0 at 000002B88987FBD8
Last good object: 000002B88987F388.
object 000002b8898c1780: bad member 000002B88A0632E0 at 000002B8898C17C8
Last good object: 000002B8898C1740.
object 000002b8898c2908: bad member 000002B88A0632E0 at 000002B8898C2940
Last good object: 000002B8898C20F0.
object 000002b8898db890: bad member 000002B88A0632E0 at 000002B8898DB8D8
Last good object: 000002B8898DB850.
object 000002b8898dc200: bad member 000002B88A0632E0 at 000002B8898DC238
Last good object: 000002B8898DC1E8.
...

The bad reference should be array of object references (wrapped by ImmutableArray).

@jkotas jkotas added the tenet-reliability Reliability/stability related issue (stress, load problems, etc.) label Dec 3, 2020
@jaredpar
Copy link
Member

jaredpar commented Dec 3, 2020

This is hitting Roslyn multiple times a day right now. If you need more dumps to narrow down the problem you can follow the results from this search for more logs

https://runfo.azurewebsites.net/search/build-logs/?bq=definition%3Aroslyn-ci++started%3A%7E3&lq=logName%3ATest_Windows_CoreClr+text%3A%22Internal+CLR+error%22

@jkotas
Copy link
Member

jkotas commented Dec 4, 2020

This crash has a very regular pattern. It is always bad reference to ImmutableArray from Roslyn.Utilities.ObjectBinderSnapshot. It suggests that it is a bad GC info produced by the JIT.

@JulieLeeMSFT Could you please find somebody on your team to investigate? This is hitting Roslyn CI multiple times a day right now.

@jkotas jkotas added area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI and removed area-VM-coreclr labels Dec 4, 2020
@jkotas
Copy link
Member

jkotas commented Dec 4, 2020

@jaredpar It would help us a lot to get a crash dump with these stress log settings:

COMPlus_StressLog=1
COMPlus_LogLevel=6
COMPlus_LogFacility=0x00080001
COMPlus_StressLogSize=2000000
COMPlus_TotalStressLogSize=40000000

Any chance you can enable these stress log settings in the Roslyn CI temporarily to get a dump with stresslog? We should be able to root cause the problem immediately from a dump with stresslog.

The stresslog will make the runtime run a bit slower and consume more memory, but it should not be prohibitive (like 20% slower).

@jaredpar
Copy link
Member

jaredpar commented Dec 4, 2020

Yes we can add those. Our CoreCLR jobs are not the long pole in CI so the extra 20% here will be fine.

jaredpar added a commit to jaredpar/roslyn that referenced this issue Dec 4, 2020
This sets environment variables that will help make the dumps for a CLR
bug more actionable by the runtime team.

dotnet/runtime#45557
@JulieLeeMSFT
Copy link
Member

@BruceForstall Please look into this.

@JulieLeeMSFT JulieLeeMSFT removed the untriaged New issue has not been triaged by the area owner label Dec 6, 2020
@JulieLeeMSFT JulieLeeMSFT added this to the 6.0.0 milestone Dec 6, 2020
@jkotas
Copy link
Member

jkotas commented Dec 7, 2020

This is bad GC info generated for Roslyn.Utilities.ObjectBinder.GetSnapshot() method: https://github.com/dotnet/roslyn/blob/cf55f3a58e47298426fa971d3bd9d8857c746c65/src/Compilers/Core/Portable/Serialization/ObjectBinder.cs#L45

@jkotas
Copy link
Member

jkotas commented Dec 7, 2020

Relevant fragment of the stress log:

 e90 107.640837900 : `GC`GCROOTS`             GC Root 00000041137FB728 RELOCATED 00000200D2BCFD78 -> 00000200D23D6148  MT = 00007FFEA7A97EE0 (System.Collections.Generic.Dictionary`2[[System.Type, System.Private.CoreLib],[System.Int32, System.Private.CoreLib]])
 e90 107.640837400 : `GC`GCROOTS`             GC Root 00000041137FB778 RELOCATED 00000200D2BCFCC0 -> 00000200D23D6090  MT = 0000000000000000 (BAD MethodTable)
 e90 107.640836500 : `GCROOTS`            Scanning Frameless method 00007FFEA635EE98 (Roslyn.Utilities.ObjectBinder.GetSnapshot()) ControlPC = 00007FFEA86D121D
 e90 107.640834800 : `GCROOTS`            Scanning Frameless method 00007FFEA7A582A8 (System.Collections.Immutable.ImmutableArray.CreateRange[[System.__Canon, System.Private.CoreLib]](System.Collections.Generic.IEnumerable`1<System.__Canon>)) ControlPC = 00007FFEA86B4D7E
 e90 107.640833000 : `GCROOTS`            Scanning Frameless method 00007FFEA7AD00B8 (System.Collections.Immutable.ImmutableExtensions.ToArray[[System.__Canon, System.Private.CoreLib]](System.Collections.Generic.IEnumerable`1<System.__Canon>, Int32)) ControlPC = 00007FFEA86DB3B6
 e90 107.640830900 : `GCROOTS`            Scanning ExplicitFrame 00000041137FB5A8 AssocMethod = 0000000000000000 frameVTable = 00007FFF04FD2A70 (coreclr!HelperMethodFrame::`vftable')
 e90 107.640830500 : `GC`GCROOTS`         } Starting scan of Thread 00000200F0ACE0E0 ID = 4

@jkotas
Copy link
Member

jkotas commented Dec 7, 2020

Method IL:

0:005> !DumpIL 00007ffea635ee98
ilAddr is 00000200F41E8914 pImport is 0000014D759C0FF0
ilAddr = 00000200F41E8914
IL_0000: ldsfld Roslyn.Utilities.ObjectBinder::s_gate
IL_0005: stloc.0 
IL_0006: ldc.i4.0 
IL_0007: stloc.1 
.try
{
  IL_0008: ldloc.0 
  IL_0009: ldloca.s VAR OR ARG 1
  IL_000b: call void System.Threading.Monitor::Enter(object,bool&)
  IL_0010: ldsflda Roslyn.Utilities.ObjectBinder::s_lastSnapshot
  IL_0015: call bool ()
  IL_001a: brtrue.s IL_003a
  IL_001c: ldsfld Roslyn.Utilities.ObjectBinder::s_typeToIndex
  IL_0021: ldsfld Roslyn.Utilities.ObjectBinder::s_types
  IL_0026: ldsfld Roslyn.Utilities.ObjectBinder::s_typeReaders
  IL_002b: newobj void Roslyn.Utilities.ObjectBinderSnapshot::.ctor(class [System.Collections]System.Collections.Generic.Dictionary`2<class [System.Runtime]System.Type,int32>,class [System.Collections]System.Collections.Generic.List`1<class [System.Runtime]System.Type>,class [System.Collections]System.Collections.Generic.List`1<class [System.Runtime]System.Func`2<class [*]Roslyn.Utilities.ObjectReader,class [*]Roslyn.Utilities.IObjectWritable>>)
  IL_0030: newobj void (!0)
  IL_0035: stsfld Roslyn.Utilities.ObjectBinder::s_lastSnapshot
  IL_003a: ldsflda Roslyn.Utilities.ObjectBinder::s_lastSnapshot
  IL_003f: call !0 ()
  IL_0044: stloc.2 
  IL_0045: leave.s IL_0051
} // end .try
.finally
{
  IL_0047: ldloc.1 
  IL_0048: brfalse.s IL_0050
  IL_004a: ldloc.0 
  IL_004b: call void System.Threading.Monitor::Exit(object)
  IL_0050: endfinally 
} // end .finally
IL_0051: ldloc.2 
IL_0052: ret 

@jkotas
Copy link
Member

jkotas commented Dec 7, 2020

Method disassembly with GC Info:

0:005> !U -gcinfo 00007ffea86d1140  
Normal JIT generated code
Roslyn.Utilities.ObjectBinder.GetSnapshot()
ilAddr is 00000200F41E8914 pImport is 0000014D759C1990
Begin 00007FFEA86D1140, size 1d0
Prolog size: 0
Security object: <none>
GS cookie: <none>
PSPSym: initial.sp+20
Generics inst context: <none>
PSP slot: caller.sp+20
GenericInst slot: <none>
Varargs: 0
Frame pointer: rbp
Wants Report Only Leaf: 1
Size of parameter area: 20
Return Kind: Scalar
Code size: 1d0
Untracked: +rbp-38 +rbp-40 +rbp-48
>>> 00007ffe`a86d1140 55              push    rbp
00007ffe`a86d1141 4157            push    r15
00007ffe`a86d1143 4156            push    r14
00007ffe`a86d1145 57              push    rdi
00007ffe`a86d1146 56              push    rsi
00007ffe`a86d1147 53              push    rbx
00007ffe`a86d1148 4883ec68        sub     rsp,68h
00007ffe`a86d114c 488dac2490000000 lea     rbp,[rsp+90h]
00007ffe`a86d1154 c5d857e4        vxorps  xmm4,xmm4,xmm4
00007ffe`a86d1158 c5f97f65b0      vmovdqa xmmword ptr [rbp-50h],xmm4
00007ffe`a86d115d c5f97f65c0      vmovdqa xmmword ptr [rbp-40h],xmm4
00007ffe`a86d1162 33c0            xor     eax,eax
00007ffe`a86d1164 488945d0        mov     qword ptr [rbp-30h],rax
00007ffe`a86d1168 48896590        mov     qword ptr [rbp-70h],rsp
00007ffe`a86d116c 488bd9          mov     rbx,rcx
0000002f interruptible
0000002f +rbx(interior)
00007ffe`a86d116f 48ba184d67de00020000 mov rdx,200DE674D18h
00007ffe`a86d1179 488b12          mov     rdx,qword ptr [rdx]
0000003c +rdx
00007ffe`a86d117c 488955a8        mov     qword ptr [rbp-58h],rdx
00000040 -rdx +rbp-58
00007ffe`a86d1180 807dd000        cmp     byte ptr [rbp-30h],0
00007ffe`a86d1184 0f8505010000    jne     Microsoft_CodeAnalysis!Roslyn.Utilities.ObjectBinder.GetSnapshot()+0x11f175f (00007ffe`a86d128f)
00007ffe`a86d118a 488d55d0        lea     rdx,[rbp-30h]
00007ffe`a86d118e 488b4da8        mov     rcx,qword ptr [rbp-58h]
00000052 +rcx
00007ffe`a86d1192 e8296d555c      call    coreclr!JIT_MonReliableEnter_Portable (00007fff`04c27ec0)
00000057 -rcx
00007ffe`a86d1197 48b9104d67de00020000 mov rcx,200DE674D10h
00007ffe`a86d11a1 488b09          mov     rcx,qword ptr [rcx]
00000064 +rcx
00007ffe`a86d11a4 80790800        cmp     byte ptr [rcx+8],0
00007ffe`a86d11a8 0f85b4000000    jne     Microsoft_CodeAnalysis!Roslyn.Utilities.ObjectBinder.GetSnapshot()+0x11f1732 (00007ffe`a86d1262)
0000006e -rcx
00007ffe`a86d11ae 48b9204d67de00020000 mov rcx,200DE674D20h
00007ffe`a86d11b8 488b31          mov     rsi,qword ptr [rcx]
0000007b +rsi
00007ffe`a86d11bb 48b9284d67de00020000 mov rcx,200DE674D28h
00007ffe`a86d11c5 488b39          mov     rdi,qword ptr [rcx]
00000088 +rdi
00007ffe`a86d11c8 48b9304d67de00020000 mov rcx,200DE674D30h
00007ffe`a86d11d2 4c8b31          mov     r14,qword ptr [rcx]
00000095 +r14
00007ffe`a86d11d5 48b9e07ea9a7fe7f0000 mov rcx,7FFEA7A97EE0h (MT: System.Collections.Generic.Dictionary`2[[System.Type, System.Private.CoreLib],[System.Int32, System.Private.CoreLib]])
00007ffe`a86d11df e86c98625c      call    coreclr!JIT_TrialAllocSFastMP_InlineGetThread (00007fff`04cfaa50)
000000a4 +rax
00007ffe`a86d11e4 4c8bf8          mov     r15,rax
000000a7 +r15
00007ffe`a86d11e7 498bcf          mov     rcx,r15
000000aa +rcx
00007ffe`a86d11ea 488bd6          mov     rdx,rsi
000000ad +rdx
00007ffe`a86d11ed 4533c0          xor     r8d,r8d
000000b0 +r8
00007ffe`a86d11f0 e8f31cd5fc      call    CLRStub[MethodDescPrestub]@7ffea5422ee8 (00007ffe`a5422ee8) (System.Collections.Generic.Dictionary`2[[System.__Canon, System.Private.CoreLib],[System.Int32, System.Private.CoreLib]]..ctor(System.Collections.Generic.IDictionary`2<System.__Canon,Int32>, System.Collections.Generic.IEqualityComparer`1<System.__Canon>), mdToken: 00000000060060D3)
000000b5 -r8 -rsi -rdx -rcx -rax
00007ffe`a86d11f5 488bd7          mov     rdx,rdi
000000b8 +rdx
00007ffe`a86d11f8 48b908fdaba7fe7f0000 mov rcx,7FFEA7ABFD08h (MD: System.Collections.Immutable.ImmutableArray.ToImmutableArray[[System.Type, System.Private.CoreLib]](System.Collections.Generic.IEnumerable`1<System.Type>))
00007ffe`a86d1202 e829cee0fe      call    CLRStub[MethodDescPrestub]@7ffea74de030 (00007ffe`a74de030) (System.Collections.Immutable.ImmutableArray.ToImmutableArray[[System.__Canon, System.Private.CoreLib]](System.Collections.Generic.IEnumerable`1<System.__Canon>), mdToken: 0000000006000155)
000000c7 -rdi -rdx +rax
00007ffe`a86d1207 488945a0        mov     qword ptr [rbp-60h],rax
000000cb +rbp-60
00007ffe`a86d120b 498bd6          mov     rdx,r14
000000ce +rdx
00007ffe`a86d120e 48b980fdaba7fe7f0000 mov rcx,7FFEA7ABFD80h (MD: System.Collections.Immutable.ImmutableArray.ToImmutableArray[[System.Func`2[[Roslyn.Utilities.ObjectReader, Microsoft.CodeAnalysis],[Roslyn.Utilities.IObjectWritable, Microsoft.CodeAnalysis]], System.Private.CoreLib]](System.Collections.Generic.IEnumerable`1<System.Func`2<Roslyn.Utilities.ObjectReader,Roslyn.Utilities.IObjectWritable>>))
000000d8 -rbp-60
00007ffe`a86d1218 e813cee0fe      call    CLRStub[MethodDescPrestub]@7ffea74de030 (00007ffe`a74de030) (System.Collections.Immutable.ImmutableArray.ToImmutableArray[[System.__Canon, System.Private.CoreLib]](System.Collections.Generic.IEnumerable`1<System.__Canon>), mdToken: 0000000006000155)
000000dd -r14 -rdx
00007ffe`a86d121d 48894598        mov     qword ptr [rbp-68h],rax
000000e1 +rbp-68
00007ffe`a86d1221 488b45a0        mov     rax,qword ptr [rbp-60h]
00007ffe`a86d1225 488b5598        mov     rdx,qword ptr [rbp-68h]
000000e9 +rdx
00007ffe`a86d1229 488d4db8        lea     rcx,[rbp-48h]
000000ed +rcx(interior)
00007ffe`a86d122d 4c8939          mov     qword ptr [rcx],r15
00007ffe`a86d1230 48894108        mov     qword ptr [rcx+8],rax
00007ffe`a86d1234 48895110        mov     qword ptr [rcx+10h],rdx
00007ffe`a86d1238 c645b001        mov     byte ptr [rbp-50h],1
00007ffe`a86d123c 48b8104d67de00020000 mov rax,200DE674D10h
00000106 -rax
00007ffe`a86d1246 488b38          mov     rdi,qword ptr [rax]
00000109 +rdi
00007ffe`a86d1249 4883c708        add     rdi,8
0000010d -rdi +rdi(interior)
00007ffe`a86d124d 488d75b0        lea     rsi,[rbp-50h]
00000111 +rsi(interior)
00007ffe`a86d1251 48a5            movs    qword ptr [rdi],qword ptr [rsi]
00000113 -rbp-68
00007ffe`a86d1253 e8b894625c      call    coreclr!JIT_ByRefWriteBarrier (00007fff`04cfa710)
00000118 -r15 -rdx -rcx(interior)
00007ffe`a86d1258 e8b394625c      call    coreclr!JIT_ByRefWriteBarrier (00007fff`04cfa710)
00007ffe`a86d125d e8ae94625c      call    coreclr!JIT_ByRefWriteBarrier (00007fff`04cfa710)
00000122 -rdi(interior) -rsi(interior)
00007ffe`a86d1262 48b9104d67de00020000 mov rcx,200DE674D10h
00007ffe`a86d126c 488b09          mov     rcx,qword ptr [rcx]
0000012f +rcx
00007ffe`a86d126f 4883c108        add     rcx,8
00000133 -rcx +rcx(interior)
00007ffe`a86d1273 803900          cmp     byte ptr [rcx],0
00007ffe`a86d1276 7411            je      Microsoft_CodeAnalysis!Roslyn.Utilities.ObjectBinder.GetSnapshot()+0x11f1759 (00007ffe`a86d1289)
00007ffe`a86d1278 4883c108        add     rcx,8
00007ffe`a86d127c 488b31          mov     rsi,qword ptr [rcx]
0000013f +rsi
00007ffe`a86d127f 488b7908        mov     rdi,qword ptr [rcx+8]
00000143 +rdi
00007ffe`a86d1283 4c8b7110        mov     r14,qword ptr [rcx+10h]
00000147 +r14
00007ffe`a86d1287 eb0c            jmp     Microsoft_CodeAnalysis!Roslyn.Utilities.ObjectBinder.GetSnapshot()+0x11f1765 (00007ffe`a86d1295)
00000149 -r14 -rdi -rsi -rbx(interior) -rcx(interior)
00007ffe`a86d1289 e882f6acfc      call    CLRStub[MethodDescPrestub]@7ffea51a0910 (00007ffe`a51a0910) (System.ThrowHelper.ThrowInvalidOperationException_InvalidOperation_NoValue(), mdToken: 000000000600183D)
00007ffe`a86d128e cc              int     3
00007ffe`a86d128f e81456aafc      call    CLRStub[MethodDescPrestub]@7ffea51768a8 (00007ffe`a51768a8) (System.Threading.Monitor.ThrowLockTakenException(), mdToken: 000000000600256B)
00007ffe`a86d1294 cc              int     3
00000155 +r14 +rdi +rsi +rbx(interior)
00007ffe`a86d1295 807dd000        cmp     byte ptr [rbp-30h],0
00007ffe`a86d1299 7409            je      Microsoft_CodeAnalysis!Roslyn.Utilities.ObjectBinder.GetSnapshot()+0x11f1774 (00007ffe`a86d12a4)
00007ffe`a86d129b 488b4da8        mov     rcx,qword ptr [rbp-58h]
0000015f -rbp-58 +rcx
00007ffe`a86d129f e81c6b555c      call    coreclr!JIT_MonExit_Portable (00007fff`04c27dc0)
00000164 -rcx
00007ffe`a86d12a4 488bcb          mov     rcx,rbx
00000167 +rcx(interior)
00007ffe`a86d12a7 488bd6          mov     rdx,rsi
0000016a +rdx
00007ffe`a86d12aa e88193625c      call    coreclr!JIT_CheckedWriteBarrier (00007fff`04cfa630)
0000016f -rsi -rdx -rcx(interior)
00007ffe`a86d12af 488d4b08        lea     rcx,[rbx+8]
00000173 +rcx(interior)
00007ffe`a86d12b3 488bd7          mov     rdx,rdi
00000176 +rdx
00007ffe`a86d12b6 e87593625c      call    coreclr!JIT_CheckedWriteBarrier (00007fff`04cfa630)
0000017b -rdi -rdx -rcx(interior)
00007ffe`a86d12bb 488d4b10        lea     rcx,[rbx+10h]
0000017f +rcx(interior)
00007ffe`a86d12bf 498bd6          mov     rdx,r14
00000182 +rdx
00007ffe`a86d12c2 e86993625c      call    coreclr!JIT_CheckedWriteBarrier (00007fff`04cfa630)
00000187 -r14 -rdx -rcx(interior)
00007ffe`a86d12c7 488bc3          mov     rax,rbx
0000018a not interruptible
0000018a -rbx(interior)
00007ffe`a86d12ca 488d65d8        lea     rsp,[rbp-28h]
00007ffe`a86d12ce 5b              pop     rbx
00007ffe`a86d12cf 5e              pop     rsi
00007ffe`a86d12d0 5f              pop     rdi
00007ffe`a86d12d1 415e            pop     r14
00007ffe`a86d12d3 415f            pop     r15
00007ffe`a86d12d5 5d              pop     rbp
00007ffe`a86d12d6 c3              ret
00007ffe`a86d12d7 55              push    rbp
00007ffe`a86d12d8 4157            push    r15
00007ffe`a86d12da 4156            push    r14
00007ffe`a86d12dc 57              push    rdi
00007ffe`a86d12dd 56              push    rsi
00007ffe`a86d12de 53              push    rbx
00007ffe`a86d12df 4883ec28        sub     rsp,28h
00007ffe`a86d12e3 488b6920        mov     rbp,qword ptr [rcx+20h]
00007ffe`a86d12e7 48896c2420      mov     qword ptr [rsp+20h],rbp
00007ffe`a86d12ec 488dad90000000  lea     rbp,[rbp+90h]
000001b3 interruptible
000001b3 +rbp-58
00007ffe`a86d12f3 807dd000        cmp     byte ptr [rbp-30h],0
00007ffe`a86d12f7 7409            je      Microsoft_CodeAnalysis!Roslyn.Utilities.ObjectBinder.GetSnapshot()+0x11f17d2 (00007ffe`a86d1302)
00007ffe`a86d12f9 488b4da8        mov     rcx,qword ptr [rbp-58h]
000001bd -rbp-58 +rcx
00007ffe`a86d12fd e8be6a555c      call    coreclr!JIT_MonExit_Portable (00007fff`04c27dc0)
000001c2 -rcx
00007ffe`a86d1302 90              nop
000001c3 not interruptible
00007ffe`a86d1303 4883c428        add     rsp,28h
00007ffe`a86d1307 5b              pop     rbx
00007ffe`a86d1308 5e              pop     rsi
00007ffe`a86d1309 5f              pop     rdi
00007ffe`a86d130a 415e            pop     r14
00007ffe`a86d130c 415f            pop     r15
00007ffe`a86d130e 5d              pop     rbp
00007ffe`a86d130f c3              ret

@jkotas
Copy link
Member

jkotas commented Dec 7, 2020

The bug:

00007ffe`a86d1202 e829cee0fe      call    CLRStub[MethodDescPrestub]@7ffea74de030 (00007ffe`a74de030) (System.Collections.Immutable.ImmutableArray.ToImmutableArray[[System.__Canon, System.Private.CoreLib]](System.Collections.Generic.IEnumerable`1<System.__Canon>), mdToken: 0000000006000155)
000000c7 -rdi -rdx +rax
00007ffe`a86d1207 488945a0        mov     qword ptr [rbp-60h],rax <--- We have stored GC reference at rbp-60 here...
000000cb +rbp-60
00007ffe`a86d120b 498bd6          mov     rdx,r14
000000ce +rdx
00007ffe`a86d120e 48b980fdaba7fe7f0000 mov rcx,7FFEA7ABFD80h (MD: System.Collections.Immutable.ImmutableArray.ToImmutableArray[[System.Func`2[[Roslyn.Utilities.ObjectReader, Microsoft.CodeAnalysis],[Roslyn.Utilities.IObjectWritable, Microsoft.CodeAnalysis]], System.Private.CoreLib]](System.Collections.Generic.IEnumerable`1<System.Func`2<Roslyn.Utilities.ObjectReader,Roslyn.Utilities.IObjectWritable>>))
000000d8 -rbp-60 <--- This is the bug. We are stopping reporting for rbp-60 here.
00007ffe`a86d1218 e813cee0fe      call    CLRStub[MethodDescPrestub]@7ffea74de030 (00007ffe`a74de030) (System.Collections.Immutable.ImmutableArray.ToImmutableArray[[System.__Canon, System.Private.CoreLib]](System.Collections.Generic.IEnumerable`1<System.__Canon>), mdToken: 0000000006000155)
000000dd -r14 -rdx
00007ffe`a86d121d 48894598        mov     qword ptr [rbp-68h],rax  <--- this is the IP where GC happened
000000e1 +rbp-68
00007ffe`a86d1221 488b45a0        mov     rax,qword ptr [rbp-60h] <--- ... and using rbp-60 here unprotected!
00007ffe`a86d1225 488b5598        mov     rdx,qword ptr [rbp-68h]

@ghost ghost added the in-pr There is an active PR which will close this issue when it is merged label Dec 11, 2020
@jaredpar
Copy link
Member

I agree, I don't see a much better way to verify this. Once this change makes it to the SDK (and hence there is a drop available) I can open a PR that uses that drop and just script AzDO to rebuild that PR for a few days.

Another option is I could just run the test in a loop on my machine for a day. That would be a pretty good approximation of our CI system. If there is a dotnet runtime build you can share out I could do that.

Note: when running this locally should I use the environment variables Jan suggested earlier? Basically do they make it more likely to catch the problem or does it just add more info when a problem happens?

@jkotas
Copy link
Member

jkotas commented Dec 11, 2020

Basically do they make it more likely to catch the problem or does it just add more info when a problem happens?

These env variables just add more info. These env variables should not affect the chance of hitting the problem.

mmitche pushed a commit that referenced this issue Dec 11, 2020
…45947)

Backport of (part of) #45818 to release/5.0

Fixes: #45557

Reported by Roslyn. Bad GC info can lead to an unexplained crash that can't easily be found or worked around.

Manual, new unit test, CLR outerloop, SuperPMI asm diffs.

Low
@ghost ghost removed the in-pr There is an active PR which will close this issue when it is merged label Dec 11, 2020
@BruceForstall
Copy link
Member

fyi, it looks like the fix should make it into 5.0.2.

@ghost ghost added the in-pr There is an active PR which will close this issue when it is merged label Dec 15, 2020
BruceForstall added a commit to BruceForstall/runtime that referenced this issue Dec 16, 2020
@ghost ghost removed the in-pr There is an active PR which will close this issue when it is merged label Dec 18, 2020
BruceForstall added a commit to BruceForstall/runtime that referenced this issue Dec 18, 2020
@ghost ghost added the in-pr There is an active PR which will close this issue when it is merged label Dec 18, 2020
@BruceForstall
Copy link
Member

@jaredpar Does this Roslyn scenario ever run on arm32 or arm64?

@jaredpar
Copy link
Member

Yes: This is a pretty core method in the compiler and we do have customers using the SDK on those platforms.

@BruceForstall
Copy link
Member

Yes: This is a pretty core method in the compiler and we do have customers using the SDK on those platforms.

Thanks for the confirmation. This helps our case to port the arm32/arm64 fix to .NET 5 servicing.

Anipik pushed a commit that referenced this issue Jan 14, 2021
@ghost ghost removed the in-pr There is an active PR which will close this issue when it is merged label Jan 14, 2021
jaredpar added a commit to jaredpar/roslyn that referenced this issue Jan 22, 2021
The associated runtime version contains the fix for a CLR crash that we
are hitting in our infrastructure. Moving should help us increase
stability here.

dotnet/runtime#45557
jaredpar added a commit to dotnet/roslyn that referenced this issue Jan 22, 2021
The associated runtime version contains the fix for a CLR crash that we
are hitting in our infrastructure. Moving should help us increase
stability here.

dotnet/runtime#45557
@ghost ghost locked as resolved and limited conversation to collaborators Feb 13, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI bug runtime-coreclr specific to the CoreCLR runtime tenet-reliability Reliability/stability related issue (stress, load problems, etc.)
Projects
Archived in project
7 participants