Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ILLinker NRE on Windows #69536

Closed
radical opened this issue May 19, 2022 · 24 comments
Closed

ILLinker NRE on Windows #69536

radical opened this issue May 19, 2022 · 24 comments
Assignees
Labels
area-VM-coreclr blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' linkable-framework Issues associated with delivering a linker friendly framework

Comments

@radical
Copy link
Member

radical commented May 19, 2022

Build:

ILLink : error IL1012: IL Trimmer has encountered an unexpected error. Please report the issue at https://github.com/dotnet/linker/issues [D:\a\_work\1\s\src\coreclr\System.Private.CoreLib\System.Private.CoreLib.csproj]
##[error]ILLink(0,0): error IL1012: (NETCORE_ENGINEERING_TELEMETRY=Build) IL Trimmer has encountered an unexpected error. Please report the issue at https://github.com/dotnet/linker/issues
  Fatal error in IL Linker
  Unhandled exception. Mono.Linker.InternalErrorException: Step 'OutputStep' failed when processing assembly 'System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e'.
   ---> Mono.Linker.LinkerFatalErrorException: ILLink: error IL1011: Failed to write 'D:\a\_work\1\s\artifacts\obj\coreclr\System.Private.CoreLib\x86\Debug\System.Private.CoreLib.dll'.
   ---> System.NullReferenceException: Object reference not set to an instance of an object.
     at Mono.Cecil.PE.ByteBufferEqualityComparer.Equals(ByteBuffer x, ByteBuffer y)
     at System.Collections.Generic.Dictionary`2.FindValue(TKey key)
     at System.Collections.Generic.Dictionary`2.TryGetValue(TKey key, TValue& value)
     at Mono.Cecil.Metadata.BlobHeapBuffer.GetBlobIndex(ByteBuffer blob)
     at Mono.Cecil.MetadataBuilder.GetBlobIndex(ByteBuffer blob)
     at Mono.Cecil.MetadataBuilder.GetBlobIndex(Byte[] blob)
     at Mono.Cecil.MetadataBuilder.AddCustomDebugInformations(ICustomDebugInformationProvider provider)
     at Mono.Cecil.MetadataBuilder.AddMethodDebugInformation(MethodDebugInformation method_info)
     at Mono.Cecil.Cil.PortablePdbWriter.Write(MethodDebugInformation info)
     at Mono.Cecil.Cil.CodeWriter.WriteResolvedMethodBody(MethodDefinition method)
     at Mono.Cecil.Cil.CodeWriter.WriteMethodBody(MethodDefinition method)
     at Mono.Cecil.MetadataBuilder.AddMethod(MethodDefinition method)
     at Mono.Cecil.MetadataBuilder.AddMethods(TypeDefinition type)
     at Mono.Cecil.MetadataBuilder.AddType(TypeDefinition type)
     at Mono.Cecil.MetadataBuilder.AddTypes()
     at Mono.Cecil.MetadataBuilder.BuildTypes()
     at Mono.Cecil.MetadataBuilder.BuildModule()
     at Mono.Cecil.MetadataBuilder.BuildMetadata()
     at Mono.Cecil.ModuleWriter.<>c.<BuildMetadata>b__2_0(MetadataBuilder builder, MetadataReader _)
     at Mono.Cecil.ModuleDefinition.Read[TItem,TRet](TItem item, Func`3 read)
     at Mono.Cecil.ModuleWriter.BuildMetadata(ModuleDefinition module, MetadataBuilder metadata)
     at Mono.Cecil.ModuleWriter.Write(ModuleDefinition module, Disposable`1 stream, WriterParameters parameters)
     at Mono.Cecil.ModuleWriter.WriteModule(ModuleDefinition module, Disposable`1 stream, WriterParameters parameters)
     at Mono.Cecil.ModuleDefinition.Write(String fileName, WriterParameters parameters)
     at Mono.Cecil.AssemblyDefinition.Write(String fileName, WriterParameters parameters)
     at Mono.Linker.Steps.OutputStep.WriteAssembly(AssemblyDefinition assembly, String directory, WriterParameters writerParameters)
     --- End of inner exception stack trace ---
     at Mono.Linker.Steps.OutputStep.WriteAssembly(AssemblyDefinition assembly, String directory, WriterParameters writerParameters)
     at Mono.Linker.Steps.OutputStep.WriteAssembly(AssemblyDefinition assembly, String directory)
     at Mono.Linker.Steps.OutputStep.OutputAssembly(AssemblyDefinition assembly)
     at Mono.Linker.Steps.OutputStep.ProcessAssembly(AssemblyDefinition assembly)
     at Mono.Linker.Steps.BaseStep.Process(LinkContext context)
     --- End of inner exception stack trace ---
     at Mono.Linker.Steps.BaseStep.Process(LinkContext context)
     at Mono.Linker.Pipeline.ProcessStep(LinkContext context, IStep step)
     at Mono.Linker.Pipeline.Process(LinkContext context)
     at Mono.Linker.Driver.Run(ILogger customLogger, Boolean throwOnFatalLinkerException)
     at Mono.Linker.Driver.Main(String[] args)
Build Kind Start Time
1813081 Rolling 2022-08-06
1814339 Rolling 2022-08-06
@radical radical added blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' linkable-framework Issues associated with delivering a linker friendly framework labels May 19, 2022
@ghost ghost added the untriaged New issue has not been triaged by the area owner label May 19, 2022
@ghost
Copy link

ghost commented May 19, 2022

Tagging subscribers to 'linkable-framework': @eerhardt, @vitek-karas, @LakshanF, @sbomer, @joperezr
See info in area-owners.md if you want to be subscribed.

Issue Details

Build:

ILLink : error IL1012: IL Trimmer has encountered an unexpected error. Please report the issue at https://github.com/dotnet/linker/issues [D:\a\_work\1\s\src\coreclr\System.Private.CoreLib\System.Private.CoreLib.csproj]
##[error]ILLink(0,0): error IL1012: (NETCORE_ENGINEERING_TELEMETRY=Build) IL Trimmer has encountered an unexpected error. Please report the issue at https://github.com/dotnet/linker/issues
  Fatal error in IL Linker
  Unhandled exception. Mono.Linker.InternalErrorException: Step 'OutputStep' failed when processing assembly 'System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e'.
   ---> Mono.Linker.LinkerFatalErrorException: ILLink: error IL1011: Failed to write 'D:\a\_work\1\s\artifacts\obj\coreclr\System.Private.CoreLib\x86\Debug\System.Private.CoreLib.dll'.
   ---> System.NullReferenceException: Object reference not set to an instance of an object.
     at Mono.Cecil.PE.ByteBufferEqualityComparer.Equals(ByteBuffer x, ByteBuffer y)
     at System.Collections.Generic.Dictionary`2.FindValue(TKey key)
     at System.Collections.Generic.Dictionary`2.TryGetValue(TKey key, TValue& value)
     at Mono.Cecil.Metadata.BlobHeapBuffer.GetBlobIndex(ByteBuffer blob)
     at Mono.Cecil.MetadataBuilder.GetBlobIndex(ByteBuffer blob)
     at Mono.Cecil.MetadataBuilder.GetBlobIndex(Byte[] blob)
     at Mono.Cecil.MetadataBuilder.AddCustomDebugInformations(ICustomDebugInformationProvider provider)
     at Mono.Cecil.MetadataBuilder.AddMethodDebugInformation(MethodDebugInformation method_info)
     at Mono.Cecil.Cil.PortablePdbWriter.Write(MethodDebugInformation info)
     at Mono.Cecil.Cil.CodeWriter.WriteResolvedMethodBody(MethodDefinition method)
     at Mono.Cecil.Cil.CodeWriter.WriteMethodBody(MethodDefinition method)
     at Mono.Cecil.MetadataBuilder.AddMethod(MethodDefinition method)
     at Mono.Cecil.MetadataBuilder.AddMethods(TypeDefinition type)
     at Mono.Cecil.MetadataBuilder.AddType(TypeDefinition type)
     at Mono.Cecil.MetadataBuilder.AddTypes()
     at Mono.Cecil.MetadataBuilder.BuildTypes()
     at Mono.Cecil.MetadataBuilder.BuildModule()
     at Mono.Cecil.MetadataBuilder.BuildMetadata()
     at Mono.Cecil.ModuleWriter.<>c.<BuildMetadata>b__2_0(MetadataBuilder builder, MetadataReader _)
     at Mono.Cecil.ModuleDefinition.Read[TItem,TRet](TItem item, Func`3 read)
     at Mono.Cecil.ModuleWriter.BuildMetadata(ModuleDefinition module, MetadataBuilder metadata)
     at Mono.Cecil.ModuleWriter.Write(ModuleDefinition module, Disposable`1 stream, WriterParameters parameters)
     at Mono.Cecil.ModuleWriter.WriteModule(ModuleDefinition module, Disposable`1 stream, WriterParameters parameters)
     at Mono.Cecil.ModuleDefinition.Write(String fileName, WriterParameters parameters)
     at Mono.Cecil.AssemblyDefinition.Write(String fileName, WriterParameters parameters)
     at Mono.Linker.Steps.OutputStep.WriteAssembly(AssemblyDefinition assembly, String directory, WriterParameters writerParameters)
     --- End of inner exception stack trace ---
     at Mono.Linker.Steps.OutputStep.WriteAssembly(AssemblyDefinition assembly, String directory, WriterParameters writerParameters)
     at Mono.Linker.Steps.OutputStep.WriteAssembly(AssemblyDefinition assembly, String directory)
     at Mono.Linker.Steps.OutputStep.OutputAssembly(AssemblyDefinition assembly)
     at Mono.Linker.Steps.OutputStep.ProcessAssembly(AssemblyDefinition assembly)
     at Mono.Linker.Steps.BaseStep.Process(LinkContext context)
     --- End of inner exception stack trace ---
     at Mono.Linker.Steps.BaseStep.Process(LinkContext context)
     at Mono.Linker.Pipeline.ProcessStep(LinkContext context, IStep step)
     at Mono.Linker.Pipeline.Process(LinkContext context)
     at Mono.Linker.Driver.Run(ILogger customLogger, Boolean throwOnFatalLinkerException)
     at Mono.Linker.Driver.Main(String[] args)
Author: radical
Assignees: -
Labels:

blocking-clean-ci, linkable-framework

Milestone: -

@radical radical removed the untriaged New issue has not been triaged by the area owner label May 19, 2022
@radical
Copy link
Member Author

radical commented May 20, 2022

cc @eerhardt

@ghost
Copy link

ghost commented May 20, 2022

Tagging subscribers to this area: @vitek-karas, @agocke, @VSadov
See info in area-owners.md if you want to be subscribed.

Issue Details

Build:

ILLink : error IL1012: IL Trimmer has encountered an unexpected error. Please report the issue at https://github.com/dotnet/linker/issues [D:\a\_work\1\s\src\coreclr\System.Private.CoreLib\System.Private.CoreLib.csproj]
##[error]ILLink(0,0): error IL1012: (NETCORE_ENGINEERING_TELEMETRY=Build) IL Trimmer has encountered an unexpected error. Please report the issue at https://github.com/dotnet/linker/issues
  Fatal error in IL Linker
  Unhandled exception. Mono.Linker.InternalErrorException: Step 'OutputStep' failed when processing assembly 'System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e'.
   ---> Mono.Linker.LinkerFatalErrorException: ILLink: error IL1011: Failed to write 'D:\a\_work\1\s\artifacts\obj\coreclr\System.Private.CoreLib\x86\Debug\System.Private.CoreLib.dll'.
   ---> System.NullReferenceException: Object reference not set to an instance of an object.
     at Mono.Cecil.PE.ByteBufferEqualityComparer.Equals(ByteBuffer x, ByteBuffer y)
     at System.Collections.Generic.Dictionary`2.FindValue(TKey key)
     at System.Collections.Generic.Dictionary`2.TryGetValue(TKey key, TValue& value)
     at Mono.Cecil.Metadata.BlobHeapBuffer.GetBlobIndex(ByteBuffer blob)
     at Mono.Cecil.MetadataBuilder.GetBlobIndex(ByteBuffer blob)
     at Mono.Cecil.MetadataBuilder.GetBlobIndex(Byte[] blob)
     at Mono.Cecil.MetadataBuilder.AddCustomDebugInformations(ICustomDebugInformationProvider provider)
     at Mono.Cecil.MetadataBuilder.AddMethodDebugInformation(MethodDebugInformation method_info)
     at Mono.Cecil.Cil.PortablePdbWriter.Write(MethodDebugInformation info)
     at Mono.Cecil.Cil.CodeWriter.WriteResolvedMethodBody(MethodDefinition method)
     at Mono.Cecil.Cil.CodeWriter.WriteMethodBody(MethodDefinition method)
     at Mono.Cecil.MetadataBuilder.AddMethod(MethodDefinition method)
     at Mono.Cecil.MetadataBuilder.AddMethods(TypeDefinition type)
     at Mono.Cecil.MetadataBuilder.AddType(TypeDefinition type)
     at Mono.Cecil.MetadataBuilder.AddTypes()
     at Mono.Cecil.MetadataBuilder.BuildTypes()
     at Mono.Cecil.MetadataBuilder.BuildModule()
     at Mono.Cecil.MetadataBuilder.BuildMetadata()
     at Mono.Cecil.ModuleWriter.<>c.<BuildMetadata>b__2_0(MetadataBuilder builder, MetadataReader _)
     at Mono.Cecil.ModuleDefinition.Read[TItem,TRet](TItem item, Func`3 read)
     at Mono.Cecil.ModuleWriter.BuildMetadata(ModuleDefinition module, MetadataBuilder metadata)
     at Mono.Cecil.ModuleWriter.Write(ModuleDefinition module, Disposable`1 stream, WriterParameters parameters)
     at Mono.Cecil.ModuleWriter.WriteModule(ModuleDefinition module, Disposable`1 stream, WriterParameters parameters)
     at Mono.Cecil.ModuleDefinition.Write(String fileName, WriterParameters parameters)
     at Mono.Cecil.AssemblyDefinition.Write(String fileName, WriterParameters parameters)
     at Mono.Linker.Steps.OutputStep.WriteAssembly(AssemblyDefinition assembly, String directory, WriterParameters writerParameters)
     --- End of inner exception stack trace ---
     at Mono.Linker.Steps.OutputStep.WriteAssembly(AssemblyDefinition assembly, String directory, WriterParameters writerParameters)
     at Mono.Linker.Steps.OutputStep.WriteAssembly(AssemblyDefinition assembly, String directory)
     at Mono.Linker.Steps.OutputStep.OutputAssembly(AssemblyDefinition assembly)
     at Mono.Linker.Steps.OutputStep.ProcessAssembly(AssemblyDefinition assembly)
     at Mono.Linker.Steps.BaseStep.Process(LinkContext context)
     --- End of inner exception stack trace ---
     at Mono.Linker.Steps.BaseStep.Process(LinkContext context)
     at Mono.Linker.Pipeline.ProcessStep(LinkContext context, IStep step)
     at Mono.Linker.Pipeline.Process(LinkContext context)
     at Mono.Linker.Driver.Run(ILogger customLogger, Boolean throwOnFatalLinkerException)
     at Mono.Linker.Driver.Main(String[] args)
Author: radical
Assignees: -
Labels:

blocking-clean-ci, area-Host, linkable-framework

Milestone: -

@eerhardt
Copy link
Member

@radical - has this happened on any other build than your PR? Does it repro in main? If yes - do you have repro steps on what kind of build, and on what platform, this repros on?

@radical
Copy link
Member Author

radical commented May 20, 2022

No, I have seen it only this one time.

@eerhardt
Copy link
Member

No, I have seen it only this one time.

Are you sure the failure isn't caused by your changes? Can you repro it yourself with your changes?

@radical
Copy link
Member Author

radical commented May 20, 2022

It passed on the second attempt by azdo for the same commit of my branch. My PR has very few non-wasm specific changes, and this is not for a wasm run.

@eerhardt
Copy link
Member

Hmm, that's odd for a NRE in the linker to be non-deterministic. @agocke @vitek-karas @sbomer - thoughts?

@AaronRobinsonMSFT
Copy link
Member

AaronRobinsonMSFT commented May 21, 2022

Occurred in #69619 on "Libraries Build windows allConfigurations x64 Debug"

https://dev.azure.com/dnceng/public/_build/results?buildId=1782592&view=logs&j=81a0f8c2-d4db-561f-9b8e-3f045280d7fe&t=2a104984-c43b-5b93-1361-6bd8a1e96b04&l=960

##[error]ILLink(0,0): error IL1012: (NETCORE_ENGINEERING_TELEMETRY=Build) IL Trimmer has encountered an unexpected error. Please report the issue at https://github.com/dotnet/linker/issues
  Fatal error in IL Linker
  Unhandled exception. Mono.Linker.InternalErrorException: Step 'OutputStep' failed when processing assembly 'System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e'.
   ---> Mono.Linker.LinkerFatalErrorException: ILLink: error IL1011: Failed to write 'D:\a\_work\1\s\artifacts\obj\coreclr\System.Private.CoreLib\x64\Debug\System.Private.CoreLib.dll'.
   ---> System.NullReferenceException: Object reference not set to an instance of an object.
     at Mono.Cecil.PE.ByteBufferEqualityComparer.Equals(ByteBuffer x, ByteBuffer y)
     at System.Collections.Generic.Dictionary`2.FindValue(TKey key)
     at System.Collections.Generic.Dictionary`2.TryGetValue(TKey key, TValue& value)
     at Mono.Cecil.Metadata.BlobHeapBuffer.GetBlobIndex(ByteBuffer blob)
     at Mono.Cecil.MetadataBuilder.GetBlobIndex(ByteBuffer blob)
     at Mono.Cecil.MetadataBuilder.GetBlobIndex(Byte[] blob)
     at Mono.Cecil.MetadataBuilder.AddCustomDebugInformations(ICustomDebugInformationProvider provider)
     at Mono.Cecil.MetadataBuilder.AddMethodDebugInformation(MethodDebugInformation method_info)
     at Mono.Cecil.Cil.PortablePdbWriter.Write(MethodDebugInformation info)
     at Mono.Cecil.Cil.CodeWriter.WriteResolvedMethodBody(MethodDefinition method)
     at Mono.Cecil.Cil.CodeWriter.WriteMethodBody(MethodDefinition method)
     at Mono.Cecil.MetadataBuilder.AddMethod(MethodDefinition method)
     at Mono.Cecil.MetadataBuilder.AddMethods(TypeDefinition type)
     at Mono.Cecil.MetadataBuilder.AddType(TypeDefinition type)
     at Mono.Cecil.MetadataBuilder.AddTypes()
     at Mono.Cecil.MetadataBuilder.BuildTypes()
     at Mono.Cecil.MetadataBuilder.BuildModule()
     at Mono.Cecil.MetadataBuilder.BuildMetadata()
     at Mono.Cecil.ModuleWriter.<>c.<BuildMetadata>b__2_0(MetadataBuilder builder, MetadataReader _)
     at Mono.Cecil.ModuleDefinition.Read[TItem,TRet](TItem item, Func`3 read)
     at Mono.Cecil.ModuleWriter.BuildMetadata(ModuleDefinition module, MetadataBuilder metadata)
     at Mono.Cecil.ModuleWriter.Write(ModuleDefinition module, Disposable`1 stream, WriterParameters parameters)
     at Mono.Cecil.ModuleWriter.WriteModule(ModuleDefinition module, Disposable`1 stream, WriterParameters parameters)
     at Mono.Cecil.ModuleDefinition.Write(String fileName, WriterParameters parameters)
     at Mono.Cecil.AssemblyDefinition.Write(String fileName, WriterParameters parameters)
     at Mono.Linker.Steps.OutputStep.WriteAssembly(AssemblyDefinition assembly, String directory, WriterParameters writerParameters)
     --- End of inner exception stack trace ---
     at Mono.Linker.Steps.OutputStep.WriteAssembly(AssemblyDefinition assembly, String directory, WriterParameters writerParameters)
     at Mono.Linker.Steps.OutputStep.WriteAssembly(AssemblyDefinition assembly, String directory)
     at Mono.Linker.Steps.OutputStep.OutputAssembly(AssemblyDefinition assembly)
     at Mono.Linker.Steps.OutputStep.ProcessAssembly(AssemblyDefinition assembly)
     at Mono.Linker.Steps.BaseStep.Process(LinkContext context)
     --- End of inner exception stack trace ---
     at Mono.Linker.Steps.BaseStep.Process(LinkContext context)
     at Mono.Linker.Pipeline.ProcessStep(LinkContext context, IStep step)
     at Mono.Linker.Pipeline.Process(LinkContext context)
     at Mono.Linker.Driver.Run(ILogger customLogger, Boolean throwOnFatalLinkerException)
     at Mono.Linker.Driver.Main(String[] args)

@AaronRobinsonMSFT AaronRobinsonMSFT changed the title ILLinker NRE on windows/x86 ILLinker NRE on Windows May 21, 2022
@MichalStrehovsky
Copy link
Member

If it's indeed non-deterministic, it's probably an issue in the .NET runtime running the tool.

We started to see various mystery bugs in managed tools that run as part of the build ever since switching to the .NET 7 SDK/runtime.

Here's a similar NRE in crossgen2: #69454. The theory is that these are all related, possibly fixed with #68443 and will be fixed once we update to Preview 5 (#69593 (comment)).

@radical
Copy link
Member Author

radical commented May 26, 2022

Another NRE, though it doesn't look the same. But I'll leave it up to others to decide whether to split it off into a new issue or not. This is on p5 rolling build:

/_/src/libraries/System.Private.CoreLib/src/System/IO/File.cs(112,17): error IL1005: System.IO.File.Exists(String): Error processing method 'System.IO.FileSystem.FileExists(ReadOnlySpan<Char>)' in assembly 'System.Private.CoreLib.dll'. [D:\a\_work\1\s\src\libraries\Microsoft.Extensions.Logging.Abstractions\tests\Microsoft.Extensions.Logging.Generators.Tests\Microsoft.Extensions.Logging.Generators.Roslyn3.11.Tests.csproj]
##[error]/_/src/libraries/System.Private.CoreLib/src/System/IO/File.cs(112,17): error IL1005: (NETCORE_ENGINEERING_TELEMETRY=Build) System.IO.File.Exists(String): Error processing method 'System.IO.FileSystem.FileExists(ReadOnlySpan<Char>)' in assembly 'System.Private.CoreLib.dll'.
  Mono.Linker.LinkerFatalErrorException: /_/src/libraries/System.Private.CoreLib/src/System/IO/File.cs(112,17): error IL1005: System.IO.File.Exists(String): Error processing method 'System.IO.FileSystem.FileExists(ReadOnlySpan<Char>)' in assembly 'System.Private.CoreLib.dll'.
   ---> System.NullReferenceException: Object reference not set to an instance of an object.
     at Mono.Linker.Steps.MarkStep.IsMethodNeededByInstantiatedTypeDueToPreservedScope(MethodDefinition method)
     at Mono.Linker.Steps.MarkStep.GetRequiredMethodsForInstantiatedType(TypeDefinition type)+MoveNext()
     at Mono.Linker.Steps.MarkStep.MarkRequirementsForInstantiatedTypes(TypeDefinition type)
     at Mono.Linker.Steps.MarkStep.MarkType(TypeReference reference, DependencyInfo reason, Nullable`1 origin)
     at Mono.Linker.Steps.MarkStep.MarkMethodBody(MethodBody body)
     at Mono.Linker.Steps.MarkStep.ProcessMethod(MethodDefinition method, DependencyInfo& reason, MessageOrigin& origin)
     at Mono.Linker.Steps.MarkStep.ProcessQueue()
     --- End of inner exception stack trace ---
     at Mono.Linker.Steps.MarkStep.ProcessQueue()
     at Mono.Linker.Steps.MarkStep.ProcessPrimaryQueue()
     at Mono.Linker.Steps.MarkStep.Process()
     at Mono.Linker.Steps.MarkStep.Process(LinkContext context)
     at Mono.Linker.Pipeline.ProcessStep(LinkContext context, IStep step)
     at Mono.Linker.Pipeline.Process(LinkContext context)
     at Mono.Linker.Driver.Run(ILogger customLogger, Boolean throwOnFatalLinkerException)

System.IO.File.Exists(String): Error processing method 'System.IO.FileSystem.FileExists(ReadOnlySpan<Char>)' in assembly 'System.Private.CoreLib.dll'

The linker was updated recently with #69644 .

cc @vitek-karas @agocke @marek-safar @lewing

@agocke
Copy link
Member

agocke commented May 26, 2022

Hmm, that method body hasn't been modified recently. It also doesn't look like it happened in the next build. Seems unlikely to be a linker issue.

@buyaa-n
Copy link
Contributor

buyaa-n commented May 31, 2022

Another failure on https://dev.azure.com/dnceng/public/_build/results?buildId=1797953&view=logs&j=55f4943c-f76d-585b-7250-deab324f0a54&t=18a666a9-d033-504f-063f-baac7386b04a

ILLink : error IL1005: System.Runtime.InteropServices.SafeHandle.ReleaseHandle(): Error processing method 'Microsoft.Win32.SafeHandles.SafeFileHandle.ReleaseHandle()' in assembly 'System.Private.CoreLib.dll'. [D:\a\_work\1\s\src\libraries\Microsoft.Extensions.Configuration\tests\FunctionalTests\Microsoft.Extensions.Configuration.Functional.Tests.csproj]
##[error]ILLink(0,0): error IL1005: (NETCORE_ENGINEERING_TELEMETRY=Build) System.Runtime.InteropServices.SafeHandle.ReleaseHandle(): Error processing method 'Microsoft.Win32.SafeHandles.SafeFileHandle.ReleaseHandle()' in assembly 'System.Private.CoreLib.dll'.
  Mono.Linker.LinkerFatalErrorException: ILLink: error IL1005: System.Runtime.InteropServices.SafeHandle.ReleaseHandle(): Error processing method 'Microsoft.Win32.SafeHandles.SafeFileHandle.ReleaseHandle()' in assembly 'System.Private.CoreLib.dll'.
   ---> System.NullReferenceException: Object reference not set to an instance of an object.
     at Mono.Linker.Steps.MarkStep.MarkBaseMethods(MethodDefinition method)
     at Mono.Linker.Steps.MarkStep.ProcessMethod(MethodDefinition method, DependencyInfo& reason, MessageOrigin& origin)
     at Mono.Linker.Steps.MarkStep.ProcessQueue()
     --- End of inner exception stack trace ---
     at Mono.Linker.Steps.MarkStep.ProcessQueue()
     at Mono.Linker.Steps.MarkStep.ProcessPrimaryQueue()
     at Mono.Linker.Steps.MarkStep.Process()
     at Mono.Linker.Steps.MarkStep.Process(LinkContext context)
     at Mono.Linker.Pipeline.ProcessStep(LinkContext context, IStep step)
     at Mono.Linker.Pipeline.Process(LinkContext context)
     at Mono.Linker.Driver.Run(ILogger customLogger, Boolean throwOnFatalLinkerException)
D:\a\_work\1\s\.dotnet\sdk\7.0.100-preview.3.22179.4\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.ILLink.targets(108,5): error NETSDK1144: Optimizing assemblies for size failed. Optimization can be disabled by setting the PublishTrimmed property to false. [D:\a\_work\1\s\src\libraries\Microsoft.Extensions.Configuration\tests\FunctionalTests\Microsoft.Extensions.Configuration.Functional.Tests.csproj]
##[error].dotnet\sdk\7.0.100-preview.3.22179.4\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.ILLink.targets(108,5): error NETSDK1144: (NETCORE_ENGINEERING_TELEMETRY=Build) Optimizing assemblies for size failed. Optimization can be disabled by setting the PublishTrimmed property to false.

@radical
Copy link
Member Author

radical commented Jun 3, 2022

Another NRE, on a rolling build:

  Mono.Linker.LinkerFatalErrorException: /_/src/libraries/System.Private.CoreLib/src/System/Object.cs(42,13): error IL1005: System.Object.ToString(): Error processing method 'Microsoft.Diagnostics.Runtime.ClrAppDomain.ToString()' in assembly 'Microsoft.Diagnostics.Runtime.dll'.
   ---> System.NullReferenceException: Object reference not set to an instance of an object.
     at Mono.Collections.Generic.Collection`1.get_Item(Int32 index)
     at Mono.Cecil.MetadataReader.ReadScope(MethodDefinition method)
     at Mono.Cecil.Cil.PortablePdbReader.ReadScope(MethodDebugInformation method_info)
     at Mono.Cecil.Cil.PortablePdbReader.Read(MethodDefinition method)
     at Mono.Cecil.Cil.CodeReader.ReadMethodBody()
     at Mono.Cecil.Cil.CodeReader.ReadMethodBody(MethodDefinition method)
     at Mono.Cecil.MetadataReader.ReadMethodBody(MethodDefinition method)
     at Mono.Cecil.MethodDefinition.<>c.<get_Body>b__41_0(MethodDefinition method, MetadataReader reader)
     at Mono.Cecil.ModuleDefinition.Read[TItem,TRet](TRet& variable, TItem item, Func`3 read)
     at Mono.Cecil.MethodDefinition.get_Body()
     at Mono.Linker.Steps.UnreachableBlocksOptimizer.ProcessMethod(MethodDefinition method)
     at Mono.Linker.Steps.MarkStep.ProcessMethod(MethodDefinition method, DependencyInfo& reason, MessageOrigin& origin)
     at Mono.Linker.Steps.MarkStep.ProcessQueue()
     --- End of inner exception stack trace ---
     at Mono.Linker.Steps.MarkStep.ProcessQueue()
     at Mono.Linker.Steps.MarkStep.ProcessPrimaryQueue()
     at Mono.Linker.Steps.MarkStep.Process()
     at Mono.Linker.Steps.MarkStep.Process(LinkContext context)
     at Mono.Linker.Pipeline.ProcessStep(LinkContext context, IStep step)
     at Mono.Linker.Pipeline.Process(LinkContext context)
     at Mono.Linker.Driver.Run(ILogger customLogger, Boolean throwOnFatalLinkerException)

@lewing
Copy link
Member

lewing commented Jun 3, 2022

Do we have a working theory for what is causing these?

@radical
Copy link
Member Author

radical commented Jun 3, 2022

The other recent linker exceptions - dotnet/linker#2789 , and #70031 .

@eerhardt
Copy link
Member

eerhardt commented Jun 3, 2022

Or a way to get a trace dump so someone can investigate? Has anyone been able to repro this outside of CI?

@vitek-karas
Copy link
Member

As far as I know we haven't seen this outside of CI - and it's not very common in CI either (less than once a day). The current theory is that it's related to #68443.
For example here is a similar failure in crossgen2 (similar as in relatively random NRE): #69192

Problem is that crossgen2 and linker both run on the runtime which comes with the SDK and not with the runtime which is live built. So even though the possible cause of this has been fixed for a while, it can still show up for the tools.

As for the crash dump - we have discussed this a little, but I'll continue that discussion today. Right now I don't think we have a way to get a crash dump of these failures from the CI. And even locally it will be hard - it's an exception which is handled by the process, so one would have to run it under debugger to catch this.

@lewing
Copy link
Member

lewing commented Jun 3, 2022

So to clarify, it looks like we're waiting on #70117 which @ViktorHofer would prefer not to land until p5 is officially released? How much confidence do we have that the fixes in p5 will resolve the issues?

@maraf
Copy link
Member

maraf commented Jun 7, 2022

Another one on #20220603.2

Mono.Linker.LinkerFatalErrorException: /_/src/libraries/System.Private.CoreLib/src/System/IO/File.cs(112,17): error IL1005: System.IO.File.Exists(String): Error processing method 'System.IO.FileSystem.FileExists(ReadOnlySpan<Char>)' in assembly 'System.Private.CoreLib.dll'.
   ---> System.NullReferenceException: Object reference not set to an instance of an object.
     at Mono.Linker.Steps.MarkStep.IsMethodNeededByInstantiatedTypeDueToPreservedScope(MethodDefinition method)
     at Mono.Linker.Steps.MarkStep.GetRequiredMethodsForInstantiatedType(TypeDefinition type)+MoveNext()
     at Mono.Linker.Steps.MarkStep.MarkRequirementsForInstantiatedTypes(TypeDefinition type)
     at Mono.Linker.Steps.MarkStep.MarkType(TypeReference reference, DependencyInfo reason, Nullable`1 origin)
     at Mono.Linker.Steps.MarkStep.MarkMethodBody(MethodBody body)
     at Mono.Linker.Steps.MarkStep.ProcessMethod(MethodDefinition method, DependencyInfo& reason, MessageOrigin& origin)
     at Mono.Linker.Steps.MarkStep.ProcessQueue()
     --- End of inner exception stack trace ---
     at Mono.Linker.Steps.MarkStep.ProcessQueue()
     at Mono.Linker.Steps.MarkStep.ProcessPrimaryQueue()
     at Mono.Linker.Steps.MarkStep.Process()
     at Mono.Linker.Steps.MarkStep.Process(LinkContext context)
     at Mono.Linker.Pipeline.ProcessStep(LinkContext context, IStep step)
     at Mono.Linker.Pipeline.Process(LinkContext context)
     at Mono.Linker.Driver.Run(ILogger customLogger, Boolean throwOnFatalLinkerException)

@mangod9
Copy link
Member

mangod9 commented Jun 7, 2022

This doesnt appear to be the potential GC issue. @radical, Is there a dump available to look at?

@radical
Copy link
Member Author

radical commented Jun 7, 2022

This doesnt appear to be the potential GC issue. @radical, Is there a dump available to look at?

I don't see one. Maybe @vitek-karas has an answer (ref: #69536 (comment)).

@agocke
Copy link
Member

agocke commented Jun 7, 2022

AFAIK this is only happening in builds, and we don't collect crash dumps on build machines. However, we should change the linker to stop swallowing exceptions, dotnet/linker#2824

@agocke
Copy link
Member

agocke commented Jun 17, 2022

If this was a GC hole, as we suspected, then Preview 5 is merged and this shouldn't happen anymore. Closing until we see new faliures.

@agocke agocke closed this as completed Jun 17, 2022
@ghost ghost locked as resolved and limited conversation to collaborators Jul 17, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
area-VM-coreclr blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' linkable-framework Issues associated with delivering a linker friendly framework
Projects
None yet
Development

No branches or pull requests