NativeAOT: dwarfdump --verify produces thousands of warnings and errors

[TheSpydog]

Running llvm-dwarfdump-12 --verify on NativeAOT-generated linux object files produces a tremendous number of validation warnings and errors. I first noticed this on my Switch project, but I was able to reproduce the problem even when testing with the basic "Hello World" program created by dotnet new console.

A full Hello World validation log for linux-x64 can be found in this gist if you want to wade through 236,000+ lines of dwarfdump output. 😄

Thankfully despite the verbosity of the output, there are only a few unique types of warnings and errors generated.

• warning: DW_TAG_subprogram has DW_CHILDREN_yes but DIE has no children (occurs 2585 times)
• error: Unable to resolve location list offset pair: Base address not defined (occurs 17599 times)
• error: DIEs have overlapping address ranges: (occurs 1683 times)
• error: DIE address ranges are not contained in its parent's ranges: (occurs 793 times)

The latter two appear to be related to exception handling, since they're almost always proceeded by a DW_TAG_try_block or DW_TAG_catch_block.

I'm pretty out of my depth here, but I would like to at least try to debug and fix these issues. Any recommendations for what I should be looking at in ObjWriter?

[kant2002]

When I have issues with DWARF, I add following lines to ObjWriter.EmitObject

if ((node as IMethodNode)?.Method?.DiagnosticName == "InitializeModules")
{
    Debugger.Break();
}

Approximately at this position
https://github.com/dotnet/runtimelab/blob/6f2cfcdb8a4484f5a62234353923ee8073c57593/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/ObjectWriter.cs#L1023

Then I start from there and debug actual DWARF generation code. Since I'm debugging inside VS on Windows, so when working with Linux ARM JIT, I was passing --targetarch arm --targetos linux so ILC pickup proper JIT. Maybe @MichalStrehovsky will give you some idea what else can be done.

[MichalStrehovsky]

Start with building the ObjWriter locally: build.sh/.cmd nativeaot.objwriter (append -c or -rc/-lc how you built the rest of the tree, or just build everything the way I do when I do development: build.cmd/.sh nativeaot.objwriter+nativeaot+libs -rc Debug -lc Release). If you don't build your own ObjWriter, it will be downloaded from NuGet and that one is hard to debug because there's no symbols.

Like @kant2002 said - ILC can crosscompile so you can debug on Windows, passing --targetos to ILC, or debug on Linux natively. Then just run dwarfdump on the produced .o file to inspect.

Dwarf generation is done here: https://github.com/dotnet/runtimelab/tree/feature/NativeAOT/src/coreclr/tools/aot/ObjWriter/debugInfo/dwarf

I recommend glancing over the Dwarf spec - on a high level Dwarf involves a dictionary - the executable defines it's own vocabulary of abbreviations and the debug info refers to entries in the abbreviation list. E.g. here's a couple DW_TAG_subprogram abbreviations with DW_CHILDREN_yes set:

https://github.com/dotnet/runtimelab/blob/6f2cfcdb8a4484f5a62234353923ee8073c57593/src/coreclr/tools/aot/ObjWriter/debugInfo/dwarf/dwarfAbbrev.cpp#L88-L126

[MichalStrehovsky]

Fixed in dotnet/runtimelab#164.

Verifying repro.o:       file format elf64-x86-64
Verifying .debug_abbrev...
Verifying .debug_info Unit Header Chain...
Verifying .debug_info references...
Verifying .debug_types Unit Header Chain...
No errors.

[agocke]

Reopened as the previous fix was just papering over the problems. One interesting thing I noticed was that many of the methods with errors called runtime intrinsics, so there may be something broken with DWARF info for intrinsics.

[agocke]

At this point I think everything except for the overlapping ranges are resolved, which I've created a separate issue for: #72552

I'm going to close this broader issue as complete.

[AustinWise]

I think this is the issue you meant to link: #83233