Symbols missing and sourcelink broken for some shared-framework only assemblies

[natemcmaster]

Opening on behalf of our testing team.

It appears that symbols generated for Microsoft.AspNetCore.App do not contain the correct sourcelink information. Here are instructions to reproduce the problem.

Here is a specific example of a symbol which appears to be missing sourcelink info: Microsoft.AspNetCore.Hosting.ni.pdb for build 3.0.0-preview-19075-0444. native-symbols.zip

Here is how source code round trips via symbols to an end user:

1. Our build generates a JSON file which tells the compiler "these *.cs files came from github.com/aspnet/AspNetCore/commit/$(commithash)"
2. The C# compiler is invoked via MSBuild, which produces a portable PDB file that has this sourcelink info embedded into it.
3. The projects in src/Framework/ collect the .dll and .pdb from step 1 and execute crossgen. This produces the ni.pdb and ni.dll versions
4. The build packages the ni.pdb and ni.dll files into various installers, such as the runtime.$(rid).Microsoft.AspNetCore.App.$(version).symbols.nupkg package.
5. After a build completes, our CI publishes symbols to two symbols servers, MSDL and SymWeb.
6. When a customer is running a debugger, the debugger fetches symbols from one of these symbols servers. The debugger reads the symbols file and finds the source link mapping.
7. The debugger using the mapping to download the source code from GitHub.

Currently, it is the last step in this chain that appears broken. The symbols are downloaded by debuggers, but the sourcelink metadata is missing.

The most likely possibilities:

• Step 1- Our projects might not be generating the correct sourcelink metadata before running the C# compiler.
• Step 3- crossgen does not correctly create symbols. It may be dropping the source link data.

[natemcmaster]

Investigated this a little more. At least part of this is the way we run crossgen. The crossgen tool requires the original symbols in order to correctly re-generate symbols which match the native image. We don't have symbols for assemblies which are not produced in this assembly.

To fix this, we need to download symbols during the build for the shared framework so we can rewrite them with crossgen.

dotnet/extensions#1101 produces a transport package that we can use to get the symbols for Extensions binaries.

We'll need help from the dotnet/corefx and dotnet/core-setup repos to get symbols for the following:

From dotnet/core-setup

• Microsoft.Extensions.DependencyModel
• Microsoft.DotNet.PlatformAbstractions

From dotnet/corefx

• Microsoft.Win32.Registry.dll
• System.Diagnostics.DiagnosticSource.dll
• System.IO.Pipelines.dll
• System.Reflection.Metadata.dll
• System.Runtime.CompilerServices.Unsafe.dll
• System.Security.AccessControl.dll
• System.Security.Cryptography.Cng.dll
• System.Security.Cryptography.Pkcs.dll
• System.Security.Cryptography.Xml.dll
• System.Security.Permissions.dll
• System.Security.Principal.Windows.dll
• System.Text.Encodings.Web.dll
• System.Threading.Channels.dll

[natemcmaster]

I believe this is complete. Will close once I can validate with @JohnTortugo that source link is fixed for all aspnetcore binaries.

[natemcmaster]

Source link is still broken for binaries which are using the gitsubmodule to MessagePack:

C:\Users\namc\Downloads\symbols\Microsoft.AspNetCore.Components.Server.pdb
	14 Documents with errors:
	ea596b1157418cb284b5888540765730adf275d1c003da949b954c8461c4eca8 sha256 csharp /_/src/submodules/MessagePack-CSharp/src/MessagePack/SequenceReader.cs
	https://raw.githubusercontent.com/aspnet/AspNetCore/f0df10f211b5ea8192579808c223e82aab549268/src/submodules/MessagePack-CSharp/src/MessagePack/SequenceReader.cs
	error: url failed NotFound: Not Found
	16e7abfcff6ba9407b77eb9c4874fb5601eced2adf271afc147521cab3286be1 sha256 csharp /_/src/submodules/MessagePack-CSharp/src/MessagePack/SequenceReaderExtensions.cs
	https://raw.githubusercontent.com/aspnet/AspNetCore/f0df10f211b5ea8192579808c223e82aab549268/src/submodules/MessagePack-CSharp/src/MessagePack/SequenceReaderExtensions.cs
	error: url failed NotFound: Not Found
	9c91c861db1bc24a521de3f49c8aeaec2731669edb3fa595e510f5684598f252 sha256 csharp /_/src/submodules/MessagePack-CSharp/src/MessagePack/BufferWriter.cs
	https://raw.githubusercontent.com/aspnet/AspNetCore/f0df10f211b5ea8192579808c223e82aab549268/src/submodules/MessagePack-CSharp/src/MessagePack/BufferWriter.cs
	error: url failed NotFound: Not Found
	01046861ec361fd30a7620771d75f8796515cbfeee5e9714f196a156e9bfdae1 sha256 csharp /_/src/submodules/MessagePack-CSharp/src/MessagePack/ExtensionHeader.cs
	https://raw.githubusercontent.com/aspnet/AspNetCore/f0df10f211b5ea8192579808c223e82aab549268/src/submodules/MessagePack-CSharp/src/MessagePack/ExtensionHeader.cs
	error: url failed NotFound: Not Found
	d776cee8de8901a698bc4cef220561951802ea46711184affdf9f9428da61935 sha256 csharp /_/src/submodules/MessagePack-CSharp/src/MessagePack/ExtensionResult.cs
	https://raw.githubusercontent.com/aspnet/AspNetCore/f0df10f211b5ea8192579808c223e82aab549268/src/submodules/MessagePack-CSharp/src/MessagePack/ExtensionResult.cs
	error: url failed NotFound: Not Found
	f5d910c34faee15dad3ae76d91c4a0b994ef8479a074843bfe264288f3e1d87f sha256 csharp /_/src/submodules/MessagePack-CSharp/src/MessagePack/MessagePackCode.cs
	https://raw.githubusercontent.com/aspnet/AspNetCore/f0df10f211b5ea8192579808c223e82aab549268/src/submodules/MessagePack-CSharp/src/MessagePack/MessagePackCode.cs
	error: url failed NotFound: Not Found
	b51eb57195c463f59a23825ef5aec54bfeea2e67de09c1e61cb799e1dff0ae46 sha256 csharp /_/src/submodules/MessagePack-CSharp/src/MessagePack/MessagePackReader.cs
	https://raw.githubusercontent.com/aspnet/AspNetCore/f0df10f211b5ea8192579808c223e82aab549268/src/submodules/MessagePack-CSharp/src/MessagePack/MessagePackReader.cs
	error: url failed NotFound: Not Found
	a5b8c0a28d1c74706eba0bca8babd0b476fbb9d48146e3adff4508cdaa4af530 sha256 csharp /_/src/submodules/MessagePack-CSharp/src/MessagePack/MessagePackReader.Integers.cs
	https://raw.githubusercontent.com/aspnet/AspNetCore/f0df10f211b5ea8192579808c223e82aab549268/src/submodules/MessagePack-CSharp/src/MessagePack/MessagePackReader.Integers.cs
	error: url failed NotFound: Not Found
	9efbc9769d7affe597a932f482cbabf1e43d14faa90c8dc4ebabdb8fced1451c sha256 csharp /_/src/submodules/MessagePack-CSharp/src/MessagePack/MessagePackWriter.cs
	https://raw.githubusercontent.com/aspnet/AspNetCore/f0df10f211b5ea8192579808c223e82aab549268/src/submodules/MessagePack-CSharp/src/MessagePack/MessagePackWriter.cs
	error: url failed NotFound: Not Found
	2376e85b1d3638fd0613bc777714508ec5d8a9dc7b8d60f62263c38850b361b5 sha256 csharp /_/src/submodules/MessagePack-CSharp/src/MessagePack/Nil.cs
	https://raw.githubusercontent.com/aspnet/AspNetCore/f0df10f211b5ea8192579808c223e82aab549268/src/submodules/MessagePack-CSharp/src/MessagePack/Nil.cs
	error: url failed NotFound: Not Found
	6edd153c95d7c406037c329465e3dc2442c050e1bfd649dec153ed79ca9537a3 sha256 csharp /_/src/submodules/MessagePack-CSharp/src/MessagePack/FloatBits.cs
	https://raw.githubusercontent.com/aspnet/AspNetCore/f0df10f211b5ea8192579808c223e82aab549268/src/submodules/MessagePack-CSharp/src/MessagePack/FloatBits.cs
	error: url failed NotFound: Not Found
	5ed09cbd04c888a8fdfa35a904c88056171416e9112144a92150d6c401dd7efa sha256 csharp /_/src/submodules/MessagePack-CSharp/src/MessagePack/StringEncoding.cs
	https://raw.githubusercontent.com/aspnet/AspNetCore/f0df10f211b5ea8192579808c223e82aab549268/src/submodules/MessagePack-CSharp/src/MessagePack/StringEncoding.cs
	error: url failed NotFound: Not Found
	63a644da62f562e64744112dbfb5553acd257e84c3013fd0e6e8a56125e47cf1 sha256 csharp /_/src/submodules/MessagePack-CSharp/src/MessagePack/SequencePool.cs
	https://raw.githubusercontent.com/aspnet/AspNetCore/f0df10f211b5ea8192579808c223e82aab549268/src/submodules/MessagePack-CSharp/src/MessagePack/SequencePool.cs
	error: url failed NotFound: Not Found
	243119a3481fc938a2bb00b24888530733638a9efca572deda66ad6ff733d3c7 sha256 csharp /_/src/submodules/MessagePack-CSharp/src/MessagePack/Internal/DateTimeConstants.cs
	https://raw.githubusercontent.com/aspnet/AspNetCore/f0df10f211b5ea8192579808c223e82aab549268/src/submodules/MessagePack-CSharp/src/MessagePack/Internal/DateTimeConstants.cs
	error: url failed NotFound: Not Found

[natemcmaster]

@dougbu do you have bandwidth to help on this one?

Here's how to repro the error:

1. Download the latest .symbols.nupkgs from a CI build and extract all .pdbs
2. dotnet tool install -g sourcelink
3. Run sourcelink test on each pdb

I'm still seeing missing sources for the code from the MessagePack git submodule that end up Microsoft.AspNetCore.Components.Server.dll.

One likely cause of the error is a bug in the sourcelink packages that we use. They come from https://github.com/dotnet/sourcelink. It's also possible AzDO is doing something strange when it checks out gitsubmodules which Microsoft.SourceLink.Vsts.Git might not handle. (@tmat might be able to help if you get stuck).

cc @Pilchie

[dougbu]

Not sure about my bandwidth for the next week or so. Will see how things feel early next week…

Why is this issue a P1 at the moment if it affects only the MessagePack submodule?

[natemcmaster]

PRI 2 is probably more appropriate. It's something we should try to resolve before RTM. Also, this impacts @JohnTortugo who has been trying to add source-link validation to our release pipelines.

[dougbu]

Initially it appeared we had four problems when using sourcelink.exe test on all PDBs we publish:

1. “Invalid COR20 header signature.” errors for all *.ni.pdb files from the AspNetCore.App.Runtime.win-arm.3.0.0-preview8.19355.7.symbols.nupkg, Microsoft.AspNetCore.App.Runtime.win-x64.3.0.0-preview8.19355.7.symbols.nupkg, and Microsoft.AspNetCore.App.Runtime.win-x86.3.0.0-preview8.19355.7.symbols packages.
   • The project of interest here is src/Framework/src/Microsoft.AspNetCore.App.Runtime.csproj
   • The sourcelink tool doesn't deal well with Windows PDBs and this could be a false report; see Need API to read SourceLink information from Windows PDBs arcade#2519 and sourcelink test does not match URLs correctly ctaggart/SourceLink#386
   • However the strings output for sample *.ni.pdb files do not include a "documents" entry
   • It's also not clear the symbol servers return our *.ni.pdb files; downloads are always the smaller *.pdb files
2. “Invalid COR20 header signature.” errors for a few native assemblies e.g. src/Servers/IIS/AspNetCoreModuleV2/InProcessRequestHandler/InProcessRequestHandler.vcxproj
   • While @rrelyea and @karann-msft assured me .vcxproj projects do not support <PackageReference>, those items seem to work fine in Directory.Build.targets files above the project (actually via multiple imports within the Arcade SDK which Directory.Build.targets brings in).
   • Put another way, this is a false report; see Need API to read SourceLink information from Windows PDBs arcade#2519 and sourcelink test does not match URLs correctly ctaggart/SourceLink#386
   • Successfully debugged and loaded src\Servers\IIS\AspNetCoreModuleV2\CommonLib\StandardStreamRedirection.cpp outside the AspNetCore repo
3. 404s when attempting to retrieve sources for MessagePack-CSharp (a submodule) files that are compiled into our Microsoft.AspNetCore.Components.Server.dll assembly. The project of interest is src/Components/Server/src/Microsoft.AspNetCore.Components.Server.csproj. The submodule configuration looks perfect. But, does https://github.com/dotnet/sourcelink have trouble with either submodule logical names containing slashes or projects compiling both local and submodule files?
   • This is a false report; see sourcelink test does not match URLs correctly ctaggart/SourceLink#386
   • Successfully debugged and loaded files from the submodule outside the AspNetCore repo
4. 404s when attempting to retrieve just two generated files, both of which are generated using the <GenerateResxSource>true</GenerateResxSource> configuration (the only place where we have that set)
   • This appears to be a real issue: Source for a generated file is not found in the assembly
   • Attempts to break at a property in Microsoft.AspNetCore.Server.Kestrel.Core.CoreStrings.cs result in The debugger could not locate the source file '/_/artifacts/obj/Microsoft.AspNetCore.Server.Kestrel.Core/Release/netcoreapp3.0/Microsoft.AspNetCore.Server.Kestrel.Core.CoreStrings.cs'. errors in VS

[dougbu]

@tmat we've had <EmbedUntrackedSources>true</EmbedUntrackedSources> in our root Directory.Build.props file for quite some time. Nonetheless, see 4. above. I see the string "netcoreapp3.07Microsoft.AspNetCore.Server.Kestrel.Core.CoreStrings.cs" when executing

strings ./Symbols/Microsoft.AspNetCore.Server.Kestrel.Core.pdb/0F03C7C1BB02470EA6CBC0F464FF294Dffffffff/Microsoft.AspNetCore.Server.Kestrel.Core.pdb |grep -i strings

but that's it.

I also built the project locally and confirmed $(EmbedUntrackedSources) is true throughout execution. The Microsoft.Build.Tasks.Git.GetUntrackedFiles task doesn't have artifacts/obj/Microsoft.AspNetCore.Server.Kestrel.Core/Debug/netcoreapp3.0/Microsoft.AspNetCore.Server.Kestrel.Core.CoreStrings.cs as an input and the Microsoft.AspNetCore.Server.Kestrel.Core.AssemblyInfo.cs file in the same directory is its only output.

Anything else I should be looking for?

[dougbu]

Overall, the 404s for Microsoft.AspNetCore.Server.Kestrel.Core.CoreStrings.cs and Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketsStrings.cs relate to real a real problem but it's likely in https://github.com/dotnet/sourcelink/ and not this repo.

---

Will continue to look at what's going on with the crossgen and sourcelink mix in src/Framework/src/Microsoft.AspNetCore.App.Runtime.csproj…

[dougbu]

The shared framework issue is that artifacts\bin\Microsoft.AspNetCore.App.Runtime\Release\netcoreapp3.0\win-x64 contains both crossgened and non-crossgened PDBs for many assemblies. The non-crossgened PDBs are copy-local for Microsoft.AspNetCore.App.Runtime but not overwritten when creating the crossgened PDBs. We end with e.g. Microsoft.AspNetCore.Antiforgery.pdb and Microsoft.AspNetCore.Antiforgery.ni.pdb in the runtime package.

The symbol upload also executes correctly but seems to confuse the symbol server when it uploads both of these PDBs. Probably means symbol resolution is given insufficient information to pick the .ni.pdb file over the .pdb file. Fortunately, we rarely put the same assemblies in a specific package and the runtime package e.g. there's no Microsoft.AspNetCore.Antiforgery.nupkg containing the non-crossgened assembly.

Two choices:

1. Specify -out to overwrite {blah}.pdb when crossgening the PDBs
   • I'm going to try this first with a private official build
2. Remove {blah}.pdb when creating {blah}.ni.pdb

[dougbu]

Many thanks to @tmat and @gregg-miskelly for their help on the questions in this issue❕❕ All questions are now resolved or the subject of issues elsewhere e.g.

• Need API to read SourceLink information from Windows PDBs arcade#2519, Need API to read SourceLink information from Windows PDBs (issue with native and crossgen'd assemblies)
• sourcelink test does not match URLs correctly ctaggart/SourceLink#386, sourcelink test does not match URLs correctly (issue with submodules)
• Some bug somewhere with embedded sources not being found or perhaps not being added at all (problem affects both sourcelink tool and attempts to debug in VS)

After I took this issue, did a lot of investigation but only made small changes:

• Add source link support for ANCM projects #11961
• Remove supposed optimization of local builds #12174

---

Details about final investigation

Our crossgened assemblies and PDBs / .map files are generated and packaged correctly because

• The crossgened assemblies have debug headers for both managed and native PDBs (see below)
• Visual Studio and other managed debuggers only need managed PDBs to debug our code
• Native PDBs (on Windows) and .map files (elsewhere) are useful for profilers and OS-level debuggers (think windbg on Windows and perf on Linux)

Example debug headers

Because I cleared my caches recently, this example is from a nightly or private build of Microsoft.AspNetCore.Mvc.Core.dll, not anything we've released. Saw similar things with 3.0.0-preview7.

dumpbin -headers

  Debug Directories

        Time Type        Size      RVA  Pointer
    -------- ------- -------- -------- --------
    8734CB51 cv           11C 001A4ACC   1A38CC    Format: RSDS, {BEF1FC98-1EC9-48BD-DAFD-57E0990B9FBE}, 1, Microsoft.AspNetCore.Mvc.Core.ni.pdb
    8734CB51 cv            7F 001A4BE8   1A39E8    Format: RSDS, {09809CDD-9916-48D2-9A69-9D988576EA18}, 1, /_/artifacts/obj/Microsoft.AspNetCore.Mvc.Core/Release/netcoreapp3.0/Microsoft.AspNetCore.Mvc.Core.pdb
    00000000 pdbhash       27 001A4C67   1A3A67    SHA256: DD 9C 80 09 16 99 D2 C8 5A 69 9D 98 85 76 EA 18 51 CB 34 87 14 43 3A E8 F3 0D 54 F4 E0 70 BF 4C

mdv

Debug Directory:
  CodeView stamp=0x8734CB51, version=(0x0100, 0x0000), size=284
    path='Microsoft.AspNetCore.Mvc.Core.ni.pdb', guid={bef1fc98-1ec9-48bd-dafd-57e0990b9fbe}, age=1
  CodeView stamp=0x8734CB51, version=(0x0100, 0x504D), size=127
    path='/_/artifacts/obj/Microsoft.AspNetCore.Mvc.Core/Release/netcoreapp3.0/Microsoft.AspNetCore.Mvc.Core.pdb', guid={09809cdd-9916-48d2-9a69-9d988576ea18}, age=1
  PdbChecksum stamp=0x00000000, version=(0x0001, 0x0000), size=39
  Reproducible stamp=0x00000000, version=(0x0000, 0x0000), size=0

Example folder in my symbol cache

    Directory: C:\Temp\Symbols\microsoft.aspnetcore.mvc.core.pdb

Mode                LastWriteTime         Length Name
----                -------------         ------ ----
d-----       2019-07-27     16:37                09809CDD991648D29A699D988576EA18ffffffff

Native assembly not used because I was experimenting in VS. Note the 09809CDD-9916-48D2-9A69-9D988576EA18 GUID in the assembly matches the 09809CDD991648D29A699D988576EA18ffffffff folder name.