New project system doesn't copy PDBs from packages

[davkean]

From @bording on July 27, 2017 3:12

When a NuGet package includes PDBs in its lib folder, the legacy project system copies the PDB as well as the assembly into the build output folder.

The new project system only copies the assembly.

We've recently added source link support , and when using the new project system with a .NET Core target, everything works fine since the PDB is referenced correctly from the global NuGet package folder.

However, because the PDB isn't copied for a .NET Framework target, we don't get any of the benefit of our sourcelinked PDB.

It would be great if the new project system would behave like the legacy project system in this case, and also copy the PDB into the project's build output folder.

Copied from original issue: dotnet/project-system#2667

EDIT: We are also using this issue to track copying XML files

[wli3]

https://github.com/Microsoft/msbuild/blob/master/src/Tasks/AssemblyDependency/ResolveAssemblyReference.cs#L75 the old code

[wli3]

wli3@0fefc41
an end to end repo/test for this bug

[wli3]

sdk/src/Tasks/Microsoft.NET.Build.Tasks/build/Microsoft.PackageDependencyResolution.targets

Line 501 in 98ea6b2

<Target Name="_ComputeLockFileCopyLocal"

decide what to copy local, there should be a FileGroup called something like "DebugSymbol". However, FileGroup are from NuGet

https://github.com/NuGet/NuGet.Client/blob/efae789c75899ca0bdd063f34a4b73cd41510d56/src/NuGet.Core/NuGet.ProjectModel/LockFile/LockFileTargetLibrary.cs#L27

We need NugGet.ProjectModel to tell they are debug symbol

[dsplaisted]

If we add a FileGroup for .pdbs to the asset file, it probably makes sense to add one for the .xml documentation files too.

[dasMulli]

Maybe even for everything that msbuild considers to be related file extensions:

private string[] _relatedFileExtensions = new string[] { ".pdb", ".xml", ".pri" };

[bording]

@wli3 Is there any more progress on this?

[onyxmaster]

Really would like this being fixed. Having dependent PDBs is crucial for in-production diagnostics + debugging.

[jnm2]

I figured out a workaround which can be pasted into the csproj:

  <!-- https://github.com/dotnet/sdk/issues/1458 -->
  <PropertyGroup>
    <AllowedReferenceRelatedFileExtensions>.pdb</AllowedReferenceRelatedFileExtensions>
  </PropertyGroup>
  <Target Name="AddReferenceRelatedPathsToCopyLocal" AfterTargets="ResolveAssemblyReferences">
    <ItemGroup>
      <ReferenceCopyLocalPaths Include="@(_ReferenceRelatedPaths)" />
    </ItemGroup>
  </Target>

[jnm2]

Heads up: this workaround will be broken by a change which has just been merged: #1725 (comment)

@nguerrera wants to fix this issue in the same release so I'll wait and see.

[bording]

Any status update on this? I see it's now been moved to the 2.2.0 milestone.

@nguerrera There is no milestone on #1725, but am I correct in thinking that those changes haven't been included in a release yet?

[nguerrera]

@bording You are correct. They have been included only in unofficial 2.2.0 daily builds.

Small update: I am currently looking at optimization of how we resolve assets and I think I have a way to replicate the related file semantics of RAR, but using runtime assets correctly and without needing to probe on disk and without actually needing nuget changes. Depending on how things go, it is possible that 2.2.0 preview 1 will ship with #1725 but not the fix for this, but the plan or record is that we don't RTM 2.2 in that state.

[dasMulli]

If you could squeeze it in: It would be really cool to have metadata about satellite assemblies available as well. This would allow whipping up some targets to filter satellites coming from NuGet by locale. This has come up in https://github.com/dotnet/cli/issues/8060 and https://stackoverflow.com/questions/47942818/how-to-turn-off-language-specific-output-for-net-core-applications.

[nguerrera]

Will do.

[jnm2]

Which release of VS is this currently slated for? I'm keeping an eye out so I can update instructions so that people know how to use NuGet packages that contain PDBs for source-stepping.

[nguerrera]

Adding @livarcocc for the question about VS release. I don't know if the schedule is set yet.

[jnm2]

Is 15.6 definitely out of the question?

[nguerrera]

Yes, 15.6 is out of the question. This is being tracked for used to be called 2.2.0, but will actually be 2.1.300, which will not ship at the same time as 15.6.

[bording]

@nguerrera How is the transition to the new SDK versioning scheme happening? Currently we have 2.1.4, so what's the next release going to be? Are there going to be additional 2.1 SDKs that still have the 2.0 runtime? Is it known which SDK release will be the one that includes the 2.1 runtime?

I'm trying to get a feel for how far out 2.1.300 is because not having PDBs copied is a pretty big pain point.

[dasMulli]

@bording see dotnet/designs#29 for the new versioning scheme.

[bording]

@dasMulli I follow that repo, so I've seen that already, but thanks for the link!

My questions were more about things that aren't explicitly called out in that document. It implies that 2.1.300 will be the first version to include the 2.1 runtime. It also mentions 2.1.100. Would that be an intermediate feature release, or would that be a release from the servicing branch?

I'd really like to see this scheduled sooner than the SDK release tied to the 2.1 runtime, so I'm trying to see if my understanding of the new versioning scheme and the transition to it is correct.

[ocallesp]

Fixed with #27580

[ocallesp]

To copy to the output directory xml/documentation files, then set CopyDocumentationFilesFromPackages to true.

To copy debug symbol files to the output directory, then set CopyDebugSymbolFilesFromPackages to true.

[dsplaisted]

To copy to the output directory xml/documentation files, then set CopyDocumentationFilesFromPackages to true.

To copy debug symbol files to the output directory, then set CopyDebugSymbolFilesFromPackages to true.

This should be available in .NET SDK 7.0.100 RC2.

[amaitland]

To copy to the output directory xml/documentation files, then set CopyDocumentationFilesFromPackages to true.

So CopyDocumentationFilesFromPackages defaults to false (or empty)? Basically meaning users have to opt into this feature to get IntelliSense to work for Nuget packages?

[dsplaisted]

To copy to the output directory xml/documentation files, then set CopyDocumentationFilesFromPackages to true.

So CopyDocumentationFilesFromPackages defaults to false (or empty)? Basically meaning users have to opt into this feature to get IntelliSense to work for Nuget packages?

We decided for now to make it opt in, because it is fairly late in the release cycle of 7.0.100. Any change we make can end up breaking people, so we would like to have a longer preview period where it is enabled. We will probably turn it on by default in a later release.

However, I don't think this would normally affect whether you get IntelliSense in the IDE from APIs that come from NuGet packages. I believe that if you have a PackageReference to a package which has an XML documentation file in it, you would already get IntelliSense. Copying the documentation files to the output folder was for scenarios such as Swagger where they are needed at runtime.

What is the scenario where you aren't seeing IntelliSense working for NuGet packages?

[olivier-spinelli]

@ocallesp CopyDocumentationFilesFromPackages is great!
I have a workaround for my current .Net 6 projects (that is independent of CopyLocalLockFileAssemblies as opposed to @gitfool and @jnm2 solutions), and I think you'll imagine my subsequent question:

  <Target Name="CopyXmlDocFile" AfterTargets="ResolveReferences">
    <ItemGroup>
      <XmlDocFiles Include="@(ReferencePath->'%(RootDir)%(Directory)%(Filename).xml')"
                   Condition="!( $([System.String]::new('%(FileName)').StartsWith('System.'))
                                 Or
                                 $([System.String]::new('%(FileName)').StartsWith('Microsoft.'))
                                 Or
                                 $([System.String]::new('%(FileName)').StartsWith('CommunityToolkit.'))
                                 Or
                                 '%(FileName)' == 'netstandard'
                               )" />
    </ItemGroup>
    <Copy SourceFiles="@(XmlDocFiles)"
          Condition="Exists('%(FullPath)')"
          DestinationFolder="$(OutDir)"
          SkipUnchangedFiles="true" />
  </Target>

Will there be a way with CopyDocumentationFilesFromPackages to filter (exclude or explicitly include) the documentation files?

[astrowalker]

To copy to the output directory xml/documentation files, then set CopyDocumentationFilesFromPackages to true.

Thank you, but I am lost, where should I put it? I use VS 2019, I have a nuget package, I checked it, next to .dll file I have .xml one. I added such entry in project which uses this package (i.e. in consumer project):

  <PropertyGroup>
     ...
    <CopyDocumentationFilesFromPackages>true</CopyDocumentationFilesFromPackages>
  </PropertyGroup>

Then I built the project and... xml didn't show up. Just .dll file (from the nuget, I mean).

[dsplaisted]

This is fixed in an upcoming release of the .NET SDK (7.0.100). It won't be fixed with Visual Studio 2019, as the new SDK will require a recent version of Visual Studio (ideally 17.4).

[emmenlau]

Thanks @dsplaisted . Will this get fixed in .NET 6 (LTS)?

[dsplaisted]

Thanks @dsplaisted . Will this get fixed in .NET 6 (LTS)?

No, this is a new feature for .NET 7

[dsplaisted]

I should say that this is a new feature for the .NET 7 SDK. You can use the .NET 7 SDK to build projects targeting .NET 6 or earlier, and in that case you will be able to set the CopyDocumentationFilesFromPackages and CopyDebugSymbolFilesFromPackages properties and have them work.

[olivier-spinelli]

@dsplaisted Any answer to my (humble I hope) need to fitler the copied files?

[dsplaisted]

Will there be a way with CopyDocumentationFilesFromPackages to filter (exclude or explicitly include) the documentation files?

There's not a built-in way to do this. The PDB and XML files to copy get added to the ReferenceCopyLocalPaths item, so you could remove the ones you don't want from that in a target after ResolvePackageAssets. The items will have NuGetPackageId for the package they came from which may help you to do the filtering.

Also FYI the MSBuild Log Viewer is very helpful when writing or debugging custom MSBuild code like this.

[olivier-spinelli]

This is perfect.
Many thanks for your answer!

[kirsan31]

Will there be a way with CopyDocumentationFilesFromPackages to filter (exclude or explicitly include) the documentation files?

I'm interested in a simpler thing - CopyDocumentationFilesFromPackages and CopyDebugSymbolFilesFromPackages per package not per project?
The reason is simple - I need bdp only for my packages. And with current CopyDebugSymbolFilesFromPackages implementation I ended up with tons of third party pdbs in the output :(

[dsplaisted]

@kirsan31 In that case you would have to write some custom MSBuild logic to filter the files: #1458 (comment)

[kirsan31]

@dsplaisted

In that case you would have to write some custom MSBuild logic to filter the files: #1458 (comment)

Yes, that's exactly what I meant. With introduction of CopyDebugSymbolFilesFromPackages we cover other edge case - we need all pdb files. Either everything or nothing. And most users are somewhere in between - need to use manual filtering anyway :(
If such flags were at the package level, then this would reduce manual filtering to a minimum...

[POnakS]

@dsplaisted any plans to make this true by defualt? I feel this is very important option to have, yet it's hard to find it.

[SimonCropp]

@ocallesp @dsplaisted should CopyDebugSymbolFilesFromPackages copy pdbs from runtime dirs?

#38322