Support automatic transient dependency packaging for source generators

[Turnerj]

Originally raised as part of an issue with Roslyn (dotnet/roslyn#52017), it was suggested that I raise it here to potentially get the ball rolling on this.

Basically from a user's point of view, having private dependencies in your source generator can become unwieldy very fast when factoring in transient dependencies. For example, this is what it looks like to add System.Text.Json as a private dependency while factoring in transient dependencies.

  <ItemGroup Label="Package References">
    <PackageReference Include="Microsoft.CodeAnalysis.CSharp" Version="3.9.0" />
    <PackageReference Include="System.Text.Json" Version="5.0.1" GeneratePathProperty="true" PrivateAssets="all" />
    <PackageReference Include="Microsoft.Bcl.AsyncInterfaces" Version="5.0.0" GeneratePathProperty="true" PrivateAssets="all" />
    <PackageReference Include="System.Runtime.CompilerServices.Unsafe" Version="5.0.0" GeneratePathProperty="true" PrivateAssets="all" />
    <PackageReference Include="System.Threading.Tasks.Extensions" Version="4.5.4" GeneratePathProperty="true" PrivateAssets="all" />
    <PackageReference Include="System.Text.Encodings.Web" Version="5.0.1" GeneratePathProperty="true" PrivateAssets="all" />
    <PackageReference Include="System.Buffers" Version="4.5.1" GeneratePathProperty="true" PrivateAssets="all" />
    <PackageReference Include="System.Memory" Version="4.5.4" GeneratePathProperty="true" PrivateAssets="all" />
    <PackageReference Include="System.Numerics.Vectors" Version="4.4.0" GeneratePathProperty="true" PrivateAssets="all" />
  </ItemGroup>

  <PropertyGroup>
    <GetTargetPathDependsOn>$(GetTargetPathDependsOn);GetDependencyTargetPaths</GetTargetPathDependsOn>
  </PropertyGroup>

  <Target Name="GetDependencyTargetPaths">
    <ItemGroup>
      <TargetPathWithTargetPlatformMoniker Include="$(PKGSystem_Text_Json)\lib\netstandard2.0\*.dll" IncludeRuntimeDependency="false" />
      <TargetPathWithTargetPlatformMoniker Include="$(PKGMicrosoft_Bcl_AsyncInterfaces)\lib\netstandard2.0\*.dll" IncludeRuntimeDependency="false" />
      <TargetPathWithTargetPlatformMoniker Include="$(PKGSystem_Runtime_CompilerServices_Unsafe)\lib\netstandard2.0\*.dll" IncludeRuntimeDependency="false" />
      <TargetPathWithTargetPlatformMoniker Include="$(PKGSystem_Threading_Tasks_Extensions)\lib\netstandard2.0\*.dll" IncludeRuntimeDependency="false" />
      <TargetPathWithTargetPlatformMoniker Include="$(PKGSystem_Buffers)\lib\netstandard2.0\*.dll" IncludeRuntimeDependency="false" />
      <TargetPathWithTargetPlatformMoniker Include="$(PKGSystem_Memory)\lib\netstandard2.0\*.dll" IncludeRuntimeDependency="false" />
      <TargetPathWithTargetPlatformMoniker Include="$(PKGSystem_Numerics_Vectors)\lib\netstandard2.0\*.dll" IncludeRuntimeDependency="false" />
      <TargetPathWithTargetPlatformMoniker Include="$(PKGSystem_Text_Encodings_Web)\lib\netstandard2.0\*.dll" IncludeRuntimeDependency="false" />
    </ItemGroup>
  </Target>

Not only am I needing to be more careful about package version updates and their transient dependencies (the linked Roslyn issue came about because upgrading System.Text.Encodings.Web from 5.0.0 to 5.0.1 brought in two more dependencies for .NET Standard 2.0) but it seems like I'm doing the work the compiler/build system should be doing for me.

So far to avoid doing that, I've managed to hack in a semi-automated behaviour with the following:

  <ItemGroup Label="Package References">
    <PackageReference Include="Microsoft.CodeAnalysis.CSharp" Version="3.9.0" />
    <PackageReference Include="System.Text.Json" Version="5.0.1" PrivateAssets="all" />
  </ItemGroup>

  <PropertyGroup>
    <GetTargetPathDependsOn>$(GetTargetPathDependsOn);GetDependencyTargetPaths</GetTargetPathDependsOn>
  </PropertyGroup>

  <Target Name="GetDependencyTargetPaths" AfterTargets="ResolvePackageDependenciesForBuild">
    <ItemGroup>
      <TargetPathWithTargetPlatformMoniker Include="@(ResolvedCompileFileDefinitions)" IncludeRuntimeDependency="false" />
    </ItemGroup>
  </Target>

It is not great for multiple reasons (ResolvedCompileFileDefinitions isn't always available & includes more than the dependencies that we actually want to bundle) however this still seems like a more practical solution than manually specifying transient dependencies.

From my point of view, the ideal scenario is simply specifying PrivateAssets="all" and the build system takes care of the rest.

[dotnet-issue-labeler]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[sfoslund]

@ericstj are you the right person to comment here?

[ericstj]

@ericstj are you the right person to comment here?

I don't know, depends on what you want in the way of comments. Here's some thoughts. cc @jaredpar @chsienki @KathleenDollard

I would have expected that CopyLocalLockFileAssemblies would be enough to make Roslyn happy, but it doesn't seem to probe next to the analyzer assembly. You have to pass it into the compiler. Here's @Turnerj's repro with that change that demonstrates that (seems like a fix to Roslyn could make this better). ericstj/SourceGeneratorDependencyTest@e80fbd3
There's probably a better sample than the manual one shared in the original post that can do the right thing based on what the SDK would have copied. I don't have time to sketch that out but might get back to it later. Perhaps such an example should be added to the source generator cookbook?

Related to this is how analyzers pack. Ideally the same mechanism that decides what should be passed via a project reference to an analyzer also determines what's included in a an analyzers package.

Next issue: source generators and analyzers are netstandard2.0 libraries. Creating a netstandard2.0 runnable layout with dependencies is not recommended, since netstandard2.0 isn't a runnable framework. By choosing netstandard2.0 assets from package dependencies you may be missing functionality, missing platform-specific loading, or be missing type unification. Something to consider around the cost of having dependencies.

Next issue: dependencies for plugins that run on desktop don't really work well unless the plugin model supports bindingRedirects. Analyzers do not as far as I know. @jaredpar @chsienki how does this work when dependencies are passed to the compiler via the /analyzer flag, will roslyn automatically unify everyone to the set of dependencies passed in (deduping as well)?

[jaredpar]

Analyzers do not as far as I know. @jaredpar @chsienki how does this work when dependencies are passed to the compiler via the /analyzer flag, will roslyn automatically unify everyone to the set of dependencies passed in (deduping as well)?

At a high level the compiler approach is to call Assembly.Load(string path) on the analyzer and then let the runtime deal with the dependencies. The details are more complicated but at a high level that is what we are doing.

Most analyzers / generators that have dependencies tend to be very simple, one layer deep dependencies. Likely because that is all that will work due to the issues you're outlining above. There also isn't a lot of room for us to do binding redirects. In the command line compiler / server yes we could think about doing this. But it would be custom implemented because we can't use AppDomain isolation. Other hosts though like Visual Studio or VS Code we can't safely implement binding redirects without possibly compromising the integrity of the host process.

Overall though the solution here is getting to a point where the compiler is always running on .NET Core. In that mode we use AssemblyLoadContext and the dependency loading story is more flexible and has a lot more options.

[ericstj]

At a high level the compiler approach is to call Assembly.Load(string path) on the analyzer and then let the runtime deal with the dependencies. The details are more complicated but at a high level that is what we are doing.

That's what I would have expected, in which case the runtime would probe next to the analyzer assembly, but that doesn't seem to be the case. Looks to me like roslyn makes a copy of the analyzer before loading it and misses copying any adjacent files. This is from a run with /analyzer:C:\scratch\SourceGeneratorDependencyTest\SourceGeneratorProject\bin\Debug\netstandard2.0\SourceGeneratorProject.dll where all the dependencies were copied next to the dll.

LOG: Attempting download of new URL file:///C:/Program Files (x86)/Microsoft Visual Studio/2019/Preview/MSBuild/Current/Bin/Roslyn/System.Text.Json.DLL.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/Microsoft Visual Studio/2019/Preview/MSBuild/Current/Bin/Roslyn/System.Text.Json/System.Text.Json.DLL.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/Microsoft Visual Studio/2019/Preview/MSBuild/Current/Bin/Roslyn/System.Text.Json.EXE.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/Microsoft Visual Studio/2019/Preview/MSBuild/Current/Bin/Roslyn/System.Text.Json/System.Text.Json.EXE.
LOG: Attempting download of new URL file:///C:/Users/erics/AppData/Local/Temp/VBCSCompiler/AnalyzerAssemblyLoader/fbe76030cce94d90a0cf41943286fe57/3/System.Text.Json.DLL.
LOG: Attempting download of new URL file:///C:/Users/erics/AppData/Local/Temp/VBCSCompiler/AnalyzerAssemblyLoader/fbe76030cce94d90a0cf41943286fe57/3/System.Text.Json/System.Text.Json.DLL.
LOG: Attempting download of new URL file:///C:/Users/erics/AppData/Local/Temp/VBCSCompiler/AnalyzerAssemblyLoader/fbe76030cce94d90a0cf41943286fe57/3/System.Text.Json.EXE.
LOG: Attempting download of new URL file:///C:/Users/erics/AppData/Local/Temp/VBCSCompiler/AnalyzerAssemblyLoader/fbe76030cce94d90a0cf41943286fe57/3/System.Text.Json/System.Text.Json.EXE.
LOG: All probing URLs attempted and failed.

This was when loading analyzer from a p2p reference. I don't see how this would ever work since Roslyn is copying each analyzer to its own directory.
I notice different behavior when running the compiler directly and avoiding the compiler server, in that case no copy happens, so the runtime will probe next to the analyzer assembly. This works correctly so long as no dependency requires a bindingRedirect.

The repro required a bindingRedirect for System.Text.Encodings.Web so it fails no matter what.

In both cases it seems like roslyn shares the same AppDomain for analyzer loads so compiler server path will work if you happen to pass all dependencies to the compiler and dependencies happen to be passed in breadth first order from lowest in the stack up.

All of these were tested using .NETFramework. I see no failure in .NETCore, where I believe this case happens to work because the dependencies all happen to exist in the shared framework.

[jaredpar]

That's what I would have expected, in which case the runtime would probe next to the analyzer assembly, but that doesn't seem to be the case. Looks to me like roslyn makes a copy of the analyzer before loading it and misses copying any adjacent files

Yep that is the "details are more complicated part". 😄

How you invoke the compiler changes the details here. The command line will just use Assembly.Load in place. It's a short lived process hence there are no locking concerns. The server will copy all of the values passed via /analyzer into a temp directory and load from there. That prevents file locking in the dev work space.

The end result is roughly the same though as the shadow copy directory is meant to mirror the original. Hence at a high level it should still be Assembly.Load

Looks to me like roslyn makes a copy of the analyzer before loading it and misses copying any adjacent files.

This is by explicit design. The compiler will only copy the arguments to /analyzer as those are the only tracked input to the compilers. This means that dependencies to analyzers must be passed with /analyzer. This is actually quite natural though. The build rules default into passing all DLLs in an analyzer NuPkg via /analyzer hence you essentially have to go out of your way for this to not happen.

[ericstj]

I see. So in order to have dependencies passed to the compiler you need to pass everything as an /analyzer, this happens automatically when you package everything into a nuget package, but doesn't happen when you use ProjectReferences. This issue is presumably asking the SDK to make better support for Analyzers/Source-generators as ProjectReferences to address this.

You can't rely on assemblies being next to the analyzer, though this works for csc command, it doesn't work for the compiler server due to the copying done by the compiler (this means that in general analyzers can't rely on doing any file relative loading, good thing to remember).

I was noticing what I thought was an ordering issue with loading, but I think this was actually a bindingRedirect issue. It seems the compiler loads all analyzer assemblies before it attempts to get types from those assemblies. This is what allows passing dependencies as analyzers to work, since the compiler loads them up front. The lack of bindingRedirect handling seems like a pretty significant limitation and seems like something folks writing analyzers/generators with dependencies need to consider up front.

It is not great for multiple reasons (ResolvedCompileFileDefinitions isn't always available

Solve this by adding DependsOnTargets="ResolveReferences"

includes more than the dependencies that we actually want to bundle)

You're using the wrong item (compile assets, instead of runtime) and you're using an earlier stage of it (before conflict resolution and RAR). Try setting CopyLocalLockFileAssemblies=true and using ReferenceCopyLocalPaths item instead. This will have all the runtime assemblies which might be copied as if you had an application.

[Turnerj]

Thanks @ericstj , I'll try that out when I can as it would be a good workaround until the SDK does this automatically. I mean if it works well, could the SDK have this target built-in so myself (and others) wouldn't need that target for every source generator? Would that simplify things on your end or actually make it more complicated?

[ericstj]

I'm not sure having support for that in the SDK makes sense unless the SDK adds first class support for analyzers as project references, and potentially as packages too. If it did that, I'd expect it to have some sort of default understanding for what an analyzer project reference looked like and how to treat it's dependencies.

[Turnerj]

From my point of view (one where I don't know how much work would actually be required to do this), if adding first class support for analyzers as project references etc too gets me/us/.NET developers closer to a seamless experience with transient dependencies, that would be fantastic.

Just right now, dealing with transient dependencies for source generators suck.

[Turnerj]

I hate to be that person but is there anyway I can help push this forward? Do I need to convince more people this is a problem or help contribute to a PR for this?

[user72356]

@Turnerj I can tell you it's a problem for me, and if you get just ONE person to comment, that means there are 100 others with the same problem but not commenting... I don't have the bandwidth to learn the complex inner workings of Roslyn, msbuild and whatnot just to deliver a source generator. Not a wise use of my dev time. So I'm also quite supportive of a better dev experience for source generators, be it when referenced from another solution project, or when packaged in a nuget.

[HavenDV]

I am also interested in an automatic solution. Here is a workaround for automatically detecting explicit and transient dependencies (with a bit of overhead) for those publishing generators via NuGet:

  <!-- 
    https://github.com/dotnet/roslyn/issues/52017#issuecomment-1046216200
    This automatically adds explicit and transient dependencies so that they are available at the time the generator is executed. 
  -->
  <Target Name="AddGenerationTimeReferences" AfterTargets="ResolvePackageDependenciesForBuild">
    <ItemGroup>
      <None Include="@(ResolvedCompileFileDefinitions)" Pack="true" PackagePath="analyzers/dotnet/cs" />
    </ItemGroup>
  </Target>

[kceiw]

This issue needs fixing. I don't find a good way to make it work for "dotnet msbuild" and "msbuild".

Here is a project:
https://github.com/Azure/azure-powershell/blob/6e2afb2f97b02f1914502267056c8d2efa8bd230/tools/Az.Tools.Predictor/Az.Tools.Predictor.SourceGenerator/Az.Tools.Predictor.SourceGenerator.csproj

Here is the snippet that has a workaround

  <Target Name="GetDependencyTargetPaths" AfterTargets="ResolvePackageDependenciesForBuild">
    <ItemGroup>
      <TargetPathWithTargetPlatformMoniker Include="@(ResolvedCompileFileDefinitions)" IncludeRuntimeDependency="false" />
    </ItemGroup>
  </Target>

That works when I build it from Visual Studio or use "msbuild" from the command line. But that doesn't work with "dotnet msbuild"

This works with "dotnet msbuild" but not "msbuild" or Visual Studio

  <Target Name="GetDependencyTargetPaths" AfterTargets="ResolvePackageDependenciesForBuild">
    <ItemGroup>
      <None Include="@(ResolvedCompileFileDefinitions)" Pack="true" PackagePath="analyzers/dotnet/cs" />
    </ItemGroup>
  </Target>

[HavenDV]

This issue needs fixing. I don't find a good way to make it work for "dotnet msbuild" and "msbuild".

Here is a project: https://github.com/Azure/azure-powershell/blob/6e2afb2f97b02f1914502267056c8d2efa8bd230/tools/Az.Tools.Predictor/Az.Tools.Predictor.SourceGenerator/Az.Tools.Predictor.SourceGenerator.csproj

Here is the snippet that has a workaround

  <Target Name="GetDependencyTargetPaths" AfterTargets="ResolvePackageDependenciesForBuild">
    <ItemGroup>
      <TargetPathWithTargetPlatformMoniker Include="@(ResolvedCompileFileDefinitions)" IncludeRuntimeDependency="false" />
    </ItemGroup>
  </Target>

That works when I build it from Visual Studio or use "msbuild" from the command line. But that doesn't work with "dotnet msbuild"

This works with "dotnet msbuild" but not "msbuild" or Visual Studio

  <Target Name="GetDependencyTargetPaths" AfterTargets="ResolvePackageDependenciesForBuild">
    <ItemGroup>
      <None Include="@(ResolvedCompileFileDefinitions)" Pack="true" PackagePath="analyzers/dotnet/cs" />
    </ItemGroup>
  </Target>

If you need both to work, you can try to specify it explicitly at run via your custom property, for example msbuild -property:FromMSBuild=true and

  <Target Name="GetDependencyTargetPaths" AfterTargets="ResolvePackageDependenciesForBuild">
    <ItemGroup Condition="'$(FromMSBuild)">
      <TargetPathWithTargetPlatformMoniker Include="@(ResolvedCompileFileDefinitions)" IncludeRuntimeDependency="false" />
    </ItemGroup>
    <ItemGroup Condition="'$(FromDotnet)">
      <None Include="@(ResolvedCompileFileDefinitions)" Pack="true" PackagePath="analyzers/dotnet/cs" />
    </ItemGroup>
  </Target>

[kceiw]

@HavenDV Thanks for the suggestion. That'll work. The only issue is that we need to remember to set that property, depend on what we use.

[KalleOlaviNiemitalo]

Can you use the MSBuildRuntimeType property? MSBuild reserved and well-known properties

[kceiw]

Can you use the MSBuildRuntimeType property? MSBuild reserved and well-known properties

That works. Thanks.

[HavenDV]

I often work with generators and ended up with the following helper library:
https://github.com/HavenDV/H.Generators.Extensions
It will add the .props file to your build which will automatically infer all generation-time dependencies including transient and add them to the tests and NuGet package. It also solves the problem with system libraries.
The usage is pretty simple:

<PackageReference Include="H.Generators.Extensions" Version="1.4.2" PrivateAssets="all" />

I want to note that PrivateAssets="all" is required to rule out some issues.

[alexrosenfeld10]

@HavenDV this package is super useful, thank you for sharing it!

[kkukshtel]

@HavenDV I tried using this package with just this in my generator project:

  <ItemGroup>
    <PackageReference Include="H.Generators.Extensions" Version="1.4.2" PrivateAssets="all" />
    <PackageReference Include="System.Text.Json" Version="6.0.0" GeneratePathProperty="true" />
  </ItemGroup>

And upon building the generator I get:

failed to generate source. It will not contribute to the output and compilation errors may occur as a result. Exception was of type 'FileNotFoundException' with message 'Could not load file or assembly 'Microsoft.Bcl.AsyncInterfaces, Version=6.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'.

The generator works as normal (minus the issue as stated from this ticket) with just this in the csproj:

  <ItemGroup>
    <PackageReference Include="Microsoft.CodeAnalysis.CSharp" Version="3.9.0" PrivateAssets="all" />
    <PackageReference Include="Microsoft.CodeAnalysis.Analyzers" Version="3.0.0" PrivateAssets="all" />
    <PackageReference Include="System.Text.Json" Version="6.0.0" GeneratePathProperty="true" />
  </ItemGroup>

Any tips on what's going on? Also not sure if this is related, but I've been having a hell of a time trying to get this thing packaged as nuget for .netstandard2.0. What put me on this path was trying to install this package in nuget and getting this error:

An instance of analyzer Depot.SourceGenerator.DepotSourceGenerator cannot be created from C:\Users\kyle\.nuget\packages\afterschool.depot.sourcegenerator\1.0.5\analyzers\dotnet\cs\Depot.Sourcegen.dll: Could not load file or assembly 'System.Runtime, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.

Here's a link to the generator as well for good measure. It works locally but is totally borked with nuget.

[HavenDV]

@HavenDV I tried using this package with just this in my generator project:

  <ItemGroup>
    <PackageReference Include="H.Generators.Extensions" Version="1.4.2" PrivateAssets="all" />
    <PackageReference Include="System.Text.Json" Version="6.0.0" GeneratePathProperty="true" />
  </ItemGroup>

And upon building the generator I get:

failed to generate source. It will not contribute to the output and compilation errors may occur as a result. Exception was of type 'FileNotFoundException' with message 'Could not load file or assembly 'Microsoft.Bcl.AsyncInterfaces, Version=6.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'.

The generator works as normal (minus the issue as stated from this ticket) with just this in the csproj:

  <ItemGroup>
    <PackageReference Include="Microsoft.CodeAnalysis.CSharp" Version="3.9.0" PrivateAssets="all" />
    <PackageReference Include="Microsoft.CodeAnalysis.Analyzers" Version="3.0.0" PrivateAssets="all" />
    <PackageReference Include="System.Text.Json" Version="6.0.0" GeneratePathProperty="true" />
  </ItemGroup>

Any tips on what's going on? Also not sure if this is related, but I've been having a hell of a time trying to get this thing packaged as nuget for .netstandard2.0. What put me on this path was trying to install this package in nuget and getting this error:

An instance of analyzer Depot.SourceGenerator.DepotSourceGenerator cannot be created from C:\Users\kyle\.nuget\packages\afterschool.depot.sourcegenerator\1.0.5\analyzers\dotnet\cs\Depot.Sourcegen.dll: Could not load file or assembly 'System.Runtime, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.

Here's a link to the generator as well for good measure. It works locally but is totally borked with nuget.

H.Generators.Extensions are for Roslyn 4.0+. You can look at this generator, it uses automatic dependency inference based on this package:
https://github.com/HavenDV/H.NSwag.Generator
In general, I recommend using it only for those who have a lot of transitive dependencies at the generation stage, like the generator above

[kkukshtel]

@HavenDV should generator packing "just work" then without a dep on transient dependencies? I know this is dipping outside the context of the initial issue but this and @Turnerj's inital blog post are the only resources I've found on people actually shipping generators to nuget as well as the surprisingly large amount of issues that occur.

[HavenDV]

https://github.com/HavenDV/H.Generators.Extensions/blob/main/src/libs/H.Generators.Extensions/build/H.Generators.Extensions.props
All my NuGet package does is deliver this .props file. You can copy it to your project

[DaneSpiritGOD]

It seems that roslyn-sdk official sample might give us the current solution: CSharpSourceGeneratorSamples

[thepigeonfighter]

Just here to agree this is awful and needs fixed. I couldn't get the roslyn-sdk official sample solution to work, or any other solutions except for explicitly adding all deps. Now I have a bunch of build warnings but it seems to compile.....for now.

[JaggerJo]

+1

This should work out of the box.

[DaneSpiritGOD]

I have received.

[nev-21]

It seems that roslyn-sdk official sample might give us the current solution: CSharpSourceGeneratorSamples

This is confusing, that official sample and the cookbook has different ways:

https://github.com/dotnet/roslyn/blob/main/docs/features/incremental-generators.cookbook.md#use-functionality-from-nuget-packages

Also agree that this should work out of the box

[DaneSpiritGOD]

I have received.

[kkukshtel]

Just because the thread was bumped... tbh it would be nice to have a target for generators that supports the latest net core release instead of being pinned to netstandard2.0 - or on the flipside, would just love to have first class modern System.Text.JSON in a source generator without needing the additional package download and dep tree. I suspect that a large % of generators use it (or Newtonsoft) and honestly just having it packed would solve my dependency woes because it's the only thing I need.

[StefanPoelloth]

Its wild that this issue is still open 3 years later.

[DaneSpiritGOD]

I have received.