Feature request: Simple way to express an order-only dependency.

[jhudsoncedaron]

Previously (net core 2.1), this worked:

  <ItemGroup>
    <ProjectReference Include="..\buildtool\buildtool.csproj" Properties="RuntimeIdentifier=">
      <ReferenceOutputAssembly>false</ReferenceOutputAssembly>
    </ProjectReference>
  </ItemGroup>

Now, it doesn't, and even all this doesn't work:

  <ItemGroup>
    <ProjectReference Include="..\buildtool\buildtool.csproj" ExcludeAssets="all" Properties="RuntimeIdentifier=">
      <ReferenceOutputAssembly>false</ReferenceOutputAssembly>
      <CopyToOutputDirectory>Never</CopyToOutputDirectory>
      <CopyLocal>false</CopyLocal>
      <SkipGetTargetFrameworkProperties>true</SkipGetTargetFrameworkProperties>
      <ExcludeAssets>all</ExcludeAssets>
    </ProjectReference>
  </ItemGroup>

What's going on is I have a build tool in the solution, and dotnet publish is trying (incorrectly) to ship it. Almost minimized sample attached. If it were any smaller it wouldn't make any sense.
depbuildtool.zip

[teneko]

I can confirm that this behaviour is occuring, I have a synthetic project that only needs to be built but any

<None>
<CopyToPublishDirectory>PreserveNewest</CopyToPublishDirectory>
<!-- or -->
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</None>

is copied to projects with reference to synthetic project

<ProjectReference>
<ReferenceOutputAssembly>false</ReferenceOutputAssembly>
</ProjectReference>

[jhudsoncedaron]

Thanks to a random stackoverflow post, we now know that <Private>false</Private> works, but this isn't very sensible. It tells the compiler the assembly is in the GAC, which it isn't.

[teneko]

You made my day. Here the SO post for reference: https://stackoverflow.com/questions/26242937/what-does-the-private-setting-do-on-a-projectreference-in-a-msbuild-project-file. <Private>false</Private> can be applied to <ProjectReference />.

When you don't need any CopyTo[..]Directory functionality in synthetic project, an another workaround might be:

<Project>

  <Import Sdk="Microsoft.NET.Sdk" Project="Sdk.props" />

  <PropertyGroup>
    <TargetFramework>netstandard2.0</TargetFramework>
    <!-- ... -->
  </PropertyGroup>

  <Import Sdk="Microsoft.NET.Sdk" Project="Sdk.targets" />

  <!--
  Overrides that return empty @(AllItemsFullPathWithTargetPath)
  -->

  <!-- https://github.com/dotnet/msbuild/blob/116af13e6760ebbb8466174201f1ebbc8df11dfa/src/Tasks/Microsoft.Common.CurrentVersion.targets#L4561 -->
  <Target Name="GetCopyToOutputDirectoryItems" />
  <!-- or -->
  <!--  https://github.com/dotnet/msbuild/blob/116af13e6760ebbb8466174201f1ebbc8df11dfa/src/Tasks/Microsoft.Common.CurrentVersion.targets#L4624 -->
  <Target Name="GetCopyToPublishDirectoryItems" />

</Project>

But I am not sure which further implications this workaround may have.

[teneko]

Hi, I want just to state, that <Private>false</Private> may not work when using <MSBuild Projects="$(MSBuildProjectFullPath)" Targets="Publish" Properties="$(_MSBuildProperties)" /> and project $(MSBuildProjectFullPath) have ProjectReferences that have <None><CopyToPublishDirectory>PreserveNewest</CopyToPublishDirectory></None>
. I've read the source code around https://github.com/dotnet/sdk/blob/master/src/Tasks/Microsoft.NET.Build.Tasks/targets/Microsoft.NET.Publish.targets and found the solution. You need to define _GetChildProjectCopyToPublishDirectoryItems=false so a example would be: <MSBuild Projects="$(MSBuildProjectFullPath)" Targets="Publish" Properties="TargetFramework=$(TargetFramework);_GetChildProjectCopyToPublishDirectoryItems=false" />

[benvillalobos]

cc @dsplaisted for the nuget/CLI/SDK sync

[JanKrivanek]

Investigated with @rainersigwald:

tl;dr;: Strictly speaking ReferenceOutputAssembly is behaving as it should (preventing the output assembly to be referenced and pulled to output folder by referencing project). However it might feel unexpected when referencing application (OutputType=exe), as other supplementary files (deps.json, runtimeconfig.json and mainly <app>.exe) are still copied to output folder.

Workaround: Specifying <Private>false</Private> metadata on such ProjectReference will make sure that those additional files (including the exe) are not pulle to output folder of referencing project. Note that it will as well block copying of any other files as for example those defined as <None Include="my-config.cfg" CopyToOutputDirectory="PreserveNewest" />

---

Background: The netcore application produces the .dll assembly, and few supplementary files (e.g. the .exe which is actually a native shim calling into the managed .dll), those supplementary files are added as None item by the sdk _ComputeNETCoreBuildOutputFiles target. This then down the road causes the files to be added to the output folder, as None items are not excluded when the ReferenceOutputAssembly=true, as it (as the names implies) concerns only to output assembly.

Changing the behavior so that even the None items are excluded could break come other scenarios that currently work (keeping to copy items explicitly added by user).

Possible fix: So the possible way of fixing this is to define another metadata (e.g. something like BuildAfter or EnforceBuildPrecedenceOnly), that would only caused the referenced project to build prior the current project, but wouldn't cause any (direct nor transitive) flow of outputs nor items.

Conclusion:
Such a fix is questionable as it needs changes on users side and proper documentation as well - so it's similar to guiding users bit by this to use the Private metadata (despit it has above mentioned limitations - but such a usecase should be very niche).

[jhudsoncedaron]

@JanKrivanek : In .NET 6, <Private>false</Private> didn't work on a <ProjectReference> but only on a <PackageRefernece>. Are you telling me that works now?

In case anybody is wondering, the use case is there's a compiler in the build tree that outputs .cs files that are consumed by other projects. (In one case it actually edits the output DLL instead).

[JanKrivanek]

@jhudsoncedaron It works - see #4371 (comment) for more context.
Btw. I'm seeing same behavior building from net7 and net6 (6.0.406), while targetting both of those.

[jhudsoncedaron]

@JanKrivanek : Ah. I just didn't have the rest of the items to set from the other thread.

[jhudsoncedaron]

The current sequence to reference a project for dependency purposes but not actually reference a project in the output is:

  <ItemGroup>
    <ProjectReference Include="../OtherProject/OtherProject.csproj" Properties="RuntimeIdentifier=;SelfContained=false">
      <ReferenceOutputAssembly>false</ReferenceOutputAssembly>
      <DisableTransitiveProjectReferences>true</DisableTransitiveProjectReferences>
      <SkipGetTargetFrameworkProperties>true</SkipGetTargetFrameworkProperties>
      <Private>false</Private>
      <ExcludeAssets>all</ExcludeAssets>
    </ProjectReference>
  </ItemGroup>

Each of these settings does a different piece of what is necessary to get a pure build order dependency.

• Properties="RuntimeIdentifier=": Sets the runtime identifier of the other project. You don't want to inherit it. (Blank is the any RID here; typing any doesn't work)
• Properties="SelfContained=false": Sets whether or not to build the other target as self contained or not
• <ReferenceOutputAssembly>false</ReferenceOutputAssembly> Don't reference the other assembly in your assembly
• <DisableTransitiveProjectReferences>true</DisableTransitiveProjectReferences> Don't take references on any projects it references either.
• <SkipGetTargetFrameworkProperties>true</SkipGetTargetFrameworkProperties> Don't care what the other assembly's .NET Runtime version is either.
• <Private>false</Private> Don't copy the build output of the other target to your target
• <ExcludeAssets>all</ExcludeAssets> Don't copy other files from the other target to your target (such as .deps.json, .runtimesettings.json, or anything from Action=CopyToOutputDirectory)

I have to admit I'm starting to get disgusted because this list grows over time and isn't discoverable.

[chipplyman]

We are switching to solution generation, and none of the solution generators we have found have support for sln-level ProjectDependencies.

We tried creating a shortcut OrderingOnly="true" attribute and a custom BeforeTargets="BeforeBuild" Target to update ProjectReferences with all the boilerplate described by @jhudsoncedaron but we are abandoning the effort. We encountered problems that we believe are related to special treatment of ProjectReference items within msbuild and VS. One of our dependencies happens to cause a namespace ambiguity that fails the build when it is treated as an actual reference, so we know that ReferenceOutputAssembly is being treated as true at build time even when we set it false BeforeBuild.

Interestingly, some kind of caching within VS allows our fix to work during a second VS build, but not on rebuild nor when using msbuild.exe.

We believe the issue is related to special treatment of ProjectReference items because we note in the .binlog that a single ProjectReference item creates many mirrored item types at project load time: AnnotatedProjects, ProjectReferenceWithConfiguration, _MSBuildProjectReference, _MSBuildProjectReferenceExistent, _ProjectReferenceTargetFrameworkPossibilities, _ProjectReferenceTargetFrameworkPossibilitiesOriginalItemSpec. Presumably one of these is actually used internally to handle the ReferenceOutputAssembly metadata, and our transformation occurs too late for that item to get updated appropriately.

[chipplyman]

@jhudsoncedaron I find that SelfContained and SkipGetTargetFrameworkProperties cause problems . Either of them will push the referenced project into a separate batch and a separate node in the dependency graph, causing it to build twice.

You can see this happening by searching the binlog for CopyFilesToOutputDirectory under($project Foo).

In our ecosystem this double build tends to cause transient build failures during Rebuild, if the Clean of one instance runs concurrently with the Build of the other. We have an AfterTargets="Build" action that consumes the output directory, and it will sometimes find that the output files have been deleted out from under it.

[jhudsoncedaron]

@chipplyman : There's a simple way to avoid this issue. Create a .sln file with only the final build targets in it, and build it with dotnet build -p:ShouldUnsetParentConfigurationAndPlatform=false I don't know why it works, but I know that it does.

[chipplyman]

@jhudsoncedaron for us, it's simpler to just omit those problematic properties. They don't seem to have any impact on the build when omitted. That may be only because we're still singly targeting .net 4.8 but for now it works.

[jhudsoncedaron]

@chipplyman : Ah. In my case it will not build. You are certainly getting away with it by targeting .net 4.8.

[chipplyman]

@jhudsoncedaron I just hope this discussion demonstrates to the msbuild team that there exists an urgent need for a project-level ordering-only dependency declaration feature.

[baronfel]

In some ways the Aspire AppHost is driven by this same concept. The AppHost uses ProjectReferences to service projects mostly as a way to 'track' the projects, and the only build-related thing it actually uses is implicit - when runing the AppHost project this implicitly causes the ProjectReferences to be built as well, so the AppHost can launch them. This could also be orchestrated by the AppHost similar to how Docker Compose and other orchestrators work.

cc @DamianEdwards @davidfowl for awareness.

[DamianEdwards]

Hmm, random thought, what if we had something like a <ProjectDependency /> item that effectively expanded to the <ProjectReference /> pattern shown in #4795 (comment)? I'm "dependent" on this other project but I don't "reference" it.

[jhudsoncedaron]

@DamianEdwards : I mean we could; but they still have to fix the .sln double build bug. (Bug doesn't exist when you build out of a project file; but there's no way to build n project files in a single build step where n > 1 except for a .sln file so most people just build their .sln file that has everything in it.)

[jhudsoncedaron]

@chipplyman : SelfContained and SkipGetTargetFrameworkProperties are required to cross compile.

[lvde0]

Seems that the workaround here does not work anymore. My referenced project uses a different TargetFramework than the referencing project.
Honestly quite surprised this doesn't work easily. There are lots of use cases for a pure build order dependency that doesn't pull in all the assemblies / symbols.

[doxxx]

I'm in the same boat as the OP. My solution contains an OutputType=EXE project that is referenced by another project to trigger custom (non-C#) source generation but I do not want the source generator EXE to appear in the output of the referencing project. Only the magic incantation in #4795 (comment) works, but that's just gross.