Need a way to indicate dependencies on other projects

[KirillOsenkov]

Currently MSBuild uses <ProjectReference> items to indicate dependencies between projects and there's logic in the common targets to ensure that the build order is correct and ProjectReferences are respected.

However it is my current understanding that this is specific to typical C# projects, and there's no MSBuild generic way to indicate a project's dependency on other projects. There also doesn't seem to be a way to hint to the solution build in which order to build the projects, apart from the ProjectReference mechanism.

If so, we need to design a generic way to indicate dependency across projects, to minimize work and initially building the projects in the right order vs. imperatively triggering a build for dependencies once we encounter the dependency.

[KirillOsenkov]

Example investigation:
microsoft/Git-Credential-Manager-for-Windows#529

Example fix 1:
microsoft/Git-Credential-Manager-for-Windows@83f2e7a
Here we imperatively trigger the build for dependencies instead of declaratively specifying them.

Example fix 2:
microsoft/Git-Credential-Manager-for-Windows@bfdf2b5
Here we need to make sure we enumerate the directory produced by our dependencies AFTER the dependencies have been built imperatively. If we enumerate the directory normally (during evaluation), it will run before the dependencies have been built. If we had a declarative mechanism to specify project dependencies the evaluation of the dependent project would start after the dependencies have been built, and so everything would work as expected, no surprises.

[rainersigwald]

Solutions allow specifying build order dependencies, which MSBuild respects. And ProjectReference works for vcxproj too, at least . . . what have you seen to make you think otherwise?

[whoisj]

Well, we've seen that solutions only work if the VS toolchain is relied upon. If a dev makes a project from scratch, there's very little rolling to assist with dependencies and easy way to debug / understand the problem.

Additionally, most code bases are too large for solutions

[chrispat]

@rainersigwald we regularly get complaints of project build order in solutions not being respected by msbuild.

[rainersigwald]

@chrisrpatterson Send them to us, please. Any such problem is a bug in MSBuild.

[whoisj]

@rainersigwald Hi, I was sent here by @KirillOsenkov and @chrisrpatterson 😏

So the net issue was that I had a custom script that was being replaced with MSBuild. The script did the following:

1. Use Pandoc to convert .md into .html for help files.
2. Collect the output of a bunch of related projects.
3. Combine the outputs into an installer using InnoSetup.
4. Generate a ZIP xcopy-able alternative to the installer.

The primary motivations for moving away from the script were:

1. The script was fairly opaque.
2. Binary signing wasn't something we could do using existing tool set from a script.
3. Custom scripts aren't supposed to be easier/better/more reliable than build systems.

Since I was replacing a .cmd BATCH script, I started with an empty .proj file and built it up from there. I believed that should be completely doable, and it mostly worked. Dependencies and therefore build ordering didn't.

I'd attempted to establish dependencies via stack of ProjectReference nodes like the one below

    <ProjectReference Include="..\Cli-Askpass\Cli-Askpass.csproj">
      <Project>{19770407-B33A-4EBE-92B8-04C93F43CAE0}</Project>
      <Name>Cli-Askpass</Name>
    </ProjectReference>

But they were seemingly ignored, and every build was failing. The solution (found by @KirillOsenkov) was <MSBuild Projects="@(ProjectReference)" Targets="Build" />.

TL:DR:

The outcome was that MSBuild only cared about project dependencies when invoked from Visual Studio. Why? When building from the "Developer Command Prompt for VS 2017" and/or from AppVeyor, MSBuild seemed to completely ignore the dependencies, with absolutely no indication as to why.

[rainersigwald]

The outcome was that MSBuild only cared about project dependencies when invoked from Visual Studio. Why?

Because the solution lies to Visual Studio and claims that this new project is a normal .csproj solution line: https://github.com/Microsoft/Git-Credential-Manager-for-Windows/blob/b77f58639f18a6525f07967ef2006ff1a4b9f9bb/GitCredentialManager.sln#L49

That confused VS into thinking that the ProjectReferences would do something, because it has a parallel-but-inaccurate understanding of the nature of references. MSBuild sees that nothing consumes @(ProjectReference) and does nothing with it.

This is a bug in VS, though probably not one worth fixing.

I'd love to get your feedback on how we can make the documentation on how to implement ProjectReferences more obvious!

[whoisj]

I'd love to get your feedback on how we can make the documentation on how to implement ProjectReferences more obvious!

Firstly, I would have love to have know that docs on GitHub even existed. I trolled Bing and Google and came up mostly empty. In fact, the only reason I used ProjectReferences and I "lied" to Visual Studio is that I was attempting to make anything work using one part first-principles and one part hack-it.

Secondly, does msbuild even have a help command? I mean a deep diving or, at least, redirecting to official documentation command?

[rainersigwald]

What kinds of search terms were you using? Maybe we can SEO a bit.

Secondly, does msbuild even have a help command? I mean a deep diving or, at least, redirecting to official documentation command?

No, and we should. Filed #2892 as the minimum.

[whoisj]

What kinds of search terms were you using? Maybe we can SEO a bit.

"msbuild project ordering", "msbuld project dependencies", "msbuild choosing last project built"

There were lots of stackoverflow.com threads of people with questions about the same topic in the results, none of which had sufficient answers. 😒

There were lots of examples of how to order Targets, and even conflicting reports of how to use AfterTargets and BeforeTargets, and let's be honest the "wording" of those names can be looked at two different ways (read: very confusing). Still, nothing to unblock me - except on blog about adding ProjectReferences and using that .sln hack which mostly unblocked me.

Remember, I was adamant that the final solution did not require Visual Studio to be installed or even available. I work on Visual Studio for a living, and I love Visual Studio, but I am/was not willing to accept that I needed an IDE to build source code. 😏

[japj]

If I may add to this discussion, we have a software product that consists out of 30+ solutions (containing a mix of c++/c#/java/etc) and we created our own msbuild project files that orchestrate building these “subsystems” solutions in the right order (using a similar dependency mechanism as project references does in c#).
It would be really appreciated it if there is a better/official/documented way of managing and building dependencies of (also non c#) projects.

[breyed]

@rainersigwald When considering whether MSBuild should lean on solution-based dependencies, it's worth keeping in mind that MSBuild behavior differs from that of Visual Studio when working with dependencies specified in a solution. In particular, if a solution specifies a dependency with ProjectSection(ProjectDependencies) = postProject and the project being depended on has an item with <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>, MSBuild will include that item in the dependent project's output, whereas Visual Studio will not.

This behavior caused surprise results on the automated build for us when working with a project that was a dependency because it needed to build so that MSBuild would use it as a custom task. To avoid the unwanted files in the dependent project's output, we ended up building the dependency using a MSBuild task inside our custom target. In this way, MSBuild does a nice job of supporting arbitrary dependencies.

[JanKrivanek]

Related: #4795

[KirillOsenkov]

That's a circular reference to the same bug @JanKrivanek :)

[JanKrivanek]

That's a circular reference to the same bug @JanKrivanek

Oops :-)
Edited

[chipplyman]

Solutions allow specifying build order dependencies, which MSBuild respects.

Sln-level dependencies do not scale in environments with multiple .slns, nor those using solution generation.

[chipplyman]

Our use cases for this feature are:

1. CodeGen. We need a way to say "the codegen project builds before any project that executes it", without shipping the codegen exe with the consuming projects.
2. Plugins. We need a way to say "the plugins all build before the exe container that exercises them", without allowing any references to plugin-specific code to sneak into the container project, and without placing the plugin dlls in the exe output directory.

We use multiple slns, and it has proven very difficult, brittle, and unintuitive to maintain the sln-level ordering dependencies across all of them.
We are moving towards solution generation, and it has proven impossible to find a solution generator that supports sln-level ordering dependencies.

[KirillOsenkov]

Almost feels like the need for a new top-level syntax:

<Project>

  <Dependencies>
    <Dependency Include="..\path\to\Other.proj" />
    <Dependency Include="..\path\to\Another.proj" OutputItemType="OutputFromAnotherProject" />
  </Dependencies>

</Project>

And MSBuild/VS would "peek" into the project to get a list of dependencies, take that information into account in the metaproj to sort projects topologically (potentially overriding the information coming from solution dependencies in VS), such that when the project begins evaluation and execution it is guaranteed that the dependencies have been built.

If the project is built standalone, we'd peek as well, before evaluation, build the dependent projects, then evaluate and build this one.

Realistically the above sounds like science fiction, doubt this will ever happen. But one can brainstorm!

The ProjectReference protocol and mechanism would remain untouched, it's just when ResolveProjectReferences calls into referenced projects to be built, they will have been built already. This would make sure that at the time of evaluating a project, it's project references have been built already.

[chipplyman]

I've considered making a local partial copy of the ResolveProjectReferences target and transforming it to reference a new custom Item type, to implement this functionality without taking all the other cruft that comes with ProjectReferences. AFAICT at the core, ResolveProjectReferences effectively just says "build this project now instead of waiting for it to happen as part of the top-level list". I don't know whether the build system will handle that correctly; I fear it might try to build the same project twice (which is a dealbreaker) if we can't get the ordering metadata into the project dependency graph up front - and of course we'd need to account for the fact that msbuild and VS have entirely distinct mechanisms for this functionality.

[KirillOsenkov]

The problem is that you often want the references to be built by the time the evaluation starts. The current project reference infrastructure goes and ensures the references are built after the evaluation of our project is already done and we're in the middle of building the current project's targets (e.g. everything that comes before ResolveProjectReferences will run before the references have been built).