MSBuild (Task) Does Not Expose Several Switches

[aolszowka]

We use the MSBuild (Task) (https://docs.microsoft.com/en-us/visualstudio/msbuild/msbuild-task?view=vs-2019) pretty extensively within our build system to invoke additional solution files for build. We do this because as per the documentation:

Unlike using the Exec task to start MSBuild.exe, this task uses the same MSBuild process to build the child projects. The list of already-built targets that can be skipped is shared between the parent and child builds. This task is also faster because no new MSBuild process is created.

However this task does not appear to expose several newer switches (https://docs.microsoft.com/en-us/visualstudio/msbuild/msbuild-command-line-reference?view=vs-2019) specifically:

• /restore - Can be worked around as per PackageReference is not resolved while it works in Visual Studio #3000 (comment) by calling ONLY /t:restore and nothing else This is also tracked in MSBuild-task equivalent of "msbuild.exe /restore /t:Target" #2811
• /graphBuild
• /isolateProjects

These are just the ones we care about; looking at the docs it looks like there are several others (maxCpuCount, warnAsError, etc) however we have not had a need to use them at this time.

Are there individual requests for this? is there a work around we can apply (like /restore's work around?)

[rainersigwald]

Restore is #2811. I don't think we have one for the graph options: this is now that request.

They're all difficult or impossible to implement because of a conceptual problem: these are options that change how the whole build works. How does that apply to only part of the build?

For example, what happens if I have Project A -(isolateProjects=true)-> Project B and Project A -(default)-> Project C -> Project B? How should B be built?

Likewise, /restore means first do a restore operation, then throw away all state and do the requested operation. Is it possible to provide a consistent view if someone builds two different targets in a project with /restore=true in separate invocations? What about two different projects building a project both with restore enabled?

[aolszowka]

Good points; some of it is an artifact of our current build system in which we use MSBuild to further coordinate separate technologies into a single unified build. Each technology generally will have its own Solution file, the MSBuild scripts will then call build on the individual technology's solution file (via the MSBuild Task); heres a somewhat cut-down version (this script is much longer than the below):

  <Target Name="BuildAll">
    <AppendToCCNetListener CCNetListenerFilePath="$(CCNetListenerFile)" Messages="Build Is Starting (Triggered by $(CCNetRequestSource))" Condition="$(CCNetListenerFile) != ''" />

    <!--Package the Reports for Distribution-->
    <AppendToCCNetListener CCNetListenerFilePath="$(CCNetListenerFile)" Messages="Starting PackageReports" Condition="$(CCNetListenerFile) != ''" />
    <!--<CallTarget Targets="PackageReports" />-->
    <!--HACK HACK HACK HACK-->
    <!--Because Microsoft does not yet provide a x64 Compatible RDL Compiler-->
    <!--Invoke the Build System with MSBuild x86 and hope for the best!-->
    <Exec Command='"$(MSBuildx86Exectuable)" $(InternalCodeRoot)\Dotnet\BuildSystems\scripts\ComputersUnlimited.Build.All.msbuild /m /t:PackageReports /p:ProductionCodeRoot=$(ProductionCodeRoot);InternalCodeRoot=$(InternalCodeRoot);InternalXRoot=$(InternalXRoot);BuildOutput=$(BuildOutput)'/>
    <AppendToCCNetListener CCNetListenerFilePath="$(CCNetListenerFile)" Messages="Finished PackageReports" Condition="$(CCNetListenerFile) != ''" />

    <!--First Build the Synergy Applications-->
    <AppendToCCNetListener CCNetListenerFilePath="$(CCNetListenerFile)" Messages="Starting BuildSynergy" Condition="$(CCNetListenerFile) != ''" />
    <CallTarget Targets="BuildSynergy" />
    <AppendToCCNetListener CCNetListenerFilePath="$(CCNetListenerFile)" Messages="Finished BuildSynergy" Condition="$(CCNetListenerFile) != ''" />

    <!--Next Build CUXFPL-->
    <AppendToCCNetListener CCNetListenerFilePath="$(CCNetListenerFile)" Messages="Starting BuildCUXFPL" Condition="$(CCNetListenerFile) != ''" />
    <CallTarget Targets="BuildCUXFPL" />
    <AppendToCCNetListener CCNetListenerFilePath="$(CCNetListenerFile)" Messages="Finished BuildCUXFPL" Condition="$(CCNetListenerFile) != ''" />

    <!--Next Build Fusion-->
    <AppendToCCNetListener CCNetListenerFilePath="$(CCNetListenerFile)" Messages="Starting BuildFusion" Condition="$(CCNetListenerFile) != ''" />
    <CallTarget Targets="BuildFusion" />
    <AppendToCCNetListener CCNetListenerFilePath="$(CCNetListenerFile)" Messages="Finished BuildFusion" Condition="$(CCNetListenerFile) != ''" />

    <!--Next Build TIMSNET-->
    <AppendToCCNetListener CCNetListenerFilePath="$(CCNetListenerFile)" Messages="Starting BuildTIMSNET" Condition="$(CCNetListenerFile) != ''" />
    <CallTarget Targets="BuildTIMSNET" />
    <AppendToCCNetListener CCNetListenerFilePath="$(CCNetListenerFile)" Messages="Finished BuildTIMSNET" Condition="$(CCNetListenerFile) != ''" />

    <!--Next Build Synergy.NET-->
    <AppendToCCNetListener CCNetListenerFilePath="$(CCNetListenerFile)" Messages="Starting BuildSynergyDotNet" Condition="$(CCNetListenerFile) != ''" />
    <CallTarget Targets="BuildSynergyDotNet" />
    <AppendToCCNetListener CCNetListenerFilePath="$(CCNetListenerFile)" Messages="Finished BuildSynergyDotNet" Condition="$(CCNetListenerFile) != ''" />

    <Message Text="Build is Completed" Importance="high"/>
    <AppendToCCNetListener CCNetListenerFilePath="$(CCNetListenerFile)" Messages="Build is Completed" Condition="$(CCNetListenerFile) != ''" />
  </Target>

Long term I am trying to replace this pattern to a single solution file (we're calling it Megalodon, owing to its size of 5,000+ Projects) which would make some of these issues go away, and also increase Core/Resource Utilization (64+ Cores, 128GB+ RAM). The struggle we are encountering is that some of these newer features (for example /graphBuild) do not properly work with all project types yet (See #5159) .

This is where the current design would shine, this is because we could set individual technologies (which do properly support these advanced features) to use /graphBuild to at least get some advantage.

Your point still stands though:

What happens if I have Project A -(isolateProjects=true)-> Project B and Project A -(default)-> Project C -> Project B? How should B be built?

While the technologies for us are pretty well isolated we have been trying to encourage more and more "cross technology pollination" where they do interop into each other. In the current build today this results in overbuild, which is why Megalodon is being pushed.

Would it be helpful to start a wiki page to describe these issues with these new switches to point other developers to? Your description is very good, unfortunately I am not smart enough to give you a viable solution :(

[solvingj]

Of note, I also thought I wanted to be able to execute with preprocess recently, but am now trying a different route.

[Forgind]

I don't think this is a good solution, but could we make executing the MSBuild task with switches redirect build outputs to a special folder (different depending on which switches were enabled) and allow the user to read from there? Then we would build each project multiple times if they specify different switches (sometimes wasteful) but should lead to a correct build? We'd have to make sure the outputs were only changed for that invocation because we don't want the project to be considered up-to-date.