"Broken pipe" errors from tasks calling Process.Start() in latest 17.4.0-preview MSBuild

[MattGal]

Issue Description

As part of investigating dotnet/arcade#10311, I realized that there seems to be some kind of change in MSBuild, where build tasks that call Process.Start fail due to non-zero exit code. Digging in, all the errors were from scripts being run getting "broken pipe" type errors on Linux. I swapped out the entire "bad" .NET Core runtime used into the "good" SDK and did not get a repro; this leads me to believe (along with the only repro beign in a build task) that this is an MSBuild bug. I've leveraged a custom version of the .NET Arcade task that hit the original issue as a repro.

Steps to Reproduce

(Assuming the files from repro.zip are copied to c:\scratch\hosting-bug-repro)

To see the expected behavior:
Happy SDK 7.0.100-rc.1.22377.7 does not repro the problem (Version 7.0.22.36704 @Commit: aafa91036e1efd3d4dcb67eeb261cb6d8f774685)
Happy SDK has MSBuild version 17.4.0-preview-22366-04+d2871ca13

1. docker run -v c:/scratch/hosting-bug-repro:/repro -it mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-20220808192747-20889cf
2. cd /repro && ./dotnet-install.sh -version 7.0.100-rc.1.22377.7
3. /root/.dotnet/dotnet build ./repro.proj

Sad SDK 7.0.100-rc.1.22379.1 repros the problem (Version 7.0.22.37802 @Commit: 97749c034e3a94c03c2ebfef94fa6aa4b257488b)
Sad sdk has MSBuild version 17.4.0-preview-22378-04+827c1bf9c

To reproduce the bug:

1. docker run -v c:/scratch/hosting-bug-repro:/repro -it mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-20220808192747-20889cf
2. cd /repro && ./dotnet-install.sh -version 7.0.100-rc.1.22379.1
3. /root/.dotnet/dotnet build ./repro.proj

repro.zip

Expected Behavior

Build succeeds as many times as run

Actual Behavior

errors like this:

**************************************************
  ****??*  Standard Err:
   /repro/dotnet-install.sh: line 53: printf: write error: Broken pipe

  **************************************************

[baronfel]

@MattGal can you try rerunning your impacted build with the environment variable DOTNET_CLI_DO_NOT_USE_MSBUILD_SERVER set to true? There was a bug on linux in the new MSBuild server functionality that was addressed in #7860 (though that may not have flowed all the way yet).

[MattGal]

@MattGal can you try rerunning your impacted build with the environment variable DOTNET_CLI_DO_NOT_USE_MSBUILD_SERVER set to true? There was a bug on linux in the new MSBuild server functionality that was addressed in #7860 (though that may not have flowed all the way yet).

Yup, setting this true/false works like an on/off switch for the repro... feel free to close this as dupe of that.

[baronfel]

Great! Well, not great, but I'm glad you're unblocked. Can you make a note to try reverting that change in like a week? We'd like the MSBuild Server daemon to be the default as much as possible once the kinks are worked out.

[MattGal]

Can you make a note to try reverting that change in like a week?

In this case the changes never got merged to take a newer SDK, so given there's a subtle msbuild bug in this particular SDK I think all we have to do is wait for the fixed version.