[Bug]: Investigate why IdenticalSubmissionsShouldCompleteAndNotHangTheBuildOnMissingTargetExceptions started failing #9245
Labels
Area: Our Own Build
Problems affecting the build or build infrastructure of the MSBuild repo itself.
bug
triaged
Comments
This was referenced Sep 21, 2023
AR-May
added
Area: Our Own Build
JaynieBai
pushed a commit
that referenced
this issue
Oct 11, 2023
…ngTargetExceptions (#9253) Fixes #9245 Context The test was disabled to unblock PR CI. Changes Made Increased the relevant timeout. Testing The test is reliably passing now. Notes This turned out to be an issue with the sleep command we use on Windows. In some cases PowerShell can take a super long time to start. I have been able to reproduce locally by enabling Fusion logging. Thread times of the powershell process: image We spend almost 10 seconds just loading assemblies, so the timeout of 10 seconds for the entire build was not enough. I don't have a full understanding of the mechanism that slows down PowerShell this much. At this point I'm happy we were able to confirm it's not a product issue, although I'm wondering if there is a better and more light-weight sleep command we could use on Windows instead (e.g. ping 127.0.0.1 -n <seconds>). Reviewers please opine. EDIT: In my trace, file system operations block extensively with wdfilter.sys on the stack, so the likely explanation for the issue appearing all of a sudden is a Defender update.
bulatgrzegorz
pushed a commit
to bulatgrzegorz/selective-condition-evaluator
that referenced
this issue
Oct 16, 2023
…ngTargetExceptions (dotnet#9253) Fixes dotnet#9245 Context The test was disabled to unblock PR CI. Changes Made Increased the relevant timeout. Testing The test is reliably passing now. Notes This turned out to be an issue with the sleep command we use on Windows. In some cases PowerShell can take a super long time to start. I have been able to reproduce locally by enabling Fusion logging. Thread times of the powershell process: image We spend almost 10 seconds just loading assemblies, so the timeout of 10 seconds for the entire build was not enough. I don't have a full understanding of the mechanism that slows down PowerShell this much. At this point I'm happy we were able to confirm it's not a product issue, although I'm wondering if there is a better and more light-weight sleep command we could use on Windows instead (e.g. ping 127.0.0.1 -n <seconds>). Reviewers please opine. EDIT: In my trace, file system operations block extensively with wdfilter.sys on the stack, so the likely explanation for the issue appearing all of a sudden is a Defender update.
MichalPavlik
pushed a commit
that referenced
this issue
Oct 17, 2023
…ngTargetExceptions (#9253) Fixes #9245 Context The test was disabled to unblock PR CI. Changes Made Increased the relevant timeout. Testing The test is reliably passing now. Notes This turned out to be an issue with the sleep command we use on Windows. In some cases PowerShell can take a super long time to start. I have been able to reproduce locally by enabling Fusion logging. Thread times of the powershell process: image We spend almost 10 seconds just loading assemblies, so the timeout of 10 seconds for the entire build was not enough. I don't have a full understanding of the mechanism that slows down PowerShell this much. At this point I'm happy we were able to confirm it's not a product issue, although I'm wondering if there is a better and more light-weight sleep command we could use on Windows instead (e.g. ping 127.0.0.1 -n <seconds>). Reviewers please opine. EDIT: In my trace, file system operations block extensively with wdfilter.sys on the stack, so the likely explanation for the issue appearing all of a sudden is a Defender update.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Issue Description
The test started failing on PR CI pipeline. Suspecting timing issues.
Steps to Reproduce
https://dev.azure.com/dnceng-public/public/_build/results?buildId=414226&view=results
Expected Behavior
Test is reliable.
Actual Behavior
Test fails in PR builds.
Analysis
No response
Versions & Configurations
No response
The text was updated successfully, but these errors were encountered: