Apparent race condition copying files from sync.cmd

[ghost]

Opened on behalf of @MattGal

Possibly related to these recent PRs?
dotnet/corefx#31097
dotnet/corefx#31139
(similar symptoms, BuildVersion.props called out in one log)

Warnings: 1
Status Message: failed

Build : Master - 20180718.04 (Product Build)
Failing configurations:

• Windows
  • AllConfigurations-x64
    Mission Control Build Info

[weshaggard]

@ViktorHofer these failures look related to your binplacepackages.depproj changes from dotnet/corefx#30897.

2018-07-18T12:09:13.7376150Z E:\A\_work\64\s\corefx\Tools\FrameworkTargeting.targets(283,5): error MSB3027: Could not copy "E:\A\_work\64\s\corefx\packages\System.Diagnostics.Tools\4.3.0\ref\netstandard1.0\System.Diagnostics.Tools.dll" to "E:\A\_work\64\s\corefx\bin/ref/netstandard1.4/System.Diagnostics.Tools.dll". Exceeded retry count of 10. Failed.  [E:\A\_work\64\s\corefx\external\binplacePackages\binplacePackages.depproj]
2018-07-18T12:09:13.7378369Z E:\A\_work\64\s\corefx\Tools\FrameworkTargeting.targets(283,5): error MSB3021: Unable to copy file "E:\A\_work\64\s\corefx\packages\System.Diagnostics.Tools\4.3.0\ref\netstandard1.0\System.Diagnostics.Tools.dll" to "E:\A\_work\64\s\corefx\bin/ref/netstandard1.4/System.Diagnostics.Tools.dll". The process cannot access the file 'E:\A\_work\64\s\corefx\bin\ref\netstandard1.4\System.Diagnostics.Tools.dll' because it is being used by another process. [E:\A\_work\64\s\corefx\external\binplacePackages\binplacePackages.depproj]
2018-07-18T12:09:13.7401526Z E:\A\_work\64\s\corefx\Tools\FrameworkTargeting.targets(283,5): error MSB3027: Could not copy "E:\A\_work\64\s\corefx\packages\System.IO\4.3.0\ref\netstandard1.3\System.IO.dll" to "E:\A\_work\64\s\corefx\bin/ref/netstandard1.4/System.IO.dll". Exceeded retry count of 10. Failed.  [E:\A\_work\64\s\corefx\external\binplacePackages\binplacePackages.depproj]
2018-07-18T12:09:13.7410764Z E:\A\_work\64\s\corefx\Tools\FrameworkTargeting.targets(283,5): error MSB3021: Unable to copy file "E:\A\_work\64\s\corefx\packages\System.IO\4.3.0\ref\netstandard1.3\System.IO.dll" to "E:\A\_work\64\s\corefx\bin/ref/netstandard1.4/System.IO.dll". The process cannot access the file 'E:\A\_work\64\s\corefx\bin\ref\netstandard1.4\System.IO.dll' because it is being used by another process. [E:\A\_work\64\s\corefx\external\binplacePackages\binplacePackages.depproj]
2018-07-18T12:09:13.7479636Z E:\A\_work\64\s\corefx\Tools\FrameworkTargeting.targets(283,5): error MSB3027: Could not copy "E:\A\_work\64\s\corefx\packages\System.Security.Cryptography.Primitives\4.3.0\ref\netstandard1.3\System.Security.Cryptography.Primitives.dll" to "E:\A\_work\64\s\corefx\bin/ref/netstandard1.4/System.Security.Cryptography.Primitives.dll". Exceeded retry count of 10. Failed.  [E:\A\_work\64\s\corefx\external\binplacePackages\binplacePackages.depproj]
2018-07-18T12:09:13.7484374Z E:\A\_work\64\s\corefx\Tools\FrameworkTargeting.targets(283,5): error MSB3027: Could not copy "E:\A\_work\64\s\corefx\packages\System.Diagnostics.Tracing\4.3.0\ref\netstandard1.3\System.Diagnostics.Tracing.dll" to "E:\A\_work\64\s\corefx\bin/ref/netstandard1.4/System.Diagnostics.Tracing.dll". Exceeded retry count of 10. Failed.  [E:\A\_work\64\s\corefx\external\netstandard\netstandard.depproj]
2018-07-18T12:09:13.7486930Z E:\A\_work\64\s\corefx\Tools\FrameworkTargeting.targets(283,5): error MSB3027: Could not copy "E:\A\_work\64\s\corefx\packages\System.Globalization.Calendars\4.3.0\ref\netstandard1.3\System.Globalization.Calendars.dll" to "E:\A\_work\64\s\corefx\bin/ref/netstandard1.4/System.Globalization.Calendars.dll". Exceeded retry count of 10. Failed.  [E:\A\_work\64\s\corefx\external\netstandard\netstandard.depproj]
2018-07-18T12:09:13.7490214Z E:\A\_work\64\s\corefx\Tools\FrameworkTargeting.targets(283,5): error MSB3021: Unable to copy file "E:\A\_work\64\s\corefx\packages\System.Diagnostics.Tracing\4.3.0\ref\netstandard1.3\System.Diagnostics.Tracing.dll" to "E:\A\_work\64\s\corefx\bin/ref/netstandard1.4/System.Diagnostics.Tracing.dll". The process cannot access the file 'E:\A\_work\64\s\corefx\bin\ref\netstandard1.4\System.Diagnostics.Tracing.dll' because it is being used by another process. [E:\A\_work\64\s\corefx\external\netstandard\netstandard.depproj]
2018-07-18T12:09:13.7493993Z E:\A\_work\64\s\corefx\Tools\FrameworkTargeting.targets(283,5): error MSB3021: Unable to copy file "E:\A\_work\64\s\corefx\packages\System.Globalization.Calendars\4.3.0\ref\netstandard1.3\System.Globalization.Calendars.dll" to "E:\A\_work\64\s\corefx\bin/ref/netstandard1.4/System.Globalization.Calendars.dll". The process cannot access the file 'E:\A\_work\64\s\corefx\bin\ref\netstandard1.4\System.Globalization.Calendars.dll' because it is being used by another process. [E:\A\_work\64\s\corefx\external\netstandard\netstandard.depproj]
2018-07-18T12:09:13.7496769Z E:\A\_work\64\s\corefx\Tools\FrameworkTargeting.targets(283,5): error MSB3021: Unable to copy file "E:\A\_work\64\s\corefx\packages\System.Security.Cryptography.Primitives\4.3.0\ref\netstandard1.3\System.Security.Cryptography.Primitives.dll" to "E:\A\_work\64\s\corefx\bin/ref/netstandard1.4/System.Security.Cryptography.Primitives.dll". The process cannot access the file 'E:\A\_work\64\s\corefx\bin\ref\netstandard1.4\System.Security.Cryptography.Primitives.dll' because it is being used by another process. [E:\A\_work\64\s\corefx\external\binplacePackages\binplacePackages.depproj]
2018-07-18T12:09:13.7740765Z 
2018-07-18T12:09:13.7743193Z Build FAILED.

https://devdiv.visualstudio.com/DefaultCollection/DevDiv/Default/_build/results?buildId=1870127&view=logs

I've seen a number of official builds and CI legs hitting this randomly.

@ericstj it seems like it is copying over the dependencies or something from all the different configurations. Which is racing with other depproj's like the netstandard one.

[weshaggard]

All configurations CI leg in dotnet/corefx#31160 failed https://ci3.dot.net/job/dotnet_corefx/job/master/job/windows-TGroup_all+CGroup_Debug+AGroup_x64+TestOuter_false_prtest/14278/ with similar issue.

[ViktorHofer]

Hmm. It seems the assembly references of System.Memory race with the other depproj files. This only happens for the following configurations as other configuration don't have assembly references for System.Memory:

netstandard1.1;
netstandard1.2;
netstandard1.3;
netstandard1.4;
netstandard1.5;
netstandard1.6;

Looking further...

[weshaggard]

IIRC our depproj's shouldn't be doing a deep copy and should only copy the direct PackageReferences but that doesn't seem to be happening.

[ViktorHofer]

Right, I just checked and BinPlaceConfiguration directives clearly do deep copies.

[weshaggard]

I just submitted dotnet/corefx#31167 to fix the deep copy for binplacepackages and harvestpackages projects which shouldn't be deep.