MSBuild up-to-date check on built projects takes 55% of clean build time - huge overhead when building coreclr tests

[4creators]

Checking if test projects are up-to-date in already built coreclr repo takes 55% of time of clean repo build. It may indicate that MSBuild overhead for building small projects is higher than actual build cost or that up-to-date check is extremely expensive. In normal development situations I would expect it will take small fraction of percent to do that. My experience with large native projects i.e. building Chrome or LLVM/Clang is that I can safely run build command on whole project after making small changes and expect that incremental builds will be fast. In case of managed projects it becomes a kind of nightmarish.

Steps to reproduce

1. Clone coreclr repo on Windows machine
2. Run build x64 Debug -skiptests from repo root
3. Run build-test x64 Debug from command root and note managed tests build timing (it will be displayed directly after managed tests are built and before wrappers and other components of tests)
4. Run build-test x64 Debug again and note managed build timing

Expected behavior

Build with all tests already built should take small fraction of time i.e. couple of percent at maximum.

Actual behavior

First build command run on my laptop takes 16:27.30 while second build which checks only if projects are up to date takes 09:04.63 on my laptop.

What I have noticed as well is that memory usage of VBCSCompiler reaches up to 5 GB, while top level MSBuild node memory usage grows to 1,8 GB and lower level nodes each cross 0,5 GB memory usage level - altogether at the peak build may consume up to 6GB of memory.

Environment data

MSBuild version is determined by BuildTools version used by repo and installed with VS (15.5.180.51428)
Windows 10 Pro x64
coreclr repo and build tools as of commit 41d23b05acac98c68d3b348c60bf151cfedd778f
Visual Studio 2017 v15.5.2
Processor Intel Core-i7 4700MQ
16GB RAM

[4creators]

Related issue tracking solution to the problem #3586