Make cache queries parallel

[cdmihai]

Can review the commits individually. Ignore whitespace helps.

Context

The project cache was being queried serially. Oops.
This is because the monolithic BuildManager._syncLock was being held during the cache query, thus serializing all access.

Changes Made

Implements the 2nd option from the design doc: https://gist.github.com/cdmihai/0955cb217b2cbd66e18c89b20bf68319#2-reuse-how-the-buildmanager-offloads-parallel-work-to-build-nodes

• Reverted code of ExecuteSubmission to what it was before project cache plugins.
• Changed ExecuteSubmission to either issue a cache request or a build request (until now it only issues build requests)
• The cache request is sent to the ProjectCacheService which submits it to the thread pool. This achieves parallelization of cache requests
• Each cache request, on its own thread:
  • evaluates the project if necessary
  • does the cache query
• When a cache request finishes in the ProjectCacheService it is posted back on the BuildManager work queue thread and is handled by either skipping the build or doing a real build.

Design time builds were a pain to get right this time. Previously design time builds were easy to deal with because the BuildManager detected them early enough. Now they get detected later in the project cache service. The build manager detects this and shuts the cache service off when design time builds are detected.

Testing

Added a parallel stress test. This should be a good test to both ensure the cache is queried in parallel and to stress test the concurrency in the engine.

Risk assessment

This should make the non project cache logic less risky than it was before, since I took the project cache logic out of BuildManager and moved it to the ProjectCacheService.

[rainersigwald]

Overall LGTM, a few questions.

[cdmihai]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).