Performance triage for JVM smoketest

[stuhood]

Before recommending that users use the new backend, we'd like to ensure that they get a good first impression. We should do a few days of performance triage to set a bar and identify potential problem areas to fix during M2.

[stuhood]

Opened #13787 and #13788 after an initial round of profiling.

Because #13787 causes a fair amount of unnecessary IO, it will likely muddy actual profiling of the JVMs being launched... but that's probably the next step. The actual sandbox creation time (see #13787) isn't horrible, but that's likely because the repository we're testing with is fairly small.

I'd also like to get a ballpark for how much total compilation time sbt sees for the test repo, so we can know where we stand.

[stuhood]

I'd also like to get a ballpark for how much total compilation time sbt sees for the test repo, so we can know where we stand.

I've done some comparison here and... there is work to do! No surprise there, but fixing (at least) both of the above issues will be necessary to get to a better baseline.

My very rough comparison involved:

• For sbt, twice in a row:
  1. git clean -fdx
  2. start an sbt --timings shell
  3. time a run of compile
• For pants, twice in a row:
  1. git clean -fdx
  2. run ./pants --no-local-cache dependencies :: to prime pantsd with analysis
     • this is slightly apples-to-oranges, since this is more work than sbt does to start a shell. but the analysis is very cacheable, so including it in the run (even a cold run) is arguably not valid
  3. time a run of ./pants --no-local-cache check ::

[stuhood]

Also obvious in here is that in macOS we're running afoul of xprotect (it keeps one core busy): I expect that we are repeatedly re-fingerprinting the coursier binary (but haven't confirmed). #12716 would help with that by creating it once.

EDIT: Yea, confirmed. And this also affects fetching artifacts with coursier (such that each fetch first applies xprotect to coursier, adding 3 or 4 seconds).

[stuhood]

With the above two issues addressed, I did a bit of profiling of the JVMs for javac and scalac, and was able to confirm that a significant fraction of their runtime is currently being spent on IO. So I expect that both to resolve the xprotect issue mentioned above, and to reduce IO contention in general, we should implement #12716.

[stuhood]

Going to call this complete. #13435 is the JVM-specific ticket for #12716 and any other remediation that we might need to do here. Will add additional details there.