SuperPMI benchmarks run should enable tiered compilation and pgo

[AndyAyersMS]

Currently we run the benchmarks SPMI collection with TC disabled (and R2R disabled), and just run a single iteration. Among other things this means we're not collecting any OSR instances.

This differs from the product default that has TC enabled and now OSR. It would be nice to find a way to collect data that more closely represents how these benchmarks are actually run in the lab and by customers. This likely requires allowing BDN to go something closer to its normal iteration strategy, which in turn may end up taking a much longer time (eg a full run takes ~16+ hours on one machine; I don't know how many partitions we create). I think we can tamp down the final iteration count and just run one so perhaps it will not be too bad.

As a bonus it would also be nice to enable a similar collection with dynamic PGO.

There is a small risk that SPMI + checked jit (if we use one) will create altered timing and so the collection still won't reflect the actual usage (eg methods may tier up more slowly and so impact BDN's iteration strategy).

category:eng-sys
theme:super-pmi
skill-level:intermediate
cost:medium
impact:medium

[dotnet-issue-labeler]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[ghost]

Tagging subscribers to this area: @JulieLeeMSFT
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Currently we run the benchmarks SPMI collection with TC disabled (and R2R disabled), and just run a single iteration. Among other things this means we're not collecting any OSR instances.

This differs from the product default that has TC enabled and now OSR. It would be nice to find a way to collect data that more closely represents how these benchmarks are actually run in the lab and by customers. This likely requires allowing BDN to go something closer to its normal iteration strategy, which in turn may end up taking a much longer time (eg a full run takes ~16+ hours on one machine; I don't know how many partitions we create). I think we can tamp down the final iteration count and just run one so perhaps it will not be too bad.

As a bonus it would also be nice to enable a similar collection with dynamic PGO.

There is a small risk that SPMI + checked jit (if we use one) will create altered timing and so the collection still won't reflect the actual usage (eg methods may tier up more slowly and so impact BDN's iteration strategy).

Author:	AndyAyersMS
Assignees:	-
Labels:	area-CodeGen-coreclr, untriaged
Milestone:	-