Reconsider normal/no_tiered_compilation jobs

[jakobbotsch]

We currently submit all innerloop and outerloop jobs twice, in the normal and the no_tiered_compilation environments. It would be useful to monitor whether we are still consistently catching bugs in one configuration and not the other, and whether there are some combinations of these that we can tone down (in particular for our "always-on" PR-validation jobs). For example, it might be reasonable that libraries-only changes does not submit jobs in both configurations.

cc @dotnet/runtime-infrastructure @dotnet/area-infrastructure-libraries @jkotas @danmoseley @MattGal

[dotnet-issue-labeler]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[ghost]

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

Issue Details

---

We currently submit all innerloop and outerloop jobs twice, in the normal and the no_tiered_compilation environments. It would be useful to monitor where we are still consistently catching bugs in one configuration and not the other and whether there are some combinations of these that we can tone down (in particular for our "always-on" PR-validation jobs). For example, it might be reasonable that libraries-only changes does not submit jobs in both configurations.

cc @dotnet/runtime-infrastructure @dotnet/area-infrastructure-libraries @jkotas @danmoseley @MattGal

Author:	jakobbotsch
Assignees:	-
Labels:	area-Infrastructure, untriaged
Milestone:	-

[jkotas]

As far as I can tell, the normal / no_tiered_compilation split kicks in only for the runtime tests in regular PR-validation jobs. The libraries tests are not running with this split in regular PR-validation jobs. Is this correct?

If this is correct, there is not much to save by limiting this for libraries-only change since the runtime tests are suppressed for libraries-only changes already.

I agree that it is worth looking into whether there is enough value in having the full normal / no_tiered_compilation split in regular PR-validation jobs. I think it may be fine to only run no_tiered_compilation flavor by default where I expect the majority of the potential regressions are going to be.

[jakobbotsch]

As far as I can tell, the normal / no_tiered_compilation split kicks in only for the runtime tests in regular PR-validation jobs. The libraries tests are not running with this split in regular PR-validation jobs. Is this correct?

Based on the logs I think you're right, it looks like we only submit the two scenarios for runtime tests and not for libraries tests.

I think it may be fine to only run no_tiered_compilation flavor by default where I expect the majority of the potential regressions are going to be.

Makes sense to me, I'll try to do it this week if there are no other objections.

[jakobbotsch]

Fixed by #69178