Allow for "archiving" of SPMI-generated diffs

[SingleAccretion]

Today, we use the very informative SPMI diffs pipeline to make decisions and observe diffs before merging changes, but the summaries quickly expire, and so they are not as useful as they could be for looking at past changes. It would be great to have some (preferably automatic) way to save them (perhaps even with the .dasms).

@dotnet/jit-contrib

category:eng-sys
theme:super-pmi
skill-level:intermediate
cost:medium
impact:medium

[ghost]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch, @kunalspathak
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Today, we use the very informative SPMI diffs pipeline to make decisions and observe diffs before merging changes, but the summaries quickly expire, and so they are not as useful as they could be for looking at past changes. It would be great to have some (preferably automatic) way to save them (perhaps even with the .dasms) for merged changes.

@dotnet/jit-contrib

Author:	SingleAccretion
Assignees:	-
Labels:	area-CodeGen-coreclr
Milestone:	-

[BruceForstall]

Thank you for opening this issue. We've talked about this internally many times but I couldn't find another GitHub issue describing it.

It would be nice if the "Extensions" page ("Summary" markdown file) was automatically posted to the GitHub comments of a triggering PR. There are some tricky security challenges to get that to happen (namely, giving the superpmi-diffs job GitHub posting permissions). The ASP.NET perf system apparently does do this kind of post-back so it should be solveable.

Similarly, you could imagine posting the .dasm file summaries to the JIT Azure Storage location (where the SPMI MCH files and JIT rolling builds are stored), with some appropriate size/time expiration policy (that is more generous than the Azure DevOps policy). Once again, the issue is security.

[JulieLeeMSFT]

I was hoping to see this happen as well. Meanwhile, I recommend the contributors to copy/paste the key diff info to the GitHub comments manually for recording/documentation purpose. CC @dotnet/jit-contrib.

It would be nice if the "Extensions" page ("Summary" markdown file) was automatically posted to the GitHub comments

[kunalspathak]

I will talk to @sebastienros to see how we can integrate this ability.