superpmi-diffs pipeline improvements

[BruceForstall]

This issue tracks a number of improvement ideas for the runtime-coreclr superpmi-diffs AzDO pipeline.

• JIT asserts (e.g., this example) and other SuperPMI errors (such as Error generating win-x86 SuperPMI asm diffs for libraries.crossgen2 collection #61482) need to be surfaced as errors in the asmdiffs jobs (this is probably already true), and shown prominently in the "Extensions" page that summarizes the diffs. They should probably exist as a prominent section in each summary.md file as well as the overall summary.md files.
• Currently, SuperPMI MISSING data causes superpmi.py to return a failure error code (of 1), which causes the AzDO asmdiffs job to be marked as "warning" (circle with exclamation point). We should change it to ignore these, at least for the AzDO job.

category:eng-sys
theme:super-pmi
skill-level:intermediate
cost:small
impact:small

[ghost]

Tagging subscribers to this area: @JulieLeeMSFT
See info in area-owners.md if you want to be subscribed.

Issue Details

---

This issue tracks a number of improvement ideas for the new runtime-coreclr superpmi-asmdiffs AzDO pipeline.

• JIT asserts (e.g., this example) and other SuperPMI errors (such as Error generating win-x86 SuperPMI asm diffs for libraries.crossgen2 collection #61482) need to be surfaced as errors in the asmdiffs jobs (this is probably already true), and shown prominently in the "Extensions" page that summarizes the diffs. They should probably exist as a prominent section in each summary.md file as well as the overall summary.md files.
• Currently, SuperPMI MISSING data causes superpmi.py to return a failure error code (of 1), which causes the AzDO asmdiffs job to be marked as "warning" (circle with exclamation point). We should change it to ignore these, at least for the AzDO job.

Author:	BruceForstall
Assignees:	-
Labels:	area-CodeGen-coreclr
Milestone:	7.0.0

[kunalspathak]

We should consider uploading back subset of .dasm files as artifacts. It will be useful to have them if someone who sees the diffs and (most likely) want to investigate. Having them handy will avoid trouble of building and producing those dasm files. They can be zipped and uploaded. On my machine, I zipped a folder contined 42K .dasm files comprising of 612 MB, but after zipping they were just 30MB.

One option would be to pass a flag to jit-analyze which after producing the diff.md will delete all but top 20 .dasm files. Then, we can just zip that folder and upload as artifact.

[kunalspathak]

We should consider uploading back subset of .dasm files as artifacts.

This is done in #61700.

[EgorBo]

https://dev.azure.com/dnceng/public/_build/results?buildId=1582186&view=results
SPMI failed with NotFound in my jit pr #64530

[kunalspathak]

https://dev.azure.com/dnceng/public/_build/results?buildId=1582186&view=results SPMI failed with NotFound in my jit pr #64530

It seems that win-x64 timed out. https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-64530-merge-dd476490704a459997/win-x64/1/console.af0c1c0e.log?sv=2019-07-07&se=2022-02-20T16%3A55%3A26Z&sr=c&sp=rl&sig=0lS82iJzZJaElO7kAHrUZaZsP9N9XszewTrYMIKMAF4%3D

[EgorBo]

https://dev.azure.com/dnceng/public/_build/results?buildId=1582186&view=results SPMI failed with NotFound in my jit pr #64530

It seems that win-x64 timed out. https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-64530-merge-dd476490704a459997/win-x64/1/console.af0c1c0e.log?sv=2019-07-07&se=2022-02-20T16%3A55%3A26Z&sr=c&sp=rl&sig=0lS82iJzZJaElO7kAHrUZaZsP9N9XszewTrYMIKMAF4%3D

ah, makes sense now, the diffs there were quite big

[kunalspathak]

the diffs there were quite big

Is that expected? I just want to make sure that we are picking up the correct baseline jit and do asmdiff only on your changes.

[EgorBo]

the diffs there were quite big

Is that expected? I just want to make sure that we are picking up the correct baseline jit and do asmdiff only on your changes.

Hm.. not sure, didn't expect it, running locally now

[BruceForstall]

Related: #75848, #80718, #80719