Automate SPMI collection for TE benchmarks

[EgorBo]

This PR adds crank-driven SPMI collections (TechEmpower, OrchardCMS, etc). For now I've added 4 benchmarks, will enable more. the ones requiring databases might be tricky to add.

TODOs (in separate PRs):

• 32-bit targets (windows-x86, linux-arm)
• Enable Linux-x64 (azurelinux.3.amd64.open) - currently, it runs out of disk space when running this script (it requires ~3GB of disk space).
• Remove git dependency and hacks around it once Add Archive source option crank#841 is landed.
• Enable more benchmarks
• Add runs with special JIT modes like TC=0, etc.

The currently produced MCH contains 32k contexts (0.5Gb)

In case if you want to have a dryrun locally:

py .\src\coreclr\scripts\superpmi_aspnet2.py --dryrun

Actual run:

py .\src\coreclr\scripts\superpmi_aspnet2.py --core_root .\artifacts\bin\coreclr\windows.x64.Checked `
--output_mch aspnet2.mch

[AndyAyersMS]

I found on recent builds I also needed to copy over some of the host dlls or I get odd CFI failures on windows. I have unmerged changes to my script with these changes. Maybe I should put it up for PR?

[EgorBo]

/azp run runtime-coreclr superpmi-collect-test

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[Copilot]

Pull Request Overview

This PR automates SPMI (Super PMI) collection for ASP.NET benchmarks, specifically the TechEmpower benchmarks, by integrating with the "crank" tool. The changes enable automated collection of JIT compilation data from real-world ASP.NET workloads to improve JIT performance analysis.

• Adds a new "aspnet2" collection type that runs TechEmpower benchmarks via crank
• Creates a self-contained Python script to manage the entire crank-based collection workflow
• Integrates the new collection type into the existing SPMI pipeline infrastructure

Reviewed Changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
src/coreclr/scripts/superpmi_collect_setup.py	Updates setup script to recognize and handle the new "aspnet2" collection type
src/coreclr/scripts/superpmi_aspnet2.py	New comprehensive script that orchestrates TechEmpower benchmark execution via crank for SPMI collection
src/coreclr/scripts/superpmi-collect.proj	Adds MSBuild configuration for aspnet2 collections including Helix work item definitions
eng/pipelines/coreclr/templates/superpmi-collect-pipeline.yml	Enables aspnet2 collection in the CI pipeline for multiple platforms
eng/pipelines/coreclr/templates/run-superpmi-collect-job.yml	Configures input directory for aspnet2 collection jobs

[EgorBo]

PTAL @dotnet/jit-contrib @AndyAyersMS

[AndyAyersMS]

Overall this looks good.

You might want to do the same thing my old script does (and we now do for benchmarks) and use a release build with checked jit.

If a particular scenario fails does the whole thing just fail? I ended up trying to carry on anyways, with the thought that a partial collection was better than nothing.

[EgorBo]

Overall this looks good.

You might want to do the same thing my old script does (and we now do for benchmarks) and use a release build with checked jit.

If a particular scenario fails does the whole thing just fail? I ended up trying to carry on anyways, with the thought that a partial collection was better than nothing.

@AndyAyersMS

You might want to do the same thing my old script does (and we now do for benchmarks) and use a release build with checked jit.

My code re-uses the same logic, see here:

runtime/src/coreclr/scripts/superpmi_collect_setup.py

Lines 519 to 520 in d353f5f

	if coreclr_args.collection_name == "benchmarks" or coreclr_args.collection_name == "realworld" or coreclr_args.collection_name == "aspnet2":
	# create a directory with release runtime bits and a checked jit

I only added the aspnet2 check.

If a particular scenario fails does the whole thing just fail? I ended up trying to carry on anyways, with the thought that a partial collection was better than nothing.

Yes, I copied that logic from your script (100%) 🙂

[AndyAyersMS]

You might want to do the same thing my old script does (and we now do for benchmarks) and use a release build with checked jit.

My code re-uses the same logic, see here:

Ah, good ... in my old script we had to do this with the script itself, so I was looking for that.

[EgorBo]

/ba-g "linux-x64 Debug NativeAOT timeout"