[RFC] Improve test time

[rolfbjarne]

• Api diff
  There are two parts here

  • We compare against the reference XML for the latest stable release (range: 1:04-1:40 in timings from above).
  • We compare against the previous commit (for PRs the commit the PR is based on). This is by far the most time-consuming, since it checks out the previous commit and builds src/ (range: 4:43-9:05 in the timings from above).
  • Improvements
    • Profile mono-api-info.exe and mono-api-html.exe to see if there’s anything simple that can be done to speed them up.
    • If assemblies are built with /deterministic, we could store the md5 somewhere and skip the diff completely for identical assemblies.
    • Upload the “reference” xml files for every commit, so that the next commit can download those instead of rebuilding src/ (the presence of the uploaded files can’t be relied upon, because internal jenkins doesn’t build every commit).

• Instead of creating (and building and running) a different test app for every BCL test, we could try putting multiple test assemblies in each test app. There might be some test assemblies that are incompatible with each other, so we might still end up with more than one test app, but a lot of the time spent testing is spent building/installing/launching apps in the simulator(s).

• Make sure we don’t build/package more XM tests than we need. PR [tests] Only build exactly what's neded for running XM tests on older bots. #5038.

• Improve our automatic test selection.

  • Only do API comparison to previous commit if anything in src/ changed
  • Only do API comparison to stable if there was a mono bump or anything in src/ changed

• Go through slower tests and see if they can be optimized/modified to be faster:

  • mmp regression tests: 8 minutes -> to be reviewed/converted in Port mmp regression tests to "normal" unit tests for XM #4975
  • mmptest: 39 minutes
  • MSBuild/iOS: 22 minutes
  • MSBuild/macOS: 4 minutes
  • mtouch tests: 1h17 minutes

• Parallelize more. I estimate that around 20-25 minutes could be saved by doing something like this (dot file):

  

  A table of some timings are available here.

[spouliot]

API & Generator comparison is likely not needed on the private Jenkins since it was done on PR (even if commit order might differ) and reviewed.

Maybe the Parallelize more should be in the build RFC since it re-orders everything. It might replace (or complement) a pie chart if we can assign time to each item and add a "end" step to identify the critical path (which is the one we should optimize first).

[chamons]

mmptest: 39 minutes This is likely because mmptest is a total hack, and we generate and build a huge number of projects, and build them multiple times to get any sort of real coverage.

Ideally we'd port many of them to a "real" msbuild test suite that we shared between ios and mac.

We could profile each test time and see if there are some exceptionally ugly tests to more conditionally run.

[spouliot]

@chamons there's probably places where several tests can be merged, i.e. build once to check many things, but I agree that as much as possible we should be testing this differently (but are blocked right now).

[spouliot]

MMP (regression) tests to be reviewed / converted -> #4975