Plan for CoreCLR and Mono to run corefx tests in their CI

[safern]

I've opened this issue to have a discussion on what would be the best plan to follow regarding on CoreFX delivering test assets to Mono and CoreCLR.

I know that @mikem8361 has been doing some work to publish tests assets to an azure blob storage with a well known name, and then CoreCLR would restore that. However there has been some discussion on why using blob storage and not NuGet packages.

I know that @ericstj had some points against using NuGet packages, regarding the package size and the layout of the package as it will not be following any NuGet conventions.

cc: @marek-safar @danmosemsft @jkotas @RussKeldorph

[mikem8361]

My plan so far. I already have the corefx uploading steps prototyped.

• Change corefx the builds to add a step to upload the test asset zips (around 270) to blob storage. It only takes 30-50 seconds to upload these zip file per "config".
  • Using Arcade’s Microsoft.DotNet.Build.Tasks.Feed/PushToBlobFeed build tasks.
  • Give the blob storage a strongly named like: corefx-tests/$(OfficialBuildId)/$(OSGroup).$(ArchGroup).$(ConfigurationGroup)/$(TargetGroup).
  • Using the dotnet-core blob storage account that already has a build def variable group with the PAT.
• Upload the corefx binaries to this storage URL too as a zip file (basically most of the current corefx Helix test job correlation payload). We also could just use the nuget packages that are published for the build, but that would really be painful to use on the coreclr test side.
• Optionally change the corefx Helix test step to use this blob storage account by sending the URL of the test asset zips instead of being part of the Helix payload. This would mean the corefx test runner/script would have to change.

On the coreclr side:

• Use the dependency flow/BAR to get a corefx build id in coreclr repo. I’m going to start with a fixed LKG corefx build and hook up the dependency flow stuff later.
• In the coreclr test pipeline, build the URL of the above test asset zips.
• Build the “correlation” payload that contains the coreclr bits to test and the corefx bits from that build id. There are some details here that need to be worked out.
• Submit the Helix job with this URL of the test zips, the correlation payload and a test runner that can download and run the tests. There is already a test runner (Anton wrote) that was written for this that also allowed filtering of tests for “breaking” changes.

[marek-safar]

@mikem8361 could the payload url builder and test runner be available in some form of shared/reusable component outside of coreclr?

[mikem8361]

Yes, I try to make it shareable/reusable.

[jkotas]

I would like the scheme to work well for test asserts of other repos, not just for corefx.

@KevinRansom Do you think the scheme described above (https://github.com/dotnet/corefx/issues/34957#issuecomment-459069106) would work for packing FSharp xunit tests?

[jkotas]

It would be a good idea to talk about requirements of the solution. Here is my take:

Requirements

• The versioning scheme and retention policy is same as what is used to exchange product bits and other binary assets between repos. (e.g. every Microsoft.Private.CoreFx.NETCoreApp version has a matching test binary drop)

• It is easy for any repo with ordinary xunit tests to participate. It should be less than a day of work to onboard a new repo into the system (e.g. to add a option to run the repo tests in CoreCLR CI). We can use FSharp to validate this.

• Ability to run the tests in different modes (e.g. JitStress modes, crossgen, asmdiffs)

• Ability for VMs contributors to run and debug the individual test locally

Thoughts?

[mikem8361]

I work towards these goals, but I'm only part time on this engineering effort.

If your requirements are saying that we package the tests in nuget packages instead of zips, I have to disagree. It is so much easier to create the helix payloads when the tests are zip files. The test asset will be pushed in the same azo blob storage as the product nupkgs and have the same versioning. The corefx tests and I'm sure any other repo's tests already package the assets up as zips in the build step. Adding another step to package as nuget doesn't make sense and adds more time to the build. And on the coreclr (and other repos) side that want to run these tests they are going to use Helix so it just make sense to packaged them as Helix playloads. The nuget package layout and restore doesn't make it easy to use them to run tests in helix.

If we use nupkgs for the test assets this is what would have to happen (roughly):

• Build and zip the test assets. Any repo that runs these tests in Helix will have to do this anyway.
• Create a nupkg package of all the zipped test assets. It wouldn't make sense to create nupkg(s) of the test assets directly since Helix needs zip files as the payloads.
• Publish the nupkg to the dotnet-core blob feed.
• Submit the test zips to Helix.

On the coreclr side:

• Install/restore the test asset package(s) somewhere. A download that isn't really needed and takes time.
• Copy the nupkg package layout to something that can be zipped unless the test assets is one nupkg of zip files.
• Create the helix payload which will upload them unnecessarily.

Here is what I plan to do and it should lend itself to any repo and any test assets:

• Build and zip the test assets. Any repo that runs these tests in Helix will have to do this anyway.
• Publish the zip files to dotnet-core azo container storage. Part of this is a manifest json file that is generated by the blob feed build task.
• Submit the test zips to Helix.

On the coreclr side (or any other repo running the tests):

• Download this manifest file and use it to build the helix payload. I plan to add a simple build task to Microsoft.DotNet.Build.Tasks.Feed to parse the manifest file entries and return a ItemGroup that can be used to create the Helix SDK's HelixWorkItems. The payloads will be uri's of the test asset zip files. The Helix SDK will need a minor modification to accept a URI PayloadArchive. It already accepts URI correllation payloads.
• Create the correlation payload combining the corefx bits and the coreclr bits to be tested. There are some details here I need to work out, but this would probably be similar in both scenarios.
• Submit the helix job without actually downloading any of the test assets directly.

This plan involves adding a small build task to Arcade to parse the manifest and a small tweak to the Helix SDK to accept a URI as a payload.

[jkotas]

CoreCLR (and I expect Mono too) needs ability to run the tests with different modes. The different modes include setting environment variables, running crossgen on the tests before the run, or running tools like asmdiffs on the output of the tests.

How are such customizations going to be injected into the prebuilt Helix payloads?

[jkotas]

Any repo that runs these tests in Helix will have to do this anyway

What does the adoption of Helix looks like across .NET Core repos? E.g. Is FSharp running its tests in Helix or plans to do so?

[mikem8361]

I'll have to think about the details, but the zip files are just the tests themselves and a script to run them. Setting up the environment, etc. is still part of the step in the repo running the tests. If the payload itself needs to be customized, I'll have to think about that. I'm currently trying to get back to what coreclr was doing before the corefx/coreclr shift to Arcade/Azo, etc. which is to just run the tests the way they were run in corefx replacing the coreclr bits.

If more has to be done to the payload, like running crossgen on the tests, then my approach would require unzip each test zip and rezipping so it may require me to rethink it.

I assume that Helix is the way to run tests, but I'm sure there is still a lot of work left to get there. I don't know about any specific repo other than coreclr.

[safern]

We could directly change the scripts in corefx to accept optional parameters to enable crossgen or other tools. That would also work.

Once that is changed, then you wouldn't need to unzip the test zips and you can just add the necessary tools as correlation payload, and pass it in to the script through the optional parameters.

[jkotas]

We could directly change the scripts in corefx to accept optional parameters to enable crossgen or other tools. That would also work.

I think it is important for the script that drive the codegen test modes to live in the CoreCLR repo and not be part of the corefx, FSharp, etc. test drops. Otherwise, it would be very hard to evolve them.

I'm currently trying to get back to what coreclr was doing

coreclr was doing repackaging before the shift. It did not try to reuse the helix payloads. Do we really believe that skipping the repackaging is that important optimization?

[marek-safar]

We are fine with zips (it's actually prefered setup for us over nugets).

CoreCLR (and I expect Mono too) needs ability to run the tests with different modes.

I am not sure we want to go there. The CoreFX tests will be in part dynamic and I think they already are. Running the tests with e.g. AOT engine should not be too different than checking for any other feature (e.g. SSL version). Where to put the runtime config options to run them is a good question but I suspect they will be a test configuration specific and therefore part of the test blob to runtime mapping logic.

[mikem8361]

Actually coreclr did reuse the helix payload before the new helix SDK changes that made the blob storage internal. We are currently testing against some very old corefx/corefx tests bits.

The test scripts Santi are referring too are the ones that run the individual test but the main script/build logic to set it all up will still live in the coreclr (or other) repo. There could be a global script included as part of the correlation payload the script that the individual test script calls to do this kind of setup or initialization. I don't think coreclr should have knowledge or scripts on how to run an individual test or even the list of tests "hard coded".

[jkotas]

Running the tests with e.g. AOT engine should not be too different than checking for any other feature (e.g. SSL version).

I meant that the test itself is AOT compiled before running. That is different from checking SSL version, correct?

[jkotas]

coreclr should have knowledge or scripts on how to run an individual test

What would be the process for investigating a test failure if there is no script to run individual test?