Enable Windows arm32 test builds and publishing

[BruceForstall]

I want to enable coreclr repo CI to test corefx on Windows/arm32. This requires that the corefx official build generate and publish a build of the test assets, and corefx-test-assets.xml file manifest. This should be as easy as setting "_publishTests: true" for the Windows/arm32 case in eng\pipelines\windows.yml. However, it looks like Windows/arm32 isn't tested in the official build (why? It is a supported platform) so I presume the tests aren't built, either.

What needs to happen to make this work?

[BruceForstall]

@ViktorHofer @ericstj PTAL

[ViktorHofer]

We currently don't run tests on Windows arm but just build it (https://github.com/dotnet/corefx/blob/master/eng/pipelines/windows.yml#L170-L173) because IIRC the machine pool is very small and enabling it in corefx would just kill it.

Maybe we got more machines meanwhile, @MattGal should know.

[ericstj]

I would imagine we have independent decisions about build, publish, and actually run, so we should be able to satisfy @BruceForstall's request regardless, right? That said, I agree we should make sure we run the tests ourselves, otherwise we're just making CoreCLR deal with our failures.

[ViktorHofer]

We can easily enable publishing the test archives without invoking them but we usually try to avoid that...

[MattGal]

@ViktorHofer assuming "windows arm" means 32-bit, I think we have machines; looking at windows.10.arm32.open and windows.10.arm32 queues there's actually quite a few machines with the caveat they're still running Python 2 and depending on your workflow may need to be updated. Based off their heartbeats, virtually noone's sending them work, so worth a try.

[danmoseley]

@ViktorHofer are you planning to do this ? Or should this be 'future'?

[ViktorHofer]

Unfortunately I wasn't able to finish this in time before my holiday. I will leave it to @ericstj.

[BruceForstall]

@ViktorHofer @ericstj Any progress here?

[ViktorHofer]

We prioritized fixing the Windows Nano Server leg over this one. This time this is really the next on my list and I will discuss with @Anipik in today's infra standup if he can help me bring the failure count down to zero.

[MattGal]

Just to chime in from the Helix perspective, there's now at least 31 healthy machines in the windows.10.arm32.open queue which seem to be barely used (most recent work item was 8/8) so if you have a functioning test build it might be worth exploring if this can handle at least your rolling outerloop runs.

[BruceForstall]

@ViktorHofer Presumably in the merged-repo world this is now a non-issue? I'll need to re-implement libraries testing for coreclr PRs anyway.

[ViktorHofer]

I'm unsure how many arm machines are available at the moment. We still need to add a leg for arm32 as we officially support it.

[MattGal]

There's 33 hummingboards / pis serving windows.10.arm32.open right now, and ~20 on windows.10.arm32 . This is likely enough (they're literally all idle as I type this) for your purposes, but if you need more @ilyas1974 can probably comment.

[safern]

@ViktorHofer @BruceForstall -- we started running libraries tests on coreclr in ARM32 a couple of weeks ago on rolling CI builds: #35690

Should we close this issue now?

[BruceForstall]

Makes sense to me.

[ViktorHofer]

Before closing let's please validate that we actually test on arm32. It sounds like we are running tests against an arm32 version of the runtime but on arm64 hardware which makes me believe that we don't exercise anything arm32 specific. As an example, the tests were failing on arm32 hardware and we had to disable tests via ActiveIssue/SkipOnPlatform but never (fully) did that for all failing tests.

So what I would want to make sure of, before closing this issue is, that issues are actually fixed now and aren't just hidden because we don't run on arm32 hardware.

[safern]

We run on arm64 using WOW to run on arm32... I did have to disable some tests on arm32 when putting that PR... (FWIW coreclr runtime tests for arm32 use the same arm64 helix queue) cc: @jashook

[safern]

So what I would want to make sure of, before closing this issue is, that issues are actually fixed now and aren't just hidden because we don't run on arm32 hardware.

I did a test run to the arm32 helix queue of libraries tests and indeed there are like 16 test assemblies with failures. We should disable those, open issues and switch to the right queue for libraries tests.

I believe runtime tests can't use that queue because of the payload size as it is very slow to download the workitems.

@MattGal and I looked at my private run and the workitems spent a few seconds downloading the payload, so it doesn't seem to be the case for libraries.

[BruceForstall]

@safern Where did you do the arm32 helix queue test? My change #36486 has been running Windows arm32 testing, and there are still some un-triaged failures, but it's been using the /eng/pipelines/coreclr/templates/helix-queues-setup.yml file to set up the queues, so maybe that sets up a different helix queue than the libraries version?

[safern]

/eng/pipelines/coreclr/templates/helix-queues-setup.yml file to set up the queues, so maybe that sets up a different helix queue than the libraries version?

Right now they setup the same queues. I did it locally in my computer by building for arm32 and sending a private test run to helix. Replicating what we do in CI. I used, Windows.10.Arm32.Open helix queue.

You could achieve this on your PR though by changing the queue temporarily, but there are more failures because the OS is Windows IOT on those devices.

[safern]

Based on the conversation we had with @richlander this week, this shouldn't be required anymore. Closing.