Android and Apple platform support for .NET 7 / MAUI

[premun]

Cloned here from https://github.com/dotnet/core-eng/issues/15176

Context

This epic is a continuation of epic Android and Apple platform support for .NET 6 (#10420) which focused on bringing the support of these mobile platforms in Helix to a state where we onboard Mono teams' tests and provide a stable test environment.

Currently, we are running tests on all types of devices and emulators we have in Helix (iOS/tvOS/Android). However, we don't yet support the full required matrix (dotnet/runtime#33381). As an example of missing things can be older Android versions, performance testing or capabilities to allow various other complex scenarios such as mono runtime tests in the AOT mode. These require more feature work and infrastructure setup still.

Stakeholders

This project is driven by requirements of other teams that have P0 .NET 6 deliverables. These teams and their related scenarios are:

• .NET Apple & Android / Steve Pfister - running BCL tests on Android/Apple/WASM
• .NET MonoVM / Sam Patel - testing the mono runtime (also CLR runtime tests running on mono)
• .NET WebAssembly / Larry Ewing - testing mono on WASM
• Xamarin Forms / .NET MAUI / Jonathan Dick - setting up the whole test infrastructure
• Xamarin Mac/iOS / Alex Soto Garcia - previous owners of XHarness, shared code with them
• Performance / Heather Arbon - running perf tests on Android/iOS

Primary objectives

The .NET 7 and MAUI efforts have a stable platform support in Helix

The Helix environment is stable, feature-complete, infrastructure is ready and we have tooling available that allows teams to run their workloads. These workloads can have different requirements when it comes to how we run them or how much capacity they need. We work with the teams to help them determine what volume of tests we can support with existing hardware.

Progress tracking:

	Android emulators	Android devices	Apple simulators	Apple devices
BCL tests (JIT)	✅	✅	✅	➖
BCL tests (AOT)	➖	➖	➖	✅
Runtime tests (JIT)	✅	arm64 ✅ / arm32 🔜	✅	➖
Runtime tests (AOT)	➖	➖	➖	🚫
MAUI perf tests	➖	✅	➖	✅

Legend:
➖ - not planned
✅ - onboarded and tests running regularly
🚫 - feature complete, owning team decided to not implement this (yet) as coverage comes from sibling area
🔜 - feature complete, owning team hasn't started running tests regularly yet

Definition of done

Client teams dependent on this work confirm they are feature complete and Helix offers a stable environment that supports their workloads on target platforms reliably.

Tasks:

• Add --no-wait switch to apple just-run and apple run commands for fire&forget behaviour xharness#846
• Use install/just-test/uninstall for default Helix SDK mobile operation #8454
• Handle --launch-timeout expiration with an appropriate exit code xharness#855
• App logs are empty on Helix Apple simulators xharness#694
• Android emulators fail to boot before accepting Helix jobs xharness#754
• Exit code detection failing on iOS/tvOS 15+ xharness#819
• Exit code detection failing on OSX 12.00 maccatalyst xharness#848
• Helix SDK fails to pull files from netcorenativeassets because of SSL errors #8703
• Add apple mlaunch command (same as android adb) xharness#843
• Starting the ADB server fails xharness#822
• https://github.com/dotnet/core-eng/issues/15097
• https://github.com/dotnet/core-eng/issues/15097
• CLI for running Android tests should allow specifying API level xharness#380
• Verify network connectivity for Android mobile devices #8232
• Add a new exit code for failure of app's exit code detection xharness#812
• Test apk targeting API 30 xharness#678
• Failed Simulator launches+uninstalls are categorized as TIMED_OUT xharness#807
• The Apple timeouts won't short-circuit device finding xharness#721
• https://github.com/dotnet/core-eng/issues/15397
• Old Android version emulators fail to boot reliably #9149
• XHarness returns TESTS_FAILED when emulator dies mid-instrumentation xharness#872
• Copying of crash reports should wait for the report to be written xharness#869
• Can't install Simulators with Xcode 13.3 xharness#840
• Simulator fails to launch on Xcode 13.3 xharness#839
• Can't detect/use/find iOS 15.4 xharness#838
• Emulator crash halfway through instrumentation should be categorized differently xharness#885
• mlaunch error missed when analyzing crash xharness#887

Finalize the Android/Apple Helix environment and hand it over to dnceng

Make the Android/Apple area maintainable by our team just like any other parts of Helix so that infrastructure issues are detected automatically, First Responders are able to attend to these and a good SLA is kept.

Definition of done

Telemetry and alerting is in place (#13607). Tooling and documentation is available and our team is educated in servicing of this area. First Responder duty takes over incident management for mobile platforms.

Tasks:

• https://github.com/dotnet/core-eng/issues/14888
• Set up 2FA process for DNCENG's Apple Developer account #9620
• Improve alerting for broken mobile devices #9092
• Create mobile platforms maintenance docs #9102

Recently Triaged Issues

All issues in this section should be triaged by the v-team into one of their business objectives or features.

[tkapin]

@premun - is anything blocking us from closing this epic?

[veikkoeeva]

@tkapin Would this mean one can use .NET with Maui? :) (Ah, yes, I'm probably too enthusiastic if this part starts to look good.)

[tkapin]

@tkapin Would this mean one can use .NET with Maui? :) (Ah, yes, I'm probably too enthusiastic if this part starts to look good.)

This issue tracks development of the infrastructure that was built to test .NET on the mobile devices in order to support .NET MAUI scenarios. If you have a concrete .NET MAUI feedback or issue, could you please share it in the https://github.com/dotnet/maui repo? Thanks!

[premun]

@tkapin we can close this. The left-over task can probably be closed soon as dotnet/runtime has moved onto OSX 12.00 and it seems this has resolved some of the Simulator stability issues that we've seen there. I will validate that this week and close it.

[premun]

This has been finished. We waited with closing until the MAUI release and then some but

• the team has been educated how to maintain the area
• all issues in this area will be triaged based on the maitenance plan.

We've already sort of sent a couple of woohoo's for this effort so not sending this last one.