Investigate feasibility of running tests on Android and iOS for MAUI compatibility

[paulirwin]

Is there an existing issue for this?

• I have searched the existing issues

Task description

The MAUI team themselves use a tool called xharness to run tests on Android and iOS. We should investigate if it's feasible to do so.

Example MAUI ADO pipeline result: https://dev.azure.com/xamarin/public/_build/results?buildId=126800&view=logs&jobId=5577a1a6-5922-5c8f-f8aa-3ff6c6e2bbde&j=5577a1a6-5922-5c8f-f8aa-3ff6c6e2bbde&t=aa670144-16c6-5487-06c9-b7432a5fb589

(Split from #258)

[NightOwl888]

The xharness project sounds promising.

It allows implementing custom test runners, which is an issue we had on J2N when adding tests for Xamarin.Android. The official XUnit test runner would not scale to run 70,000+ tests, and the official NUnit runner used NUnitLite (which is going to be an issue for the Lucene.Net test framework which depends on NUnit), so we had to make our own runners. Debugging was also very manual. You basically had to add a filter with the name of the test you want to run and then set a breakpoint in that test.

See https://github.com/NightOwl888/J2N/pull/96/files to see the configuration of Xamarin tests before they were removed - it took about 4 months of trial and error to get the tests to run on Azure DevOps being that documentation on testing class libraries was scarce. We could still use this approach if xharness doesn't pan out, the only thing we would need to do is replace the projects that build the .apk to side load into the Android emulator and update the test runners to run on MAUI.

Note that there were several limitations with Xamarin.Android that made testing worth it, though:

1. Mono has incomplete generics. We had compilation problems with both LurchTable<TKey, TValue> and with EqualityComparer<T> that required breaking API changes in J2N to fix.
2. There were limitations with memory mapped files. See: Creating a MemoryMappedFile with fixed capacity throws ArgumentException, fatal crash during attempt to write dotnet/android#5423. These didn't affect Lucene.Net, but did cause some of the Harmony tests to fail in J2N.
3. There were several problems with the way globalization was implemented in Xamarin.Android. See: Basic Turkish dotted "I" tests with ignore case fail in Turkish culture dotnet/android#5425.
4. Some other APIs in Xamarin.Android threw PlatformNotSupportedException that made it necessary to find alternatives and/or conditionally suppress tests.

As these issues were left open until Xamarin went out of support, I don't have high confidence that they have been addressed in MAUI, and may need to be re-reported. So, I agree that this is required before we can release 4.8.0 because there may be more breaking changes required to fully support MAUI.

I did a little experimentation and got the tests to run on Lucene.Net.Analysis.Common on Xamarin.Android at one point, but it has never been done for all of the Lucene.NET tests.

[paulirwin]

So, I agree that this is required before we can release 4.8.0 because there may be more breaking changes required to fully support MAUI.

I, for one, do not think this is required before we can release 4.8.0 unless the community demands it. I put it on the 4.8.0 milestone for investigation and to not lose track of this issue, but I would certainly vote to cut it if needed. If we hear radio silence about needing to ensure full tests passing on iOS and Android, and no one wants to step up to do the work, I personally do not feel like we need to go through this effort, and certainly not if it means several months of delay in the release. It's very possible that it will work as-is sufficiently for most/all iOS and Android use cases, and we welcome issues being filed if there are incompatibilities. I am not aware of any other open-source third-party (meaning, not Microsoft) projects that are going to this extent for these platforms, but if anyone has references that could make our job easier please share. But I don't think we need to spend a ton of effort and energy from our limited time available if no one speaks up about this being important, or is willing to help contribute.

I'm not saying that's the case yet (I just filed the issue yesterday), but I'd say this issue is a call to action for the community. We've only had 1 reported issue on MAUI and it was already fixed in beta 17. If you include Xamarin, we've had only 2 a few issues reported in over 3 years. (edit: my apologies, the number is more like ~3 for Xamarin) That implies that either it is working well apart from the issues we've fixed, or not many people are using it for that purpose. If anyone needs 4.8.0 to have full test coverage (or any specific test coverage) on iOS and Android, please speak up! Better yet, volunteer to take on this issue. @NightOwl888 and I have enough on our plates. Meanwhile, please install beta00017 in your MAUI apps and let us know if there are any issues, and also please let us know if it works well and what platform(s) you're using it on so we can gauge interest. Let us know what kinds of analysis, query parsing, directories, etc. you're using so we can better understand real-world workloads on mobile devices.