Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add test leg to run runtime tests on Android arm64 #44560

Merged
merged 28 commits into from
Dec 11, 2020

Conversation

fanyang-mono
Copy link
Member

@fanyang-mono fanyang-mono commented Nov 11, 2020

This work contributes to the project of bringing up test coverage for Mono (#33381)

488 runtime tests ran and passed on this newly added CI lane. 1856 runtime tests were disabled for this lane, due to long running work items and oversized Android apps. Created two issues to track those two issues separately. (#45965 and #45568)

@ghost
Copy link

ghost commented Nov 11, 2020

Tagging subscribers to this area: @directhex
See info in area-owners.md if you want to be subscribed.


Issue meta data
Issue content:
Issue author: fanyang-mono
Assignees: -
Milestone: -

@fanyang-mono fanyang-mono marked this pull request as ready for review December 7, 2020 13:55
@SamMonoRT
Copy link
Member

@safern - this is a new lane too, what needs to be done to add it to the staging pipeline first ?

@naricc
Copy link
Contributor

naricc commented Dec 7, 2020

@SamMonoRT So, my understanding is basically that to add to staging, you add the lanes to runtime-staging.yml instead of runtime.yml. You can use @safern's original PR moving the android emulator legs as an example: https://github.com/dotnet/runtime/pull/45178/files

Copy link
Member

@trylek trylek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, thank you! Please make sure to revert the instrumentation changes to runtime.yml before merging in :-).

Copy link
Member

@safern safern left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's add it to runtime-staging pipeline first to make sure it is stable before we move it to the main pipeline.

Copy link
Contributor

@naricc naricc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's put it in staging first as @SamMonoRT suggested. Other than that, looks good.

@fanyang-mono fanyang-mono changed the title Add test leg to run runtime tests on Android devices [Do NOT Merge] Add test leg to run runtime tests on Android devices Dec 9, 2020
@fanyang-mono
Copy link
Member Author

According to the log here. On some Android devices, it took ~2.5min to install an Android app (size: 11.4 Mb). This work item contains one test wrapper, which includes 82 tests. That means this work item does 82 times app installation, which is more than 3 hours. Some devices are faster than the others. It seems to me that the devices were picked randomly for each work item.

So the best solution is to avoid unnecessary installations. For each test wrapper, all tests use the same Android app. @akoeplinger suggested to install/uninstall once in the initialize/cleanup methods for each test wrapper. That would require this functionality(dotnet/xharness#397) in place. So for this PR, I will probably have to disable quite a few tests. And enable them when that functionality is ready to use.

@fanyang-mono fanyang-mono changed the title [Do NOT Merge] Add test leg to run runtime tests on Android devices Add test leg to run runtime tests on Android devices Dec 11, 2020
@fanyang-mono fanyang-mono merged commit 6553a49 into dotnet:master Dec 11, 2020
<!-- Set Test Wrapper running host OS -->
<PropertyGroup>
<TestWrapperTargetsWindows>false</TestWrapperTargetsWindows>
<TestWrapperTargetsWindows Condition=" ('$(TargetsWindows)' != '' And '$(TargetsWindows)' ) OR ('$(TargetOS)' == 'Android' And '$(TargetArchitecture)' == 'arm64' )">true</TestWrapperTargetsWindows>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we use ".cmd" scripts on arm64 android?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the trick is that, while traditional CoreCLR builds deal with a pair of platforms (hosting / targeting), for Mono device testing there's a triplet of platforms - host, test driver, target. Arm64 Android tablets use Windows test drivers so at that level yes, they use ".cmd" scripts.

@ghost ghost locked as resolved and limited conversation to collaborators Jan 19, 2021
@fanyang-mono fanyang-mono changed the title Add test leg to run runtime tests on Android devices Add test leg to run runtime tests on Android arm64 Apr 22, 2021
@fanyang-mono fanyang-mono deleted the android_device branch May 27, 2021 14:47
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants