[Android] Add testing support #1714

modocache · 2016-03-16T23:44:30Z

What's in this pull request?

This adds support for running tests for the stdlib built for Android, both on the host machine and on an Android device.

The Android variant of Swift may be built and tested using the following build-script invocation:

$ utils/build-script \
  -R \                                           # Build in ReleaseAssert mode.
  -T \                                           # Run all tests.
  --android \                                    # Build for Android.
  --android-deploy-device-path /data/local/tmp \ # Temporary directory on the device where Android tests are run.
  --android-ndk ~/android-ndk-r10e \             # Path to an Android NDK.
  --android-ndk-version 21 \
  --android-icu-uc ~/libicu-android/armeabi-v7a/libicuuc.so \
  --android-icu-uc-include ~/libicu-android/armeabi-v7a/icu/source/common \
  --android-icu-i18n ~/libicu-android/armeabi-v7a/libicui18n.so \
  --android-icu-i18n-include ~/libicu-android/armeabi-v7a/icu/source/i18n/

See docs/Testing.rst for more details.

Resolved bug number: None

Before merging this pull request to apple/swift repository:

Review and merge Port to Android #1442.
Test pull request on Swift continuous integration.

Triggering Swift CI

The swift-ci is triggered by writing a comment on this PR addressed to the GitHub user @swift-ci. Different tests will run depending on the specific comment that you use. The currently available comments are:

Smoke Testing

Platform	Comment
All supported platforms	@swift-ci Please smoke test
OS X platform	@swift-ci Please smoke test OS X platform
Linux platform	@swift-ci Please smoke test Linux platform

Validation Testing

Platform	Comment
All supported platforms	@swift-ci Please test
OS X platform	@swift-ci Please test OS X platform
Linux platform	@swift-ci Please test Linux platform

Note: Only members of the Apple organization can trigger swift-ci.

modocache · 2016-04-13T19:40:15Z

Now that #1442 has been merged, I've rebased this onto master, which makes the changeset much smaller. Here's the remaining feedback on these changes, from the discussion in #1442:

Port to Android #1442 (comment) - test/IRGen/report_dead_method_call.swift should be rewritten using StdlibUnittest, since the Android test runner is not capable of parsing the "fatal error" stderr output. Done in [IRGen] Rewrite dead method test in StdlibUnittest #2305.
~~Port to Android #1442 (comment) - validation-test/stdlib/CollectionType.swift.gyb should be split up, since currently it takes too long to ever complete when run on an Android host device.~~ Already done on master.
~~Port to Android #1442 (comment) - Add an overlay for var LC_ALL or setlocale on Android, such that we can avoid an Int32() cast in validation-test/stdlib/String.swift.~~ This appears to not be necessary anymore. Perhaps due to a change in ClangImporter?
Port to Android #1442 (comment) - Limit parallelism when running tests on an Android host device.
Port to Android #1442 (comment) and Port to Android #1442 (comment) - The fork/exec behavior for Android StdlibUnittest should be the same as posix_spawn.

The tests do currently run with these changes, though (see the instructions added in this commit to docs/Testing.rst). Here's the breakdown as of this comment:

Failing Tests (9):
    Swift :: 1_stdlib/PrintFloat.swift
    Swift :: 1_stdlib/tgmath.swift
    Swift :: 1_stdlib/tgmath_optimized.swift
    Swift :: DebugInfo/ASTSection.swift
    Swift :: Driver/many-inputs.swift
    Swift :: IRGen/condfail.sil
    Swift :: IRGen/objc_simd.sil
    Swift :: Reflection/typeref_decoding.swift
    Swift :: stdlib/RangeReplaceable.swift.gyb

  Expected Passes    : 7166
  Expected Failures  : 76
  Unsupported Tests  : 772
  Unexpected Failures: 9

jrose-apple · 2016-04-13T23:13:32Z

test/lit.cfg

+ # FIXME: This value should be read off of config.android_ndk_path,
+ # but I can't figure out how. The value is set via CMake, and the
+ # build directory contains a test-android-armv7/lit.site.cfg with
+ # its @SWIFT_ANDROID_NDK_PATH@ correctly replaced. So what's wrong?


What behavior are you observing?

When I invokve utils/built-script -T, the lit.site.cfg.in is correctly configured and placed in my build directory:

# build/Ninja-ReleaseAssert/swift-linux-x86_64/test-android-armv7/lit.site.cfg config.android_ndk_path = "/home/modocache/android-ndk-r10e"

But when the tests are run later during that same utils/build-script -T invocation, the attempt to access the value from within the test/lit.cfg fails:

# swift/test/lit.cfg android_ndk_path = config.android_ndk_path # lit.py: lit.cfg:738: note: Testing Android armv7-none-linux-androideabi # lit.py: TestingConfig.py:114: fatal: unable to parse config file '/home/modocache/GitHub/apple/swift/test/lit.cfg', traceback: Traceback (most recent call last): # File "/home/modocache/GitHub/apple/llvm/utils/lit/lit/TestingConfig.py", line 101, in load_from_path # exec(compile(data, path, 'exec'), cfg_globals, None) # File "/home/modocache/GitHub/apple/swift/test/lit.cfg", line 743, in <module> # android_ndk_path = config.android_ndk_path # AttributeError: TestingConfig instance has no attribute 'android_ndk_path'

Alternatively:

# swift/test/lit.cfg android_ndk_path = getattr(config, 'android_ndk_path') # lit.py: TestingConfig.py:114: fatal: unable to parse config file '/home/modocache/GitHub/apple/swift/test/lit.cfg', traceback: Traceback (most recent call last): # File "/home/modocache/GitHub/apple/llvm/utils/lit/lit/TestingConfig.py", line 101, in load_from_path # exec(compile(data, path, 'exec'), cfg_globals, None) # File "/home/modocache/GitHub/apple/swift/test/lit.cfg", line 743, in <module> # android_ndk_path = getattr(config, 'android_ndk_path') # AttributeError: TestingConfig instance has no attribute 'android_ndk_path'

Bizarrely, running lit directly works:

$ llvm/utils/lit/lit.py -v build/Ninja-ReleaseAssert/swift-linux-x86_64/test-android-armv7

I'm still trying to figure this out, I'm sure it's something silly I'm overlooking, or maybe the result of some build artifact on my host machine. 😅

My guess: you have this in test/lit.site.cfg.in, but not validation-test/lit.site.cfg.in. We should probably chain the two.

That was it!! Thanks for the tip. Yes, chaining would be nice. I think this would make a good starter task, so I wrote up a Swift report: https://bugs.swift.org/browse/SR-1266.

gribozavr · 2016-04-14T00:46:21Z

validation-test/stdlib/CollectionType.swift.gyb should be split up

Please don't :) We already split it on the swift-3-indexing-model branch, and we'd prefer to not have more merge conflicts.

gribozavr · 2016-04-14T00:47:17Z

test/1_stdlib/InputStream.swift.gyb

@@ -16,6 +16,9 @@
 // RUN: %S/../../utils/line-directive %t/InputStream.swift -- %target-run %t/a.out
 // REQUIRES: executable_test

+// FIXME: This test takes too long to complete on Android.
+// UNSUPPORTED: OS=linux-androideabi


I think it just hangs instead. It uses stdin.

gribozavr · 2016-05-31T04:07:02Z

Sorry for dropping the ball on this! Could you resolve the conflicts?

modocache · 2016-05-31T04:45:41Z

@gribozavr On the contrary, I've dropped the ball! I still haven't addressed most of the concerns you raised. But yes, I'll rebase soon. Thanks!

modocache · 2016-06-06T18:41:46Z

@gribozavr Sorry for the wait! I rebased this. When the Android tests are run with build-script -T, only one test fails: compiler_crashers/28299-swift-lookupvisibledecls.swift.

I still need to address some of your previous suggestions:

Port to Android #1442 (comment) - Limit parallelism when running tests on an Android host device.
Port to Android #1442 (comment) and Port to Android #1442 (comment) - The fork/exec behavior for Android StdlibUnittest should be the same as posix_spawn.

I'm hoping to get these done this week. Any advice you have in the meantime would be very helpful!!

gribozavr · 2016-06-06T20:23:05Z

utils/build-script-impl

+ set -x
+ "${SWIFT_SOURCE_DIR}"/utils/android/adb_push_built_products.py --ndk "${ANDROID_NDK}" --destination "${ANDROID_DEPLOY_DEVICE_PATH}" "$(build_directory ${HOST_TARGET} swift)"/lib/swift/android/
+ { set +x; } 2>/dev/null
+fi


Uploading should be done in test/CMakeLists.txt -- just set the command_upload_stdlib if SDK is Android. This way, just doing ninja check-swift-android will do the right thing and use the latest library.

modocache · 2016-06-07T20:08:08Z

I've made this pull request a bit smaller by reducing it to just what's necessary to get the test suite to run to completion. I moved fixes for various failing tests to #2939.

modocache · 2016-06-07T20:18:48Z

I think this is ready to go. I'd like to address the following issues in later pull requests. I'll create JIRA tasks to track them:

Use the Android linker when running the test suite targeting Android: I can either wait for Added -toolchain option to swiftc #2912 to be merged, or implement a PATH-based solution in this pull request. Thoughts?
Limit parallelism when running tests on an Android host device: Originally mentioned here: Port to Android #1442 (comment). I think this would be great, but would make this pull request even larger. I'd love to defer this to a follow-up.
The fork/exec behavior for Android StdlibUnittest should be the same as posix_spawn: Originally mentioned in Port to Android #1442 (comment) and Port to Android #1442 (comment). Frankly, I tried my hand at this several times but couldn't get it quite right. I'd love to defer this as well.
Other potential improvements: what happens if multiple devices are connected? Do we support running on an Android simulator or emulator? Genymotion? Can we split the tests up into host and non-host tests, like iOS?

gribozavr · 2016-06-08T07:14:16Z

@swift-ci Please test

gribozavr · 2016-06-08T07:14:20Z

@swift-ci Please smoke test

gribozavr · 2016-06-08T07:14:24Z

@swift-ci Please Python lint

gribozavr · 2016-06-08T07:18:55Z

I think this is ready to go. I'd like to address the following issues in later pull requests. I'll create JIRA tasks to track them:

Thanks!

Use the Android linker when running the test suite targeting Android: I can either wait for Added -toolchain option to swiftc #2912 to be merged, or implement a PATH-based solution in this pull request. Thoughts?

I think the -toolchain PR has a good chance to land soon.

Limit parallelism when running tests on an Android host device: Originally mentioned here: Port to Android #1442 (comment). I think this would be great, but would make this pull request even larger. I'd love to defer this to a follow-up.

The fork/exec behavior for Android StdlibUnittest should be the same as posix_spawn: Originally mentioned in Port to Android #1442 (comment) and Port to Android #1442 (comment). Frankly, I tried my hand at this several times but couldn't get it quite right. I'd love to defer this as well.

Makes sense.

Other potential improvements: what happens if multiple devices are connected?

This would help with test latency! Usually the host has more cores than the device, so you can usually connect multiple devices "for free" without saturating the host CPU.

Do we support running on an Android simulator or emulator? Genymotion?

That would be great!

Can we split the tests up into host and non-host tests, like iOS?

The test annotations (executable_test) should be the same, and the CMake logic for generating targets is also shared. I'm wondering if it just works already! Try running the check-swift-validation-only_non_executable-android-armv7 target.

modocache · 2016-06-08T18:44:27Z

@gribozavr Looks like everything passed! Is this OK to merge?

modocache · 2016-06-09T02:05:19Z

Rebased on top of #2919.

modocache · 2016-06-09T02:05:55Z

@swift-ci Please Python lint

modocache · 2016-06-09T02:06:14Z

@swift-ci please test

gribozavr · 2016-06-09T03:28:51Z

Re-testing after #2880 was merged.

@swift-ci Please test

gribozavr · 2016-06-09T03:28:55Z

@swift-ci Please smoke test

ddunbar · 2016-06-09T03:31:00Z

utils/build-script-impl

- # FIXME: Allow Android host tests to be enabled/disabled by the
- # build script.
- test_this_target=
+ test_this_target=$(not ${SKIP_TEST_ANDROID_HOST})


Since #2880 just landed, I believe there will need to be an update to the matching logic in build-script.

Thanks for the heads up, @ddunbar! Rebasing now.

This adds support for running tests for the stdlib built for Android, both on the host machine and on an Android device. The Android variant of Swift may be built and tested using the following `build-script` invocation: ``` $ utils/build-script \ -R \ # Build in ReleaseAssert mode. -T \ # Run all tests. --android \ # Build for Android. --android-deploy-device-path /data/local/tmp \ # Temporary directory on the device where Android tests are run. --android-ndk ~/android-ndk-r10e \ # Path to an Android NDK. --android-ndk-version 21 \ --android-icu-uc ~/libicu-android/armeabi-v7a/libicuuc.so \ --android-icu-uc-include ~/libicu-android/armeabi-v7a/icu/source/common \ --android-icu-i18n ~/libicu-android/armeabi-v7a/libicui18n.so \ --android-icu-i18n-include ~/libicu-android/armeabi-v7a/icu/source/i18n/ ``` See docs/Testing.rst for more details.

modocache · 2016-06-09T17:04:49Z

Rebased and presumably ready to go. Going to kick off some tests.

@swift-ci Please test

modocache · 2016-06-09T17:04:58Z

@swift-ci Please smoke test

modocache · 2016-06-09T17:05:09Z

@swift-ci Please Python lint

modocache · 2016-06-09T19:45:14Z

@gribozavr All green! Confirmed that Android tests run on my local machine as well. Is this good to go?

modocache · 2016-06-13T20:34:38Z

@gribozavr Friendly ping! :)

gribozavr · 2016-06-14T03:37:42Z

Sorry for the delay, your PR LGTM!

@swift-ci Please smoke test and merge

modocache · 2016-06-14T04:50:51Z

Excellent, thanks! Tomorrow I'll create a bunch of tasks for the failing tests, and for the improvements we discussed that didn't make it into this pull request.

J-Bobb · 2016-06-20T21:20:38Z

Sent from my Samsung Galaxy smartphone.-------- Original message --------From: Brian Gesiak notifications@github.com Date: 06/08/2016 10:06 PM (GMT-05:00) To: apple/swift swift@noreply.github.com Subject: Re: [apple/swift] [Android] Add testing support (#1714)
@swift-ci please test

@swift-ci Please Python lint

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

modocache mentioned this pull request Mar 16, 2016

Port to Android #1442

Merged

modocache force-pushed the tests branch 3 times, most recently from da6d0e2 to 0d8aa27 Compare April 13, 2016 18:48

jrose-apple reviewed Apr 13, 2016
View reviewed changes

gribozavr reviewed Apr 14, 2016
View reviewed changes

modocache force-pushed the tests branch 3 times, most recently from 496a0da to 1a01d57 Compare April 25, 2016 03:27

modocache mentioned this pull request Apr 25, 2016

[test] Add Android (and FreeBSD) to OS checks #2300

Merged

1 task

modocache force-pushed the tests branch from 1a01d57 to 6910d12 Compare April 25, 2016 15:29

modocache mentioned this pull request Apr 25, 2016

[IRGen] Rewrite dead method test in StdlibUnittest #2305

Merged

1 task

modocache force-pushed the tests branch 3 times, most recently from 7ef23bc to 587f448 Compare April 29, 2016 19:30

rintaro mentioned this pull request May 31, 2016

[build-script] use explicit build/test list between build-script and build-script-impl #2780

Closed

1 task

modocache force-pushed the tests branch from 587f448 to 380a4f7 Compare June 6, 2016 18:38

gribozavr reviewed Jun 6, 2016
View reviewed changes

karwa mentioned this pull request Jun 8, 2016

[build-script] Compute targets in build-script #2880

Merged

1 task

modocache force-pushed the tests branch from 25506f2 to c5f605d Compare June 9, 2016 02:04

ddunbar reviewed Jun 9, 2016
View reviewed changes

modocache force-pushed the tests branch from c5f605d to ee51b7a Compare June 9, 2016 17:03

swift-ci merged commit a1f1976 into swiftlang:master Jun 14, 2016

modocache deleted the tests branch June 14, 2016 04:49

modocache mentioned this pull request Jun 28, 2016

Added -toolchain option to swiftc #2912

Merged

1 task

modocache mentioned this pull request Jul 18, 2016

[lit] Rename config.targets_to_build => config.llvm_targets and allow… #3577

Merged

1 task

modocache mannequin mentioned this pull request Apr 19, 2016

[SR-1264] Swift doesn't build on Android #43872

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Android] Add testing support #1714

[Android] Add testing support #1714

modocache commented Mar 16, 2016

modocache commented Apr 13, 2016 •

edited

Loading

jrose-apple Apr 13, 2016

modocache Apr 14, 2016

jrose-apple Apr 14, 2016

modocache Apr 19, 2016

gribozavr commented Apr 14, 2016

gribozavr Apr 14, 2016

gribozavr commented May 31, 2016

modocache commented May 31, 2016

modocache commented Jun 6, 2016

gribozavr Jun 6, 2016

modocache commented Jun 7, 2016

modocache commented Jun 7, 2016

gribozavr commented Jun 8, 2016

gribozavr commented Jun 8, 2016

gribozavr commented Jun 8, 2016

gribozavr commented Jun 8, 2016

modocache commented Jun 8, 2016

modocache commented Jun 9, 2016

modocache commented Jun 9, 2016 •

edited

Loading

modocache commented Jun 9, 2016

gribozavr commented Jun 9, 2016

gribozavr commented Jun 9, 2016

ddunbar Jun 9, 2016

modocache Jun 9, 2016

modocache commented Jun 9, 2016

modocache commented Jun 9, 2016

modocache commented Jun 9, 2016

modocache commented Jun 9, 2016 •

edited

Loading

modocache commented Jun 13, 2016

gribozavr commented Jun 14, 2016

modocache commented Jun 14, 2016

J-Bobb commented Jun 20, 2016

[Android] Add testing support #1714

[Android] Add testing support #1714

Conversation

modocache commented Mar 16, 2016

What's in this pull request?

Resolved bug number: None

modocache commented Apr 13, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gribozavr commented Apr 14, 2016

Choose a reason for hiding this comment

gribozavr commented May 31, 2016

modocache commented May 31, 2016

modocache commented Jun 6, 2016

Choose a reason for hiding this comment

modocache commented Jun 7, 2016

modocache commented Jun 7, 2016

gribozavr commented Jun 8, 2016

gribozavr commented Jun 8, 2016

gribozavr commented Jun 8, 2016

gribozavr commented Jun 8, 2016

modocache commented Jun 8, 2016

modocache commented Jun 9, 2016

modocache commented Jun 9, 2016 • edited Loading

modocache commented Jun 9, 2016

gribozavr commented Jun 9, 2016

gribozavr commented Jun 9, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

modocache commented Jun 9, 2016

modocache commented Jun 9, 2016

modocache commented Jun 9, 2016

modocache commented Jun 9, 2016 • edited Loading

modocache commented Jun 13, 2016

gribozavr commented Jun 14, 2016

modocache commented Jun 14, 2016

J-Bobb commented Jun 20, 2016

modocache commented Apr 13, 2016 •

edited

Loading

modocache commented Jun 9, 2016 •

edited

Loading

modocache commented Jun 9, 2016 •

edited

Loading