Skip to content

Commit

Permalink
[Android] Add testing support
Browse files Browse the repository at this point in the history
This adds support for running tests for the stdlib built
for Android, both on the host machine and on an Android device.

The Android variant of Swift may be built and tested using
the following `build-script` invocation:

```
$ utils/build-script \
  -R \                                           # Build in ReleaseAssert mode.
  -T \                                           # Run all tests.
  --android \                                    # Build for Android.
  --android-deploy-device-path /data/local/tmp \ # Temporary directory on the device where Android tests are run.
  --android-ndk ~/android-ndk-r10e \             # Path to an Android NDK.
  --android-ndk-version 21 \
  --android-icu-uc ~/libicu-android/armeabi-v7a/libicuuc.so \
  --android-icu-uc-include ~/libicu-android/armeabi-v7a/icu/source/common \
  --android-icu-i18n ~/libicu-android/armeabi-v7a/libicui18n.so \
  --android-icu-i18n-include ~/libicu-android/armeabi-v7a/icu/source/i18n/
```

See docs/Testing.rst for more details.
  • Loading branch information
modocache committed Apr 13, 2016
1 parent 1c3d998 commit 0d8aa27
Show file tree
Hide file tree
Showing 24 changed files with 484 additions and 9 deletions.
42 changes: 42 additions & 0 deletions docs/Testing.rst
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,48 @@ targets mentioned above and modify it as necessary. lit.py also has several
useful features, like timing tests and providing a timeout. Check these features
out with ``lit.py -h``.

Running tests hosted on an Android device
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

You may run tests targeting android-armv7 by connecting a device to run the
tests on, then pushing the necessary dependencies to that device.

1. Connect your Android device to your computer via USB. Ensure that remote
debugging is enabled for that device by following the official instructions:
https://developer.chrome.com/devtools/docs/remote-debugging.
2. Confirm the device is connected by running ``adb devices``. You should see
your device listed.
3. Push the built products for android-armv7 to your device, using the
``utils/android/adb_push_built_products.py`` script. For example, to push
the built products at a build directory
``~/swift/Ninja-ReleaseAssert/swift-linux-x86_64``, with an Android NDK
placed at ``~/android-ndk-r10e``, run the following:

$ utils/android/adb_push_built_products.py \
~/swift/build/Ninja-ReleaseAssert/swift-linux-x86_64/lib/swift/android/ \
--ndk ~/android-ndk-r10e
4. Run the test suite for the ``android-armv7`` target.

You may run the tests using the build script as well. Specifying a
``--android-deploy-device-path`` pushes the built Android products to your
device as part of the build:

$ utils/build-script \
-R \ # Build in ReleaseAssert mode.
-T \ # Run all tests.
--android \ # Build for Android.
--android-deploy-device-path /data/local/tmp \ # Temporary directory on the device where Android tests are run.
--android-ndk ~/android-ndk-r10e \ # Path to an Android NDK.
--android-ndk-version 21 \
--android-icu-uc ~/libicu-android/armeabi-v7a/libicuuc.so \
--android-icu-uc-include ~/libicu-android/armeabi-v7a/icu/source/common \
--android-icu-i18n ~/libicu-android/armeabi-v7a/libicui18n.so \
--android-icu-i18n-include ~/libicu-android/armeabi-v7a/icu/source/i18n/

You must run the Linux tests once in order to build the Android test suite.
After that, you may run the above command with the ``--skip-test-linux`` option
to only run Android tests.

Writing tests
-------------

Expand Down
3 changes: 3 additions & 0 deletions test/1_stdlib/InputStream.swift.gyb
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,9 @@
// RUN: %S/../../utils/line-directive %t/InputStream.swift -- %target-run %t/a.out
// REQUIRES: executable_test

// FIXME: This test takes too long to complete on Android.
// UNSUPPORTED: OS=linux-androideabi

import StdlibUnittest


Expand Down
4 changes: 4 additions & 0 deletions test/1_stdlib/POSIX.swift
Original file line number Diff line number Diff line change
@@ -1,6 +1,10 @@
// RUN: %target-run-simple-swift
// REQUIRES: executable_test

// Android Bionic does not provide a working implementation of
// <semaphore.h>.
// XFAIL: OS=linux-androideabi

import StdlibUnittest
#if os(Linux)
import Glibc
Expand Down
1 change: 1 addition & 0 deletions test/ClangModules/autolinking.swift
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@
// UNSUPPORTED: OS=linux-gnu
// UNSUPPORTED: OS=linux-gnueabihf
// UNSUPPORTED: OS=freebsd
// UNSUPPORTED: OS=linux-androideabi

import LinkMusket
import LinkFramework
Expand Down
4 changes: 3 additions & 1 deletion test/IRGen/c_layout.sil
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
// RUN: %target-swift-frontend -I %S/Inputs/abi %s -emit-ir | FileCheck %s --check-prefix=CHECK-%target-cpu
// FIXME: Rather than passing `-fsigned-char`, this should test each platform's expected char
// signedness. For Android, this means implicitly unsigned. See: https://github.com/apple/swift/pull/1103
// RUN: %target-swift-frontend -Xcc -fsigned-char -I %S/Inputs/abi %s -emit-ir | FileCheck %s --check-prefix=CHECK-%target-cpu

sil_stage canonical
import c_layout
Expand Down
3 changes: 3 additions & 0 deletions test/IRGen/report_dead_method_call.swift
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,9 @@
// RUN: %target-run %t/a.out p1 p2 2> %t/err3.log; FileCheck %s < %t/err3.log
// REQUIRES: executable_test

// This test correctly outputs "fatal error" on an Android device, but
// the Android test runner interprets the abort as a test failure.
// XFAIL: OS=linux-androideabi

// The -disable-access-control option let us "call" methods, which are removed
// by dead method elimination.
Expand Down
2 changes: 1 addition & 1 deletion test/Prototypes/FloatingPoint.swift
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ extension UInt32 : FloatingPointRepresentation {
// Ewwww? <rdar://problem/20060017>
#if os(OSX) || os(iOS) || os(watchOS) || os(tvOS)
import Darwin
#elseif os(Linux)
#elseif os(Linux) || os(Android)
import Glibc
#endif

Expand Down
1 change: 1 addition & 0 deletions test/Serialization/autolinking.swift
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@
// UNSUPPORTED: OS=linux-gnu
// UNSUPPORTED: OS=linux-gnueabihf
// UNSUPPORTED: OS=freebsd
// UNSUPPORTED: OS=linux-androideabi

import someModule

Expand Down
59 changes: 59 additions & 0 deletions test/lit.cfg
Original file line number Diff line number Diff line change
Expand Up @@ -733,6 +733,65 @@ elif run_os == 'linux-gnu' or run_os == 'linux-gnueabihf' or run_os == 'freebsd'
config.target_ld = (
"ld -L%s" %
(os.path.join(test_resource_dir, config.target_sdk_name)))
elif run_os == 'linux-androideabi':
# Android
lit_config.note("Testing Android " + config.variant_triple)
# FIXME: This value should be read off of config.android_ndk_path,
# but I can't figure out how. The value is set via CMake, and the
# build directory contains a test-android-armv7/lit.site.cfg with
# its @SWIFT_ANDROID_NDK_PATH@ correctly replaced. So what's wrong?
android_ndk_path = "/home/modocache/android-ndk-r10e"

config.target_object_format = "elf"
config.target_runtime = "native"
config.target_swift_autolink_extract = inferSwiftBinary("swift-autolink-extract")
config.target_sdk_name = "android"
android_linker_opt = "-L {libcxx} -L {libgcc}".format(
libcxx=os.path.join(android_ndk_path,
"sources", "cxx-stl", "llvm-libc++", "libs",
"armeabi-v7a"),
libgcc=os.path.join(android_ndk_path,
"toolchains", "arm-linux-androideabi-4.8",
"prebuilt", "linux-x86_64", "lib", "gcc",
"arm-linux-androideabi", "4.8"))
config.target_build_swift = (
'%s -target %s -sdk %s %s -Xlinker -pie %s %s %s %s'
% (config.swiftc, config.variant_triple, config.variant_sdk,
android_linker_opt, resource_dir_opt, mcp_opt,
config.swift_test_options, swift_execution_tests_extra_flags))
config.target_swift_frontend = (
'%s -frontend -target %s -sdk %s %s %s'
% (config.swift, config.variant_triple, config.variant_sdk,
android_linker_opt, resource_dir_opt))
subst_target_swift_frontend_mock_sdk = config.target_swift_frontend
subst_target_swift_frontend_mock_sdk_after = ""
config.target_run = os.path.join(
config.swift_src_root, 'utils', 'android', 'adb_test_runner.py')
config.target_sil_opt = (
'%s -target %s %s %s' %
(config.sil_opt, config.variant_triple, resource_dir_opt, mcp_opt))
config.target_swift_ide_test = (
'%s -target %s %s %s %s' %
(config.swift_ide_test, config.variant_triple, resource_dir_opt,
mcp_opt, ccp_opt))
subst_target_swift_ide_test_mock_sdk = config.target_swift_ide_test
subst_target_swift_ide_test_mock_sdk_after = ""
config.target_swiftc_driver = (
"%s -target %s -sdk %s %s %s %s" %
(config.swiftc, config.variant_triple, config.variant_sdk,
android_linker_opt, resource_dir_opt, mcp_opt))
config.target_swift_modulewrap = (
'%s -modulewrap -target %s' %
(config.swiftc, config.variant_triple))
config.target_clang = (
"clang++ -target %s %s" %
(config.variant_triple, clang_mcp_opt))
config.target_ld = (
"ld -L%s" %
(os.path.join(test_resource_dir, config.target_sdk_name)))
# config.environment['ANDROID_NDK_HOME'] = os.getenv("ANDROID_NDK_HOME")
# The Swift interpreter is not available when targeting Android.
config.available_features.remove('swift_interpreter')

else:
lit_config.fatal("Don't know how to define target_run and "
Expand Down
1 change: 1 addition & 0 deletions test/lit.site.cfg.in
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ config.variant_sdk = "@VARIANT_SDK@"
config.variant_suffix = "@VARIANT_SUFFIX@"
config.swiftlib_dir = "@LIT_SWIFTLIB_DIR@"
config.darwin_xcrun_toolchain = "@SWIFT_DARWIN_XCRUN_TOOLCHAIN@"
config.android_ndk_path = "@SWIFT_ANDROID_NDK_PATH@"

config.coverage_mode = "@SWIFT_ANALYZE_CODE_COVERAGE@"

Expand Down
Empty file added utils/android/adb/__init__.py
Empty file.
151 changes: 151 additions & 0 deletions utils/android/adb/commands.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,151 @@
# adb/commands.py - Run executables on an Android device -*- python -*-
#
# This source file is part of the Swift.org open source project
#
# Copyright (c) 2014 - 2016 Apple Inc. and the Swift project authors
# Licensed under Apache License v2.0 with Runtime Library Exception
#
# See http://swift.org/LICENSE.txt for license information
# See http://swift.org/CONTRIBUTORS.txt for the list of Swift project authors
#
# ----------------------------------------------------------------------------
#
# Push executables to an Android device and run them, capturing their output
# and exit code.
#
# ----------------------------------------------------------------------------

from __future__ import print_function

import subprocess
import tempfile
import uuid


# A temporary directory on the Android device.
DEVICE_TEMP_DIR = '/data/local/tmp'


def shell(args):
"""
Execute 'adb shell' with the given arguments.
Raise an exception if 'adb shell' returns a non-zero exit code.
Note that this only occurs if communication with the connected device
fails, not if the command run on the device fails.
"""
return subprocess.check_output(['adb', 'shell'] + args)


def push(local_path, device_path):
"""Move the file at the given local path to the path on the device."""
return subprocess.check_output(['adb', 'push', local_path, device_path],
stderr=subprocess.STDOUT).strip()


def _create_executable_on_device(device_path, contents):
_, tmp = tempfile.mkstemp()
with open(tmp, 'w') as f:
f.write(contents)
push(tmp, device_path)
shell(['chmod', '755', device_path])


def execute_on_device(executable_path,
executable_arguments,
rerun_failures):
"""
Run an executable on an Android device.
Push an executable at the given 'executable_path' to an Android device,
then execute that executable on the device, passing any additional
'executable_arguments'. Return 0 if the executable succeeded when run on
device, and 1 otherwise.
This function is not as simple as calling 'adb shell', for two reasons:
1. 'adb shell' can only take input up to a certain length, so it fails for
long executable names or when a large amount of arguments are passed to
the executable. This function attempts to limit the size of any string
passed to 'adb shell'.
2. 'adb shell' ignores the exit code of any command it runs. This function
therefore uses its own mechanisms to determine whether the executable
had a successful exit code when run on device.
If 'rerun_failures' is specified, this function re-runs failing tests,
printing their output to stdout.
"""
# We'll be running the executable in a temporary directory in
# /data/local/tmp. `adb shell` has trouble with commands that
# exceed a certain length, so to err on the safe side we only
# use the first 10 characters of the UUID.
uuid_dir = '{}/{}'.format(DEVICE_TEMP_DIR, str(uuid.uuid4())[:10])
shell(['mkdir', '-p', uuid_dir])

# `adb` can only handle commands under a certain length. No matter what the
# original executable's name, on device we call it `__executable`.
executable = '{}/__executable'.format(uuid_dir)
push(executable_path, executable)

# When running the executable on the device, we need to pass it the same
# arguments, as well as specify the correct LD_LIBRARY_PATH. Save these
# to a file we can easily call multiple times.
executable_with_args = '{}/__executable_with_args'.format(uuid_dir)
_create_executable_on_device(
executable_with_args,
'LD_LIBRARY_PATH={uuid_dir}:{tmp_dir} '
'{executable} {executable_arguments}'.format(
uuid_dir=uuid_dir,
tmp_dir=DEVICE_TEMP_DIR,
executable=executable,
executable_arguments=' '.join(executable_arguments)))

# Write the output from the test executable to a file named '__stdout', and
# if the test executable succeeds, write 'SUCCEEDED' to a file
# named '__succeeded'. We do this because `adb shell` does not report
# the exit code of the command it executes on the device, so instead we
# check the '__succeeded' file for our string.
executable_stdout = '{}/__stdout'.format(uuid_dir)
succeeded_token = 'SUCCEEDED'
executable_succeeded = '{}/__succeeded'.format(uuid_dir)
executable_piped = '{}/__executable_piped'.format(uuid_dir)
_create_executable_on_device(
executable_piped,
'{executable_with_args} > {executable_stdout} && '
'echo "{succeeded_token}" > {executable_succeeded}'.format(
executable_with_args=executable_with_args,
executable_stdout=executable_stdout,
succeeded_token=succeeded_token,
executable_succeeded=executable_succeeded))

# We've pushed everything we need to the device.
# Now execute the wrapper script.
shell([executable_piped])

# Grab the results of running the executable on device.
stdout = shell(['cat', executable_stdout])
exitcode = shell(['cat', executable_succeeded])
if not exitcode.startswith(succeeded_token):
debug_command = '$ adb shell {}'.format(executable_with_args)
print('Executable exited with a non-zero code on the Android device.\n'
'Device stdout:\n'
'{stdout}\n'
'To debug, run:\n'
'{debug_command}\n'.format(
stdout=stdout,
debug_command=debug_command))

if rerun_failures:
print('Re-running executable with command:\n'
'{}\n'.format(debug_command))
print(shell([executable_with_args]))

# Exit early so that the output isn't passed to FileCheck, nor are any
# temporary directories removed; this allows the user to re-run
# the executable on the device.
return 1

print(stdout)

shell(['rm', '-rf', uuid_dir])
return 0
18 changes: 18 additions & 0 deletions utils/android/adb_push_built_products.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
#!/usr/bin/env python
# adb_push_build_products.py - Calls adb_push_build_products.main -*- python -*-
#
# This source file is part of the Swift.org open source project
#
# Copyright (c) 2014 - 2016 Apple Inc. and the Swift project authors
# Licensed under Apache License v2.0 with Runtime Library Exception
#
# See http://swift.org/LICENSE.txt for license information
# See http://swift.org/CONTRIBUTORS.txt for the list of Swift project authors

import sys

from adb_push_built_products.main import main


if __name__ == '__main__':
sys.exit(main())
Empty file.
Loading

0 comments on commit 0d8aa27

Please sign in to comment.