Cordova Paramedic (Test Automation)

Paramedic • noun provides advanced levels of care at the point of illness or injury, including out of hospital treatment, and diagnostic services

cordova-paramedic is a tool to automate execution of Cordova plugins tests (via cordova-plugin-test-framework).

You can use Paramedic to build and run a Cordova app with plugin tests, run these tests on local and remote emulators on Sauce Labs, and report the results. It can be used on a local or Continuous Integration environment.

Cordova Paramedic is currently used to automatically run all plugin tests on CI.

(See this workshop instructions for some additional explanation.)

Table of Contents

• Supported Cordova Platforms
• What does it do?
• Installation
• Usage
  • Common usages
• Command Line Interface
  • What to build and test
  • Emulator/Device to use for tests
  • Test Result Server
  • Test Configuration
  • Sauce Labs
• Configuration file
• API Interface
• Quirks
  • Windows

Supported Cordova Platforms

• Android
• Browser
• iOS
• Windows

What does it do?

A full Paramedic run will:

1. Create and prepare the app

   1. Create a temporary Cordova project with cordova create
   2. Install various plugins with cordova plugin add %local_path% (e.g. cordova plugin add ../cordova-plugin-inappbrowser):
      • the plugin to be tested (e.g. ../cordova-plugin-inappbrowser)
      • the tests of this plugin (e.g. ../cordova-plugin-inappbrowser/tests)
      • cordova-plugin-test-framework (from npm)
      • local paramedic-plugin
   3. Update the app start page to the test page at cdvtests/index.html (provided by cordova-plugin-test-framework and the plugin tests)
   4. Add the platform to be tested with cordova platform add ...
   5. Confirm the requirements for that platform are met with cordova requirements ...
   6. Start a local socket server for communication between the app running on a device/emulator and paramedic
   7. Make the server address known to the app

2. Run the tests

   • Either run the main tests locally...

     1. Skip main tests if option set (platform != android)
     2. Start a file transfer server if required
     3. Get the test command for the platform
     4. Manipulate permissions on iOS
     5. Run the app (open in browser, start emulator, run on device or emulator) and start the tests by doing so
     6. Skip main tests if option set
     7. Skip tests if action = run|emulate (= build)
     8. Wait for device to connect to server before timeout
     9. Wait for the tests results
        1. Time out if "connection takes to long" TODO (failure)
        2. Receive and handle "tests are done" (success) and "device disconnected" (failure) events
     10. (browser) Close the running browser
     11. Run the Appium tests (with sauce = false)
   • ... or on Sauce Labs

     1. Build, package and upload the app to Sauce Labs or (platform = browser) open the app in a browser
     2. (platform = browser) Connect to Sauce Connect (Proxy)
     3. Connect to Web Driver on Sauce Labs
     4. Navigate Web Driver to correct page (browser) or webview (apps)
     5. Click "Auto Tests" if a plugin *wkwebview* is installed
     6. Find out if the "permission buster" should be skipped (plugins splashscreen or inappbrowser, browser): skipBuster
     7. Start polling in the background for events using the Web Driver (submitting skipBuster as well) every 2.5 seconds
     8. Wait for the tests results
        1. Time out if "connection takes to long" TODO (failure)
        2. Receive and handle "tests are done" (success) and "device disconnected" (failure) events
     9. Log success or failure
     10. Quit Web Driver
     11. (platform = browser) Close the open browser
     12. Close connection to Sauce Connect (Proxy)
     13. Run the Appium tests on Sauce Labs (with sauce = true)
   • Run the Appium tests

     1. Skip if action = build
     2. Skip is Appium should be skipped
     3. Skip if platform != android or ios
     4. !sauce: Error when no targetObj TODO
     5. Create Appium options
     6. Create AppiumRunner with options
        1. Prepare the submitted options
        2. Create screenshot directory
        3. Find the tests in plugin paths
        4. Set globals for the tests
     7. Skip if no Appium tests were found
     8. Prepare App in AppiumRunner
        1. Remove server address from app
        2. Reconfigure app (modify preferences + CSP, add plugin) <!-- 367, 375, 385 -- >
        3. Build app
     9. (sauce) Package and Upload the App to Sauce Labs
     10. Run tests via AppiumRunner
         1. (!sauce) Start iOS Proxy (ios_webkit_debug_proxy)
         2. (!sauce) Install (npm install appium) and start Appium server
         3. Start to run the Appium tests
         4. Handle eventual exceptions, return the result
3. Clean up

   1. (!sauce)
      1. Handle timeouts of test execution above
      2. Collect Device Logs
      3. Uninstall App
      4. Kill Emulator Process
   2. (sauce) Display Sauce run details
   3. Clean up Project

Installation

Using npmjs registry version:

npm install -g cordova-paramedic

Using GitHub version:

npm install -g github:apache/cordova-paramedic

or

git clone https://github.com/apache/cordova-paramedic

If cloning from GitHub, you will need to run npm link inside the checkout repository.

Alternativly, replace all occurences of cordova-paramedic with the command:

• cordova-paramedic/main.js for Linux or macOS
• node cordova-paramedic/main.js for Windows

Usage

Paramedic parameters can be passed via command line arguments or separate configuration file:

By Command Line Arguments:

cordova-paramedic --platform PLATFORM --plugin PATH <other parameters>

By Configuration File:

cordova-paramedic --config ./sample-config/.paramedic.config.js

Common usages

Some common use cases of Paramedic:

Run without any parameters to get a list of supported parameters:

cordova-paramedic

Test your current plugin on an Android emulator:

cordova-paramedic --platform android --plugin ./

Test your current plugin on a specific Android device (ID via adb devices -l):

cordova-paramedic --platform android --plugin ./ --target 02e7f7e9215da7f8 --useTunnel

Test your current plugin on an Android 7.0 emulator on Sauce Labs:

cordova-paramedic --config conf/pr/android-7.0 --plugin ./

Command Line Interface

What to build and test

--platform (required)

Specifies target Cordova platform (could refer to local directory, npm or git)

cordova-paramedic --platform ios --plugin cordova-plugin-inappbrowser
cordova-paramedic --platform ios@4.0 --plugin cordova-plugin-inappbrowser
cordova-paramedic --platform ios@../cordova-ios --plugin cordova-plugin-inappbrowser
cordova-paramedic --platform ios@https://github.com/apache/cordova-ios.git#4.1.0 --plugin cordova-plugin-inappbrowser

--plugin (required)

Specifies test plugin, you may specify multiple --plugin flags and they will all be installed and tested together. You can refer to absolute path, npm registry or git repo. If the plugin requires variables to install, you can specify them along with its name.

cordova-paramedic --platform ios --plugin cordova-plugin-inappbrowser
cordova-paramedic --platform ios --plugin 'azure-mobile-engagement-cordova --variable AZME_IOS_CONNECTION_STRING=Endpoint=0;AppId=0;SdkKey=0'
cordova-paramedic --platform ios --plugin https://github.com/apache/cordova-plugin-inappbrowser
// several plugins
cordova-paramedic --platform ios --plugin cordova-plugin-inappbrowser --plugin cordova-plugin-contacts

--verbose (optional)

Verbose mode. Display more information output

cordova-paramedic --platform ios --plugin cordova-plugin-inappbrowser --verbose

--cli (optional)

A path to Cordova CLI. Useful when you're testing against locally installed Cordova version.

cordova-paramedic --platform android --plugin cordova-plugin-device --cli ./cordova-cli/bin/cordova

--justbuild (optional)

Just builds the project, without running the tests.

cordova-paramedic --platform ios --plugin cordova-plugin-inappbrowser --justbuild

Emulator/Device to use for tests

--target (optional)

For Android: The device ID (from adb devices -l) of a device the tests should be run on.

cordova-paramedic --platform android --plugin cordova-plugin-contacts --target 02e7f7e9215da7f8

For iOS: A string that is used to pick the device (from the cordova run --list --emulator output) the tests should be run on.

cordova-paramedic --platform ios --plugin cordova-plugin-contacts --target "iPhone-8"

Test Result Server

--useTunnel (optional)

Use a tunnel (via localtunnel) instead of local address (default is false). Useful when testing on real devices and don't want to specify external IP address (see --externalServerUrl below) of paramedic server.

cordova-paramedic --platform ios --plugin cordova-plugin-inappbrowser --useTunnel

--externalServerUrl (optional)

Useful when testing on real device (--device parameter) so that tests results from device could be posted back to paramedic server.

cordova-paramedic --platform ios --plugin cordova-plugin-inappbrowser --externalServerUrl http://10.0.8.254

--port (optional)

Port to use for posting results from emulator back to paramedic server (default is from 8008). You can also specify a range using --startport and endport and paramedic will select the first available.

cordova-paramedic --platform ios --plugin cordova-plugin-inappbrowser --port 8010
cordova-paramedic --platform ios --plugin cordova-plugin-inappbrowser --startport 8000 endport 8020

Test configuration

--timeout (optional)

Time in millisecs to wait for tests to pass|fail (defaults to 10 minutes).

cordova-paramedic --platform ios --plugin cordova-plugin-inappbrowser --timeout 30000

--outputDir (optional)

Directory location to store test results in junit format and the device logs

cordova-paramedic --platform ios --plugin cordova-plugin-inappbrowser --outputDir /Users/sampleuser/testresults

--cleanUpAfterRun (optional)

Flag to indicate the sample application folder must be deleted.

cordova-paramedic --platform ios --plugin cordova-plugin-inappbrowser --cleanUpAfterRun

--logMins (optional)

Windows only parameter to indicate the duration for which the device logs to be fetched.

cordova-paramedic --platform windows --plugin cordova-plugin-inappbrowser --logMins 15

--tccDb (optional)

iOS only parameter. The path to the sample TCC DB file, with permissions, to be copied to the simulator.

cordova-paramedic --platform ios --plugin cordova-plugin-contacts --tccDbPath tcc.db

--args (optional)

Add additional parameters to the cordova build and cordova run commands.

cordova-paramedic --platform ios --plugin cordova-plugin-contacts --args=--buildFlag='-UseModernBuildSystem=0'

Sauce Labs

--shouldUseSauce (optional)

Run tests on Sauce Labs. You'll need to specify Sauce Labs username and access key using either --sauceUser and --sauceKey arguments or SAUCE_USERNAME and SAUCE_ACCESS_KEY environment variables.

--sauceUser (optional)

Sauce Labs username. Alternatively set via the SAUCE_USERNAME environment variable.

--sauceKey (optional)

Sauce Labs access key. Alternatively set via the SAUCE_ACCESS_KEY environment variable.

cordova-paramedic --platform ios --plugin cordova-plugin-contacts --shouldUseSauce --sauceUser ***** --sauceKey ***** --buildName "paramedic-test-01"

--buildName (optional)

Build name to show on Sauce Labs dashboard. If omitted, will use "Paramedic sauce test" and a timestamp.

--sauceDeviceName (optional)

Name of the Sauce Labs emulator or browser. For example, "iPhone Simulator" or "firefox". Please refer to the Sauce Labs platforms list to see available device names.

--saucePlatformVersion (optional)

Platform version of the Sauce Labs emulator OS, or version of the browser (if testing browser platform). For example, "9.3" or "54.0". Please refer to the Sauce Labs platforms list to see available platform versions.

--sauceAppiumVersion (optional)

Appium version to use when running on Sauce Labs. For example, "1.5.3".

cordova-paramedic --platform ios --plugin cordova-plugin-contacts --shouldUseSauce --sauceUser ***** --sauceKey ***** --sauceDeviceName 'iPad Simulator" --saucePlatformVersion 9.1 --appiumVersion 1.5.2

Configuration file

Configuration file is used when no parameters are passed to cordova-paramedic call or explicitly specified via --config parameter:

cordova-paramedic           <- paramedic will attempt to find .paramedic.config.js in working directory
cordova-paramedic --config ./sample-config/.paramedic.config.js

Example configuration file is showed below.

module.exports = {
    // "externalServerUrl": "http://10.0.8.254",
    "useTunnel": true,
    "plugins": [
        "https://github.com/apache/cordova-plugin-inappbrowser"
    ],
    "platform": "windows",
    "action": "run",
    "args": "--archs=x64 -- --appx=uap"
}

More configuration file examples could be found in sample-config folder.

API Interface

You can also use cordova-paramedic as a module directly:

var paramedic = require('cordova-paramedic');
paramedic.run(config);

Quirks

Windows

For Paramedic to work correctly for Windows apps you'll need to allow the loopback for "HelloCordova" app using Windows Loopback Exemption Manager (download).