diff --git a/docs.json b/docs.json index 84e6b8bde1..e6da1b3398 100644 --- a/docs.json +++ b/docs.json @@ -815,11 +815,11 @@ }, { "type": "markdown", - "url": "https://github.com/ARMmbed/mbed-os-5-docs/blob/development/docs/tools/testing/testing.md" + "url": "https://github.com/ARMmbed/mbed-os-5-docs/blob/development/docs/tools/testing/testing_greentea.md" }, { "type": "markdown", - "url": "https://github.com/ARMmbed/mbed-os-5-docs/blob/development/docs/tools/testing/greentea.md" + "url": "https://github.com/ARMmbed/mbed-os-5-docs/blob/development/docs/tools/testing/testing_icetea.md" }, { "type": "markdown", diff --git a/docs/reference/contributing/connectivity/MeshInterface.md b/docs/reference/contributing/connectivity/MeshInterface.md index c3572fc194..2b37d13338 100644 --- a/docs/reference/contributing/connectivity/MeshInterface.md +++ b/docs/reference/contributing/connectivity/MeshInterface.md @@ -66,7 +66,9 @@ The following steps describe how you can create a new RF driver: 1. Check with an RF sniffer tool that you can see RF packets transmitted when you start your device. The 6LoWPAN bootstrap should start with IEEE 802.15.4 Beacon Request packets. -1. Verify the functionality of your implementation using the [Nanostack RF driver test application](https://github.com/ARMmbed/nanostack-rf-driver-tester). (This is currently only available to Mbed OS Partners.) +1. Verify the functionality of your implementation by running the Nanostack RF driver testcase set in the Mbed OS repository: + + `mbed test --clean --icetea -t -m --test-config MAC_TESTER -n address_read_and_write,send_data,send_data_indirect,send_large_payloads,create_and_join_PAN,ED_scan` ### Worker thread for Mbed OS diff --git a/docs/tools/offline/cli-test-debug.md b/docs/tools/offline/cli-test-debug.md index 045cc510ff..76591be02a 100644 --- a/docs/tools/offline/cli-test-debug.md +++ b/docs/tools/offline/cli-test-debug.md @@ -2,23 +2,31 @@ Use the `mbed test` command to compile and run tests. +There are two testing frameworks: Greentea and Icetea. Greentea provides tests designed for driver porting and target verification. Icetea provides and manages tests for multiple devices at the same time. For example, you can test the network setup for a server and multiple clients, simultaneously controlling them from the test environment. + The arguments to `test` are: -* `-m ` to select a target for the compilation. If `detect` or `auto` parameter is passed, then Mbed CLI will attempt to detect the connected target and compile against it. -* `-t ` to select a toolchain (of those defined in `mbed_settings.py`, see above), where `toolchain` can be either `ARM` (Arm Compiler 5), `GCC_ARM` (GNU Arm Embedded), or `IAR` (IAR Embedded Workbench for Arm). -* `--compile-list` to list all the tests that can be built. -* `--run-list` to list all the tests that can be run (they must be built first). -* `--compile` to only compile the tests. -* `--run` to only run the tests. -* `-n ` to limit the tests built or run to a comma separated list (ex. test1,test2,test3). -* `--source ` to select the source directory. Default is `.` (the current directory). You can specify multiple source locations, even outside the program tree. -* `--build ` to select the build directory. Default: `BUILD/` inside your program. -* `--profile ` to select a path to a build profile configuration file. Example: `mbed-os/tools/profiles/debug.json`. -* `-c or --clean` to clean the build directory before compiling. -* `--test-spec ` to set the path for the test spec file used when building and running tests (the default path is the build directory). -* `-v` or `--verbose` for verbose diagnostic output. -* `-vv` or `--very_verbose` for very verbose diagnostic output. - -Invoke `mbed test`: + +- `-m `: to select a target for the compilation. If the `detect` or `auto` parameter is passed, then Mbed CLI will attempt to detect the connected target and compile against it. +- `-t `: to select a toolchain from those defined in `mbed_settings.py`, where `toolchain` can either be `ARM` (Arm Compiler 5), `GCC_ARM` (GNU Arm Embedded), or `IAR` (IAR Embedded Workbench for Arm). +- `--compile-list`: to list all the tests that can be built. +- `--run-list`: to list all the tests that can be run, after they have been built. +- `--compile`: to only compile the tests. +- `--run`: to only run the tests. +- `-n `: to limit the tests built or run to a comma separated list, for example, `test1, test2, test3`. +- `--source `: to select the source directory. The default is `.` for the the current directory. You can specify multiple source locations, even outside the program tree. +- `--build `: to select the build directory. The default is `BUILD/` inside your program. +- `--profile `: to select a path to a build profile configuration file, for example, `mbed-os/tools/profiles/debug.json`. +- `-c or --clean`: to clean the build directory before compiling. +- `--test-spec `: to set the path for the test specification file used when building and running tests. The default path is the build directory. +- `--build-data `: dumps build_data to this file. +- `--app-config `: the path of an app configuration file. The default is to look for `mbed_app.json`. +- `--test-config `: the path or Mbed OS keyword of a test configuration file, for example, `ethernet`, `odin_wifi` or `path/to/config.json`. +- `--greentea`: to run Greentea tests. As a default, it only runs Greentea tests. +- `--icetea`: to run Icetea tests. If used without the `--greentea` flag, then it only runs Icetea tests. +- `-v` or `--verbose`: for verbose diagnostic output. +- `-vv` or `--very_verbose`: for very verbose diagnostic output. + +To invoke the `mbed test`: ``` $ mbed test -m K64F -t GCC_ARM @@ -79,6 +87,33 @@ Test Case: Path: .\TESTS\functional\test3 ``` +For Icetea: + +``` +$ mbed test -m K64F -t GCC_ARM --icetea --compile-list +Available Icetea tests for build 'K64F-GCC_ARM', location 'TEST_APPS' +Test Case: + Name: test_cmdline + Path: ./TEST_APPS/testcases/example/test_cmdline.py + Test applications: ./TEST_APPS/device/exampleapp +Test Case: + Name: UDPSOCKET_BIND_PORT + Path: ./TEST_APPS/testcases/netsocket/SOCKET_BIND_PORT.py + Test applications: ./TEST_APPS/device/socket_app +Test Case: + Name: TCPSOCKET_BIND_PORT + Path: ./TEST_APPS/testcases/netsocket/SOCKET_BIND_PORT.py + Test applications: ./TEST_APPS/device/socket_app +Test Case: + Name: TCPSERVER_ACCEPT + Path: ./TEST_APPS/testcases/netsocket/TCPSERVER_ACCEPT.py + Test applications: ./TEST_APPS/device/socket_app +Test Case: + Name: TCPSOCKET_ECHOTEST_BURST_SHORT + Path: ./TEST_APPS/testcases/netsocket/TCPSOCKET_ECHOTEST_BURST_SHORT.py + Test applications: ./TEST_APPS/device/socket_app +``` + You can find the tests that are available for **running** by using the `--run-list` option: ``` @@ -91,21 +126,38 @@ mbedgt: available tests for built 'K64F-ARM', location '.\build\tests\K64F\ARM' test 'TESTS-functional-test3' ``` +For Icetea: + +``` +$ mbed test -m K64F -t GCC_ARM --icetea --run-list +Available Icetea tests for build 'K64F-GCC_ARM', location 'TEST_APPS' + test 'UDPSOCKET_BIND_PORT' + test 'TCPSOCKET_BIND_PORT' + test 'TCPSERVER_ACCEPT' + test 'TCPSOCKET_ECHOTEST_BURST_SHORT' +``` + ### Compiling and running tests -You can specify to only **build** the tests by using the `--compile` option: +You can specify that the tests only **build** by using the `--compile` option: ``` $ mbed test -m K64F -t GCC_ARM --compile ``` -You can specify to only **run** the tests by using the `--run` option: +For Icetea, only the test applications are built: + +``` +$ mbed test -m K64F -t GCC_ARM --compile --icetea +``` + +You can specify that the tests only **run** by using the `--run` option: ``` $ mbed test -m K64F -t GCC_ARM --run ``` -If you don't specify any of these, `mbed test` will first compile all available tests and then run them. +If you don't specify any of these, `mbed test` first compiles all available tests and then runs them. ### Limiting the test scope @@ -115,7 +167,7 @@ You can limit the scope of the tests built and run by using the `-n` option. Thi $ mbed test -m K64F -t GCC_ARM -n TESTS-functional-test1,TESTS-functional-test2 ``` -You can use the wildcard character `*` to run a group of tests that share a common prefix without specifying each test individually. For instance, if you only want to run the three tests `TESTS-functional-test1`, `TESTS-functional-test2` and `TESTS-functional-test3`, but you have other tests in your project, you can run: +You can use the wildcard character `*` to run a group of tests that share a common prefix without specifying each test individually. For instance, if you only want to run the three tests, `TESTS-functional-test1`, `TESTS-functional-test2` and `TESTS-functional-test3`, but you have other tests in your project, you can run: ``` $ mbed test -m NUCLEO_F429ZI -t GCC_ARM -n TESTS-functional* @@ -154,16 +206,17 @@ mbed-os-program | .... ``` -As shown above, tests exist inside `TESTS\testgroup\testcase\` directories. Please note that `TESTS` is a special upper case directory that is excluded from module sources while compiling. +As shown above, tests exist inside `TESTS\testgroup\testcase\` directories. Please note that `TESTS` is a special upper-case directory that is excluded from module sources while compiling. -**Note:** `mbed test` does not work in applications that contain a `main` function that is outside of a `TESTS` directory. +**Note:** `mbed test` does not work in applications that contain a `main` function that is outside of a `TESTS` directory. ### Troubleshooting -#### Unable to import Mercurial (mbed.org) programs or libraries. -1. Check whether you have Mercurial installed in your system path by running `hg` in command prompt. If you're receiving "command not found" or a similar message, then you need to install Mercurial, and add it to your system path. +#### Import Mercurial (mbed.org) programs or libraries + +1. Check whether you have Mercurial installed in your system path by running `hg` in the command prompt. If you are receiving "command not found" or a similar message, then you need to install Mercurial and add it to your system path. +1. Try to clone a Mercurial repository directly. For example, `hg clone https://developer.mbed.org/teams/mbed/code/mbed_blinky/`. If you receive an error similar to `abort: error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.:590)`, then your system certificates are out of date. You need to update your system certificates and possibly add the host certificate fingerprint of `mbed.com` and `mbed.org`. You can read more about Mercurial's [certificate management](https://www.mercurial-scm.org/wiki/CACertificates). -2. Try to clone a Mercurial repository directly. For example, `hg clone https://developer.mbed.org/teams/mbed/code/mbed_blinky/`. If you receive an error similar to `abort: error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.:590)`, then your system certificates are out of date. You need to update your system certificates and possibly add the host certificate fingerprint of `mbed.com` and `mbed.org`. You can read more about Mercurial's [certificate management](https://www.mercurial-scm.org/wiki/CACertificates). +#### Various issues when running Mbed CLI in the Cygwin environment -#### Various issues when running Mbed CLI in Cygwin environment -Currently Mbed CLI is not compatible with Cygwin environment and [cannot be executed inside it](https://github.com/ARMmbed/mbed-cli/issues/299). +Mbed CLI is not currently compatible with the Cygwin environment and [cannot be executed inside it](https://github.com/ARMmbed/mbed-cli/issues/299). diff --git a/docs/tools/testing/icetea.md b/docs/tools/testing/icetea.md new file mode 100644 index 0000000000..d8b677c92e --- /dev/null +++ b/docs/tools/testing/icetea.md @@ -0,0 +1,166 @@ +

Icetea test framework

+ +Icetea is an automated testing framework for Mbed development. It automates the process of flashing Mbed boards, running tests and accumulating test results into reports. Developers use it for local development, as well as automation in a Continuous Integration environment. + +When testing Mbed OS, Icetea allows you to execute commands remotely by using the command-line interface in a device under test (DUT). The interface between the test framework and a DUT might be, for example, UART or stdio. + +More detailed documentation on the tool is available [in rst format](https://github.com/ARMmbed/icetea/tree/master/doc-source) and [in markdown format](https://github.com/ARMmbed/icetea/tree/master/doc). + +### Prerequisites + +Icetea supports Linux (Ubuntu preferred), Windows and OS X. Our main target is Linux. + +You need `pip` to install Icetea. + +Icetea supports both Python 2.7 and 3.5, or later. Some OS specific prerequisites are listed below: + +- Linux. + - python-dev and python-lxml: + `sudo apt-get install python-dev python-lxml` + - To run test cases with hardware in Linux, without sudo rights: + + ``` + sudo usermod -a -G dialout username + Log out & log in back to Linux + ``` + + This command adds the user `username` to the `dialout` group and grants the required permissions to the USB ports. + +- OS X. + - [XCode developer tools](http://osxdaily.com/2014/02/12/install-command-line-tools-mac-os-x/). + - [MacPorts](https://www.macports.org/install.php). + - [lxml](http://lxml.de/installation.html#installation): + `STATIC_DEPS=true sudo pip install lxml` + +- Windows + - python-lxml installation on Windows usually requires build tools. You can, however, install it from prebuilt binaries. + - Search the internet for a binary for your system. + - Navigate to the directory where you downloaded the binary, and install it with `pip install ` + - You can also follow [these instructions](http://lxml.de/installation.html#installation) instead. + +#### Optional + +If you wish to decorate your console log with colors, install the `coloredlogs` module by using pip: `pip install coloredlogs`. There have been issues with `coloredlogs` installation on Windows. There are no alternative solutions for this at the moment. + +### Installation + +`> pip install icetea` + +### Use + +To display the help page: + +`icetea --help` + +To list all local test cases from the examples subfolder: + +`icetea --list --tcdir examples` + +To print the Icetea version: + +`icetea --version` + +#### Typical use + +All of the commands described below might also require other commands, depending on the test case. + +**Running test cases by using the `tc` argument** + +`> icetea --tc --tcdir ` + +To run all existing test cases from the `examples` folder: + +`> icetea --tc all --tcdir examples` + +**Running an example test case with hardware** + +This example requires a compatible board connected to the computer and an application binary for the board. The test case below is available in [the Icetea GitHub repository](https://github.com/ARMmbed/icetea/blob/master/examples/test_cmdline.py). + +`> icetea --tc test_cmdline --tcdir examples --type hardware --bin ` + +**Using metadata filters** + +To run all test cases with test-type regression in the metadata: + +`> icetea --testtype regression --tcdir ` + +The following metadata filters are available: + +- test type (`--testtype`). +- test subtype (`--subtype`). +- feature (`--feature`). +- test case name (`--tc`). +- tested component (`--component`). +- test case folder (`--group`). + +**Running a premade suite** + +Icetea supports a suite file that describes a number of test cases in `json` format: + +`> icetea --suite --tcdir --suitedir ` + +**Enabling debug level logging** + +Use `-v` or `-vv` arguments to control logging levels. `-v` increases the framework's logging level to debug (default is info), the level of logging in certain plugins and external components to info (default is warning). `--vv` increases the level of logging on all Icetea loggers to debug. + +#### Creating a test case + +Icetea test cases are implemented as Python classes that inherit the bench object available in the `icetea_lib.bench` module. The test case needs to have an initialization function that defines the metadata and a case function that implements the test sequence. There are two optional functions: setup and teardown. + +An example test case is shown below: + +``` +""" +Copyright 2017 ARM Limited +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +""" + +from icetea_lib.bench import Bench + + +class Testcase(Bench): + def __init__(self): + Bench.__init__(self, + name="example_test", + title="Example test", + status="development", + purpose="Show example of a test", + component=["examples"], + type="smoke", + requirements={ + "duts": { + '*': { + "count": 1, + "type": "hardware" + } + } + } + ) + + def setup(self): + # nothing for now + pass + + + def case(self): + self.command(1, "echo hello world", timeout=5) + self.command("help") + + def teardown(self): + # nothing for now + pass +``` + +### License + +For licensing details, please see the [license agreement](https://github.com/ARMmbed/icetea/blob/master/LICENSE). diff --git a/docs/tools/testing/testing.md b/docs/tools/testing/testing_greentea.md similarity index 54% rename from docs/tools/testing/testing.md rename to docs/tools/testing/testing_greentea.md index 9ef6eb163b..f2ac38114c 100644 --- a/docs/tools/testing/testing.md +++ b/docs/tools/testing/testing_greentea.md @@ -1,4 +1,4 @@ -## Testing applications +## Greentea testing applications The way tests are run and compiled in Arm Mbed OS 5 is substantially different from previous versions of Mbed. @@ -6,9 +6,9 @@ The way tests are run and compiled in Arm Mbed OS 5 is substantially different f #### Test code structure -Tests can exist throughout Mbed OS and your project's code. They are located under a special directory called `TESTS` (case is important!). +You can run tests throughout Mbed OS and for your project's code. They are located under a special directory called `TESTS`. -Placing code under this directory means it is ignored when building applications and libraries. This code is only ever used when building tests. This is important because all tests require a `main()` function, and building it with your application would cause multiple `main()` functions to be defined. +The fact that the code is located under this directory means that it is ignored when building applications and libraries. It is only used when building tests. This is important because all tests require a `main()` function, and building them with your application would cause multiple `main()` functions to be defined. In addition to being placed under a `TESTS` directory, test sources must exist under two other directories: a test group directory and a test case directory. The following is an example of this structure: @@ -16,15 +16,15 @@ In addition to being placed under a `TESTS` directory, test sources must exist u myproject/TESTS/test_group/test_case_1 ``` -In this example, `myproject` is the project root, and all the source files under the `test_case_1` directory are included in the test. The test build also includes any other source files from the OS, libraries and project that apply to the target's configuration. +In this example, `myproject` is the project root, and all the source files under the `test_case_1` directory are included in the test. The test build also includes any other source files from the OS, libraries and projects that apply to the target's configuration. -**Note:** You can name both the test group and test case directory anything you like. However, you **must** name the `TESTS` directory `TESTS` for the tools to detect the test cases correctly. +**Note:** You can name both the test group and the test case directory anything you like. However, you must name the `TESTS` directory `TESTS` for the tools to detect the test cases correctly. ##### Test discovery Because test cases can exist throughout a project, the tools must find them in the project's file structure before building them. -Test discovery also obeys the same rules that are used when building your project. This means that tests that are placed under a directory with a prefix such as `TARGET_`, `TOOLCHAIN_` or `FEATURE_` are only discovered, built and run if your current configuration matches this prefix. +Test discovery also obeys the same rules that are used when building your project. This means that tests that are placed under a directory with a prefix, such as `TARGET_`, `TOOLCHAIN_` or `FEATURE_`, are only discovered, built and run if your current configuration matches this prefix. For example, if you place a test under the directory `FEATURE_BLE` with the following path: @@ -40,44 +40,44 @@ Tests can also be completely ignored by using the `.mbedignore` file described i ##### Test names -A test case is named from its position in your project's file structure. For instance, in the above example, a test case with the path `myproject/TESTS/test_group/test_case_1` would be named `tests-test_group-test_case_1`. The name is created by joining the directories that make up the path to the test case with a "dash" (`-`) character. This is a unique name to identify the test case. You will see this name throughout the build and testing process. +A test case is named by its position in your project's file structure. For instance, in the above example, a test case with the path `myproject/TESTS/test_group/test_case_1` would be named `tests-test_group-test_case_1`. The name is created by joining the directories that make up the path to the test case with a dash `-` character. This is a unique name to identify the test case. You will see this name throughout the build and testing process. #### Building tests You can build tests through Arm Mbed CLI. For information on using Mbed CLI, please see the [CLI documentation](arm-mbed-cli.html). -When you build tests for a target and a toolchain, the script first discovers the available tests and then builds them in parallel. You can also create a "test specification" file, which our testing tools can use to run automated hardware tests. For more information on the test specification file, please see the [Greentea documentation here](https://github.com/ARMmbed/greentea#test-specification-json-formatted-input). +When you build tests for a target and a toolchain, the script first discovers the available tests and then builds them in parallel. You can also create a **test specification** file, which our testing tools can use to run automated hardware tests. For more information on the test specification file, please see the [Greentea documentation](https://github.com/ARMmbed/greentea#test-specification-json-formatted-input). ##### Building process -The `test.py` script (not to be confused with `tests.py`) located under the `tools` directory handles the process for building tests. This handles the discovery and building of all test cases for a target and toolchain. +The `test.py` script (not to be confused with `tests.py`), located under the `tools` directory, handles the process for building tests. It handles the discovery and building of all test cases for a target and toolchain. The full build process is: -1. Build the nontest code (all code not under a `TESTS` folder), but do not link it. The resulting object files are placed in the build directory. +1. Build the non-test code (all code not under a `TESTS` folder), but do not link it. The resulting object files are placed in the build directory. 1. Find all tests that match the given target and toolchain. -1. For each discovered test, build all of its source files and link it with the nontest code that was built in step 1. -1. If specified, create a test specification file and place it in the given directory for use by testing tools. This is placed in the build directory by default when using Mbed CLI. +1. For each discovered test, build all of its source files and link it with the non-test code that was built in step 1. +1. If specified, create a test specification file and place it in the given directory for use by the testing tools. This is placed in the build directory by default when using Mbed CLI. -##### App config +##### Application configuration When building an Mbed application, the presence of an `mbed_app.json` file allows you to set or override different configuration settings from libraries and targets. However, because the tests share a common build, this can cause issues when tests have different configurations that affect the OS. -The build system looks for an `mbed_app.json` file in your shared project files (any directory not inside of a `TESTS` folder). If the system finds it, this configuration file is used for both the nontest code as well as each test case inside your project's source tree. If there is more than one `mbed_app.json` file in the source tree, the config system will error. +The build system looks for an `mbed_app.json` file in your shared project files (any directory not inside of a `TESTS` folder). If the system finds it, then this configuration file is used for both the non-test code and each test case inside your project's source tree. If there is more than one `mbed_app.json` file in the source tree, then the configuration system will error. -If you need to test with multiple configurations, you can use the `--app-config` option. This overrides the search for an `mbed_app.json` file and uses the config file you specify for the build. +If you need to test with multiple configurations, then you can use the `--app-config` option. This overrides the search for an `mbed_app.json` file and uses the configuration file that you specify for the build. #### Running tests You can run automated tests through Mbed CLI. -The testing process requires tests to be built and that a test specification JSON file exist that describes these available tests. See the [test specification format](https://github.com/ARMmbed/greentea#test-specification-json-formatted-input). +The testing process requires that tests be built and that a test specification JSON file that describes these available tests exists. See the [test specification format](https://github.com/ARMmbed/greentea#test-specification-json-formatted-input). The Greentea tool handles the actual testing process. To read more about this tool, please visit its [GitHub repository](https://github.com/ARMmbed/greentea). #### Writing tests -You can write tests for your own project, or add more tests to Mbed OS. You can write tests using the [Greentea client](https://github.com/ARMmbed/mbed-os/tree/master/features/frameworks/greentea-client), [UNITY](https://github.com/ARMmbed/mbed-os/tree/master/features/frameworks/unity) and [utest](https://github.com/ARMmbed/mbed-os/tree/master/features/frameworks/utest) frameworks, located in `/features/frameworks`. Below is an example test that uses all of these frameworks: +You can write tests for your own project or add more tests to Mbed OS. You can write tests by using the [Greentea client](https://github.com/ARMmbed/mbed-os/tree/master/features/frameworks/greentea-client), and the [UNITY](https://github.com/ARMmbed/mbed-os/tree/master/features/frameworks/unity) and [utest](https://github.com/ARMmbed/mbed-os/tree/master/features/frameworks/utest) frameworks, which are located in `/features/frameworks`. Below is an example test that uses all of these frameworks: ```c++ #include "mbed.h" @@ -118,7 +118,7 @@ int main() { } ``` -This test first runs a case that succeeds, then a case that fails. This is a good template to use when creating tests. For more complex testing examples, please see the documentation for [utest](https://github.com/ARMmbed/mbed-os/tree/master/features/frameworks/utest). +This test first runs a case that succeeds, then a case that fails. This is a good template to use for creating tests. For more complex testing examples, please see the documentation for [utest](https://github.com/ARMmbed/mbed-os/tree/master/features/frameworks/utest). ### Debugging tests @@ -128,7 +128,7 @@ Debugging tests is a crucial part of the development and porting process. This s Currently, the easiest way to export a test is to copy the test's source code from its test directory to your project's root. This way, the tools treat it like a normal application. -You can find the path to the test you wish to export by running the following command: +You can find the path to the test that you want to export by running the following command: ``` mbed test --compile-list -n @@ -144,11 +144,11 @@ You can find your exported project in the root project directory. #### Running a test while debugging -Assuming your test was exported correctly to your IDE, build the project, and load it onto your target via your debugger. +Assuming your test was exported correctly to your IDE, build the project and load it onto your target by using your debugger. -Bring the target out of reset, and run the program. Your target waits for the test tools to send a synchronizing character string over the serial port. Do not run the `mbed test` commands because that will attempt to flash the device, which you've already done with your IDE. +Bring the target out of reset and run the program. Your target waits for the test tools to send a synchronizing character string over the serial port. Do not run the `mbed test` commands because that will attempt to flash the device, which you've already done with your IDE. -Instead, you can use the underlying test tools to drive the test. [`htrun`](https://github.com/ARMmbed/htrun) is the tool you need to use in this case. Install the requirements for Mbed OS also installs `htrun`. You can also intall `htrun` by running `pip install mbed-host-tests`. +Instead, you can use the underlying test tools to drive the test. [`htrun`](https://github.com/ARMmbed/htrun) is the tool you need to use in this scenario. Installing the requirements for Mbed OS also installs `htrun`. You can also install `htrun` by running `pip install mbed-host-tests`. First, find your target's serial port by running the following command: @@ -168,7 +168,7 @@ Run the following command when your device is running the test in your debugger: mbedhtrun --skip-flashing --skip-reset -p :9600 ``` -Replace `` with the serial port you found by running `mbed detect` above. So for the example above, the command is: +Replace `` with the serial port that you found by running `mbed detect` above. So, for the example above, the command is: ``` mbedhtrun --skip-flashing --skip-reset -p COM270:9600 @@ -180,4 +180,4 @@ For an explanation of the arguments used in this command, please run `mbedhtrun ### Known issues -There cannot be a `main()` function outside of a `TESTS` directory when building and running tests. This is because this function will be included in the nontest code build as described in the [Building process](#building-process) section. When the test code is compiled and linked with the nontest code build, a linker error will occur due to their being multiple `main()` functions defined. For this reason, please either rename your main application file if you need to build and run tests or use a different project. Note that this does not affect building projects or applications, just building and running tests. +There cannot be a `main()` function outside of a `TESTS` directory when building and running tests. This is because this function will be included in the non-test code build, as described in the [Building process](#building-process) section. When the test code is compiled and linked with the non-test code build, a linker error will occur, due to their being multiple `main()` functions defined. This is why you should either rename your main application file, if you need to build and run tests, or use a different project. Note that this does not affect building projects or applications, only building and running tests. diff --git a/docs/tools/testing/testing_icetea.md b/docs/tools/testing/testing_icetea.md new file mode 100644 index 0000000000..e775c8df45 --- /dev/null +++ b/docs/tools/testing/testing_icetea.md @@ -0,0 +1,95 @@ +## Icetea testing applications + +### Using tests + +#### Test code structure + +Icetea tests are composed of two parts. One is the C++-application running on the Device Under Test (DUT). The other part is the Python script running on your computer. The provided C++ applications running on the DUT are actually command-line interface (CLI) applications that you can run on their own and give commands to the DUT over a serial console. The test cases are provided as Python scripts, and each Python script is tied to a specific C++ application. + +You can find the Icetea test applications and test cases under the `TEST_APPS` folder in `mbed-os`. `TEST_APPS/device` lists all of the C++ test applications. `TEST_APPS/icetea_plugins` contains plugins for the test cases. These are mainly parsers that provide a functionality for the test scripts to parse the DUT provided output from the serial console. `TEST_APPS/testcases` lists all the tests and any possible utility scripts the tests need. + +##### Test discovery + +When using Icetea under the `mbed-os repository`, the `TEST_APPS folder` stores the CLI applications and the test cases. The `TEST_APPS` folder is located in the root of the `mbed-os` repository. If you are using Icetea somewhere else, then you need to provide Icetea the information where the tests reside. You can do this with the `--tcdir ` option. + +##### Test names + +Test names are defined in the Python scripts themselves. For example, the `SOCKET_BIND_PORT.py` contains two test cases; `TCPSOCKET_BIND_PORT` and `UDPSOCKET_BIND_PORT` are both defined in the Python file. + +#### Building tests + +You can build tests through Arm Mbed CLI. For information on using Mbed CLI, please see the [CLI documentation](/docs/development/tools/arm-mbed-cli.html). + +When you build tests for a target and a toolchain, the script first discovers the available tests, then the C++ applications linked to the tests and then builds the C++ applications in parallel. + +##### Building process + +The `test.py` script (not to be confused with `tests.py`), located under the `tools` directory, handles the process for building tests. It handles the discovery and building of all test cases for a target and toolchain. + +The full build process is: + +1. Build the non-test code (all code not under a `TESTS` folder), but do not link it. The resulting object files are placed in the build directory. +1. Find all tests that match the given target and toolchain. +1. Find all the C++ applications that are defined in the tests. +1. For each discovered C++ application, build all of its source files and link it with the non-test code that was built in step 1. +1. If specified, create a test specification file and place it in the given directory for use by the testing tools. This is placed in the build directory by default when using Mbed CLI. + +#### Running tests + +You can run automated tests through Mbed CLI with the `--icetea` option. You can find more information about this in the [Mbed CLI test documentation](/docs/tools/offline/cli-test-debug.md). + +The testing process requires that tests be built and that a test specification JSON file that describes these available tests exists. See the [test specification format](https://github.com/ARMmbed/greentea#test-specification-json-formatted-input). + +The test specification JSON is similar for the Greentea and Icetea tests. + +The Icetea tool handles the actual testing process. To read more about this tool, please visit its [GitHub repository](https://github.com/ARMmbed/icetea). + +#### Writing tests + +##### Writing test cases + +To write your own Python test scripts, please read the further information in [Icetea Test Case API](https://github.com/ARMmbed/icetea/blob/master/doc/tc_api.md). + +##### Writing CLI applications + +To be able to run the commands given to the DUT, you need to provide a CLI application that runs on the DUT. There is a library that you can use to create your own CLI application, called [mbed-client-cli](https://github.com/ARMmbed/mbed-client-cli). You can also find example Mbed CLI applications in the `mbed-os` repository [test applications](https://github.com/ARMmbed/mbed-os/tree/master/TEST_APPS/device). + +### Debugging tests + +Debugging tests is a crucial part of the development and porting process. This section covers exporting the test, then driving the test with the test tools while the target is attached to a debugger. + +#### Exporting tests + +The most straightforward way to export a test is to copy the test application's source code from its test directory to your project's root. This way, the tools treat it like a normal application. + +You can find the path to the test application that you want to export by running the following command: + +``` +mbed test --compile-list -n --icetea +``` + +Once you've copied all of the test's source files to your project root, export your project: + +``` +mbed export -i +``` + +You can find your exported project in the root project directory. + +#### Running a test while debugging + +After you export your test to your IDE, build the project and load it onto your target by using your debugger. + +Bring the target out of reset, and run the program. Your target waits for a command over the serial line. + +You can now use Icetea commands to run the test: + +``` +icetea -m -t --tc +``` + +This detects your attached target and drives the test. If you need to rerun the test, then reset the device with your debugger, run the program and run the same command. + +For an explanation of the arguments used in this command, please run `icetea --help`. + +Note that instead of running the test, you can also open a serial connection to the target and give the Mbed CLI commands manually. diff --git a/docs/tools/testing/testing_intro.md b/docs/tools/testing/testing_intro.md index b5d9b6b7a1..b64a3a8fe5 100644 --- a/docs/tools/testing/testing_intro.md +++ b/docs/tools/testing/testing_intro.md @@ -1,8 +1,8 @@

Testing

-Testing is a critical step in the development process. The Arm Mbed ecosystem offers several tools to help you test your code. +Testing is a critical step in the development process. The Arm Mbed ecosystem offers several tools to help you test your code. -Greentea, `htrun` and `mbed-ls` are testing tools written in Python. Greentea tests serve as functional unit tests in C++, as well as integration tests for complex use cases that execute on microcontrollers. The Mbed CLI tool has a verb `test` that drives these tools to form a testing system. These comprise our automated testing framework for Mbed OS development. +Greentea, Icetea, `htrun` and `mbed-ls` are testing tools written in Python. Greentea tests serve as functional unit tests in C++, as well as integration tests for complex use cases that execute on microcontrollers. The Icetea test tool adds more support for interoperability testing by making it easier to handle multiple devices and external service during testing. Arm Mbed CLI has a verb `test` that drives these tools to form a testing system. These comprise our automated testing framework for Mbed OS development. The testing system automates the process of flashing Mbed boards, driving the tests and accumulating test results into test reports. Developers and Mbed Partners use this system for local development, as well as for automation in a Continuous Integration environment.