You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: TestingGuide.md
+45-43
Original file line number
Diff line number
Diff line change
@@ -49,7 +49,7 @@ This guide contains two main parts:
49
49
50
50
A ‘test file’ is a file containing ‘test cases’.
51
51
52
-
A ‘test case’ is a class inherited from the *ImpTestCase* class defined by the [*impUnit*](https://github.com/electricimp/impUnit) framework. There can be several test cases in a test file.
52
+
A ‘test case’ is a class based on the *ImpTestCase* class defined by the [*impUnit*](https://github.com/electricimp/impUnit) framework. There can be several test cases in a test file.
53
53
54
54
A ‘test method’ (or simply called a ‘test’) is one of a test case’s methods. It should be prefixed by *test*, eg. *testEverythingOk()*. There can be several test methods (tests) in a test case.
55
55
@@ -61,21 +61,21 @@ There must be only one [test configuration file](./CommandsManual.md#test-config
61
61
62
62
The ‘test home’ is the directory where the [test configuration file](./CommandsManual.md#test-configuration-files) is located. All of the files located in the test home and all of its sub-directories are considered as test files belonging to the corresponding test project if their names match the patterns specified in the test configuration.
63
63
64
-
**Note:**the test project entity has no any relation to the development Project entity described in the [*impt* Development Guide](./DevelopmentGuide.md). A [Project file](./CommandsManual.md#project-files) and a [test configuration file](./CommandsManual.md#test-configuration-files) may coexist in the same directory.
64
+
**Note**The test project entity has no any relation to the development Project entity described in the [*impt* Development Guide](./DevelopmentGuide.md). A [Project file](./CommandsManual.md#project-files) and a [test configuration file](./CommandsManual.md#test-configuration-files) may coexist in the same directory.
65
65
66
66
A ‘test session’ is a run of a set of tests from one test file on one device. That may include all of the tests from all of the test cases from the test file, or a subset of all the tests in the test file. Running the same set of tests on another device is another test session.
67
67
68
68
A test session is considered failed if at least one test fails.
69
69
70
70
### Test Commands ###
71
71
72
-
The *impt*tool has a dedicated set of commands which operate with test projects, called [Test Commands](./CommandsManual.md#test-commands). For a particular test project, the commands should be called from the test home.
72
+
*impt* has a dedicated set of commands which operate with test projects, called [test commands](./CommandsManual.md#test-commands). For a particular test project, the commands should be called from the test home.
73
73
74
74
Other *impt* commands may also be needed during testing, such as commands to assign devices to Device Group.
75
75
76
76
## Writing Tests ##
77
77
78
-
### Main Rules and Steps ###
78
+
### Main Rules And Steps ###
79
79
80
80
You need to perform the following steps to write your tests:
81
81
@@ -103,8 +103,8 @@ You need to perform the following steps to write your tests:
103
103
- Every test method name should start with *test*. Test methods may have identical names if they are in different test cases. There are no other rules for test method naming, but bear in mind that [running selective tests](#running-selective-tests) can have an impact on test method naming.
104
104
- A test case may have several test methods (tests).
105
105
- Additionally, any test case may have *setUp()* and *tearDown()* methods:
106
-
- If it exists, *setUp()* is called by the tool before any other methods in the test case. It may be used to perform environment setup before the test case execution.
107
-
- If it exists, *tearDown()* is called by the tool after all other methods in the test case. It may be used to clean the environment after the test case execution.
106
+
- If it exists, *setUp()* is called by the tool before any other methods in the test case. It may be used to perform environment setup before test case execution.
107
+
- If it exists, *tearDown()* is called by the tool after all other methods in the test case. It may be used to clean the environment after test case execution.
108
108
- All test methods other than *setUp()* and *tearDown()* in one test case are chosen for execution by the tool in an arbitrary order, ie. your tests should be independent and not assume any particular order of execution.
109
109
110
110
A test method may be run synchronously (the default) or [asynchronously](#asynchronous-testing).
@@ -131,7 +131,7 @@ class MyTestCase extends ImpTestCase {
131
131
}
132
132
```
133
133
134
-
### Tests for Bi-directional Device-Agent Communication ###
134
+
### Tests For Bi-directional Device-Agent Communication ###
135
135
136
136
It is possible to test an interaction between device and agent by emulating one side of the interaction.
137
137
@@ -146,11 +146,12 @@ A test file is intended to test either device or agent code, so the opposite sid
146
146
147
147
For example, `"Test1.agent.test.nut"` (a test file with test cases for agent code) would be partnered with `"Test1.device.nut"` (the corresponding partner file with emulation of the device side of the interaction).
148
148
149
-
**Note:**it is sufficient that only the test file is selected for the run, ie. it satisfies the test file search pattern defined during [test configuration](#test-configuration). The corresponding partner file will be added to the test session automatically.
149
+
**Note**It is sufficient that only the test file is selected for the run, ie. it satisfies the test file search pattern defined during [test configuration](#test-configuration). The corresponding partner file will be added to the test session automatically.
150
150
151
151
**Example**
152
152
153
153
Test file `"Test1.agent.test.nut"`:
154
+
154
155
```squirrel
155
156
class MyTestCase extends ImpTestCase {
156
157
_myVar = null;
@@ -180,6 +181,7 @@ class MyTestCase extends ImpTestCase {
180
181
```
181
182
182
183
The corresponding partner file `"Test1.device.nut"`:
184
+
183
185
```squirrel
184
186
imp.wakeup(5.0, function() {
185
187
agent.send("data", "Hello from the dDevice");
@@ -284,7 +286,7 @@ The following assertions are available in tests:
284
286
285
287
#### assertTrue() ####
286
288
287
-
`this.assertTrue(condition, [message]);`
289
+
`this.assertTrue(condition[, message])`
288
290
289
291
Asserts that the condition is truthful.
290
292
@@ -300,7 +302,7 @@ this.assertTrue(1 == 2);
300
302
301
303
#### assertEqual() ####
302
304
303
-
`this.assertEqual(expected, actual, [message])`
305
+
`this.assertEqual(expected, actual[, message])`
304
306
305
307
Asserts that two values are equal.
306
308
@@ -316,7 +318,7 @@ this.assertEqual(1, 2);
316
318
317
319
#### assertGreater() ####
318
320
319
-
`this.assertGreater(actual, cmp, [message])`
321
+
`this.assertGreater(actual, cmp[, message])`
320
322
321
323
Asserts that a value is greater than some other value.
322
324
@@ -332,7 +334,7 @@ this.assertGreater(1, 2);
332
334
333
335
#### assertLess() ####
334
336
335
-
`this.assertLess(actual, cmp, [message])`
337
+
`this.assertLess(actual, cmp[, message])`
336
338
337
339
Asserts that a value is less than some other value.
There are three ways to display diagnostic messages in the console from your tests:
426
428
427
429
- Call `this.info(<message>);` from a test method, as many times as you need.
428
-
- For synchronous tests, call `return <return_value>;` from a test method. The returned value will be displayed in the console, if it is not `null` and the test succeeds.
429
-
- For asynchronous tests, call Promise resolution or rejection methods with a value `resolve(<return_value>);` or `reject(<return_value>);`. The returned value will be displayed on the console, if it is not `null`.
430
+
- For synchronous tests, call `return <return_value>;` from a test method. The returned value will be displayed in the console, provided it is not `null` and the test succeeds.
431
+
- For asynchronous tests, call Promise resolution or rejection methods with a value `resolve(<return_value>);` or `reject(<return_value>);`. The returned value will be displayed on the console, provided it is not `null`.
430
432
431
433
Examples of tests output are provided in the [section on running tests](#running-tests).
432
434
@@ -477,11 +479,11 @@ class TestCase1 extends ImpTestCase {
477
479
478
480
### Test Device Group ###
479
481
480
-
To run your tests you need to have one or more devices associated with your account and assigned to the Device Group to which your tests will be deployed. These are the tasks you should perform in order to prepare your devices for testing:
482
+
To run your tests you need to have one or more devices associated with your account and assigned to the Device Group to which your tests will be deployed. These then are the tasks you should perform in order to prepare your devices for testing:
481
483
482
484
1. Prepare a Product. If you already have a Product, note its ID or name. If you do not have a Product, create a new one with [`impt product create`](./CommandsManual.md#product-create):
483
485
484
-
```
486
+
```bash
485
487
> impt product create --name MyTestProduct
486
488
Product "MyTestProduct" is created successfully.
487
489
Product:
@@ -493,7 +495,7 @@ To run your tests you need to have one or more devices associated with your acco
493
495
2. Prepare a Device Group. If you already have a Device Group, note its ID or name. If you do not have a Device Group, create a new one with [`impt dg create`](./CommandsManual.md#device-group-create). You need to specify the Product when creating the Device Group.
494
496
You may use a Device Group of any [type](./CommandsManual.md#device-group-type), but it is recommended that you use a Development Device Group.
Device "myDevice1" is assigned successfully to Device Group "MyTestDG".
511
513
IMPT COMMAND SUCCEEDS
@@ -521,7 +523,7 @@ The configuration settings include:
521
523
522
524
- `--dg`— the Device Group identifier. Your tests will run on all of the devices assigned to that Device Group. You may specify the Device Group by its ID or its name.
523
525
524
-
- `--device-file`, `--agent-file` — The device and agent source code which is deployed along with the tests. Usually, it is the source code of a library or other Squirrel code which you are planning to test.
526
+
- `--device-file`, `--agent-file`— The device and agent source code which is deployed along with the tests. Usually, it is the source code of a library or other Squirrel which you are planning to test.
525
527
526
528
- `--test-file`— The test file name or the pattern which specifies the test file(s) included in your test project. You may repeat this option to specify several file names and/or patterns. The values of the repeated option are combined by logical OR. The default pattern is detailed in the [command’s spec](./CommandsManual.md#test-create).
527
529
@@ -533,7 +535,7 @@ The configuration settings include:
533
535
534
536
**Example**
535
537
536
-
```
538
+
```bash
537
539
> impt test create --dg MyTestDG --agent-file MyLibrary.agent.lib.nut
538
540
Test Configuration is created successfully.
539
541
Test Configuration:
@@ -560,13 +562,13 @@ Test Configuration:
560
562
IMPT COMMAND SUCCEEDS
561
563
```
562
564
563
-
#### Updating the Configuration ####
565
+
#### Updating The Configuration ####
564
566
565
567
You may update the test configuration by calling [`impt test update`](./CommandsManual.md#test-update). The existing [test configuration file](./CommandsManual.md#test-configuration-files) will be updated with the new settings. The new `--test-file` option value(s) completely replace any existing setting.
566
568
567
569
**Example**
568
570
569
-
```
571
+
```bash
570
572
> impt test update --timeout 60 --builder-cache true
571
573
Test Configuration is updated successfully.
572
574
Test Configuration:
@@ -597,7 +599,7 @@ You may also display the current test configuration by calling [`impt test info`
597
599
598
600
**Example**
599
601
600
-
```
602
+
```bash
601
603
> impt test info
602
604
Test Configuration:
603
605
Test files: *.test.nut, tests/**/*.test.nut
@@ -629,17 +631,17 @@ These may be needed if your test files [include code from GitHub](#include-from-
629
631
630
632
For unauthenticated requests, the GitHub API allows you to make [up to 60 requests per hour](https://developer.github.com/v3/#rate-limiting), but this may be not sufficient for intensive testing. To overcome this limit, you can provide GitHub account credentials. There are two ways to do this:
631
633
632
-
- Via environment variables. **We strongly recommend this way for security reasons** — You should specify two environment variables:
634
+
- Via environment variables. **We strongly recommend this way for security reasons**<br />You should specify two environment variables:
633
635
- `GITHUB_USER`— A GitHub account username.
634
636
- `GITHUB_TOKEN`— A GitHub account password or personal access token.
635
637
636
638
- Via a GitHub credentials file.
637
-
- This file may be created or updated with [`impt test github`](./CommandsManual.md#test-github). You specify a GitHub username and password, and they are saved in the specified file. **Important:** the credentials are stored in a plain text.
639
+
- This file may be created or updated with [`impt test github`](./CommandsManual.md#test-github). You specify a GitHub username and password, and they are saved in the specified file. **Important**The credentials are stored in a plain text.
638
640
- You may have several GitHub credential files and they may be located in any place. You specify a concrete GitHub credentials file during [test configuration](#test-configuration). If the specified file exists when you [run the tests](#running-tests), the GitHub credentials are taken from it. If the specified file does not exist, the GitHub credentials are taken from the environment variables, if they are set.
639
641
640
642
**Example**
641
643
642
-
```
644
+
```bash
643
645
> impt test github --github-config github.conf --user github_username
644
646
--pwd github_password
645
647
GitHub credentials Configuration is created successfully.
@@ -652,21 +654,21 @@ To run your configured test project’s tests, call [`impt test run`](./Commands
652
654
653
655
By default, the tool searches forall test files according to the file names and/or patterns specifiedin the [test configuration](#test-configuration) file. The search starts from the test home and includes all sub-directories. The tool looks for all test cases in the files it discovers. All test methods in all located test cases are considered as viable tests for execution. For a particular run, you may select a subset of test files, test cases and test methods by specifying the `--tests` option; see [here](#running-selective-tests) for more details.
654
656
655
-
Every selected test file is a source of a new build (Deployment), so there will be as many different builds as there are selected test files for execution. Test files (builds) run in an arbitrary order.
657
+
Every selected test file is a sourceforbuilding and deploying code, so there will be as many different builds as there are selected test files for execution. Test files (builds) runin an arbitrary order.
656
658
657
659
Every test file (build) runs on all devices currently assigned to the Device Group specified in the [test configuration](#test-configuration) file one by one: no device is run until the previous device has completed testing. Devices are chosen in an arbitrary order. A test file (build) running on one device is called a test session. When the build completes on the last device, the next test file (build) starts running on the same set of devices, again one after the other.
658
660
659
661
You may clear the [*Builder* cache](#builder-cache) before the tests starts by setting the `--clear-cache` option. If the *Builder* cache is enabled in the [test configuration](#test-configuration) file, it will then be re-created during the test run.
660
662
661
663
You may run the tests in [debug mode](#debug-mode) by specifying the `--output debug` option.
662
664
663
-
A test is treated as failed if an error is thrown or a timeout, as defined in the [test configuration](#test-configuration) file, occurs during the test execution. Otherwise the test is treated as passed. If at least one test in a test session fails, the test session is treated as failed. If the [test configuration](#test-configuration) has the `stop-on-fail` setting set to `true`, test execution ends after the first failed test.
665
+
Everytest is treated as failed if an error is thrown or a timeout, as defined in the [test configuration](#test-configuration) file, occurs during the test execution. Otherwise the test is treated as passed. If at least one test in a test session fails, the test session is treated as failed. If the [test configuration](#test-configuration) has the `stop-on-fail` setting set to `true`, test execution ends after the first failed test.
664
666
665
-
When all tests are passed, the [`impt test run`](./CommandsManual.md#test-run) command outputs `IMPT COMMAND SUCCEEDS` phrase and returns zero exit code. Otherwise, it outputs `IMPT COMMAND FAILS` phrase and returns non-zero exit code.
667
+
When all tests are passed, the [`impt test run`](./CommandsManual.md#test-run) command outputs `IMPT COMMAND SUCCEEDS` and returns anexit code of zero. Otherwise, it outputs `IMPT COMMAND FAILS` and returns a non-zero exit code.
666
668
667
-
**Example - testing failed**
669
+
**Example: Testing Failed**
668
670
669
-
```
671
+
```bash
670
672
> impt test run
671
673
[info] Started at 09 Mar 2018 18:58:31 GMT+0300
672
674
[+0.01/0.01s info] Found 1 test file:
@@ -701,9 +703,9 @@ Error: Testing failed
701
703
IMPT COMMAND FAILS
702
704
```
703
705
704
-
**Example - all tests are passed**
706
+
**Example: All Tests Passed**
705
707
706
-
```
708
+
```bash
707
709
> impt test run
708
710
[info] Started at 09 Mar 2018 19:00:25 GMT+0300
709
711
[+0.01/0.01s info] Found 1 test file:
@@ -790,7 +792,7 @@ You may run your tests in debug mode by specifying the `--output debug` option o
790
792
791
793
**Example**
792
794
793
-
```
795
+
```bash
794
796
> impt test run --tests TestFile1:MyTestCase::testMe --output debug
795
797
...
796
798
[info] Started at 22 Jan 2018 22:49:25 GMT+0300
@@ -879,7 +881,7 @@ After testing is complete, you may want to clean the various entities created du
879
881
880
882
**Example**
881
883
882
-
```
884
+
```bash
883
885
> impt test delete --all
884
886
The following entities will be deleted:
885
887
Product:
@@ -929,7 +931,7 @@ Alternatively, you may fully delete the Device Group which you used for the test
929
931
930
932
**Example**
931
933
932
-
```
934
+
```bash
933
935
> impt dg delete --dg MyTestDG --builds --force
934
936
The following entities will be deleted:
935
937
Device Group:
@@ -969,7 +971,7 @@ If you only want to unassign the devices from the testing Device Group, use [`im
969
971
970
972
**Example**
971
973
972
-
```
974
+
```bash
973
975
> impt dg unassign --dg MyTestDG
974
976
The following Devices are unassigned successfully from Device Group "MyTestDG":
975
977
Device:
@@ -981,7 +983,7 @@ Device:
981
983
IMPT COMMAND SUCCEEDS
982
984
```
983
985
984
-
```
986
+
```bash
985
987
> impt device unassign --device myDevice1
986
988
Device "myDevice1" is unassigned successfully.
987
989
IMPT COMMAND SUCCEEDS
@@ -991,7 +993,7 @@ If you want to delete the Product, use [`impt product delete`](./CommandsManual.
0 commit comments