diff --git a/lib/common_test/doc/src/Makefile b/lib/common_test/doc/src/Makefile index cd5f2e2de01e..a5f2f0975e6d 100644 --- a/lib/common_test/doc/src/Makefile +++ b/lib/common_test/doc/src/Makefile @@ -50,7 +50,8 @@ XML_REF3_FILES = ct.xml \ ct_property_test.xml \ ct_netconfc.xml \ ct_hooks.xml \ - ct_testspec.xml + ct_testspec.xml \ + ct_suite.xml XML_REF6_FILES = common_test_app.xml XML_PART_FILES = part.xml @@ -86,7 +87,7 @@ XML_FILES=$(XML_APPLICATION_FILES) $(XML_REF1_FILES) $(XML_REF3_FILES) $(XML_RE TOP_SPECS_FILE = specs.xml -NO_CHUNKS = ct_hooks.xml +NO_CHUNKS = ct_hooks.xml ct_suite.xml # ---------------------------------------------------- diff --git a/lib/common_test/doc/src/common_test_app.xml b/lib/common_test/doc/src/common_test_app.xml index 3fcbda538aa7..07a2a3e2cdc2 100644 --- a/lib/common_test/doc/src/common_test_app.xml +++ b/lib/common_test/doc/src/common_test_app.xml @@ -52,564 +52,7 @@ Step-by-step execution of test cases -

The following section describes the mandatory and optional test suite - functions that Common Test calls during test execution. - For more details, see section - Writing Test Suites - in the User's Guide.

- - - - - - Test Case Callback Functions -

The following functions define the callback interface - for a test suite.

- - - Module:all() -> Tests | {skip,Reason} - Returns the list of all test case groups and test cases - in the module. - - Tests = [TestCase | {testcase,TestCase,TCRepeatProps} | {group,GroupName} | {group,GroupName,Properties} | {group,GroupName,Properties,SubGroups}] - TestCase = atom() - TCRepeatProps = [{repeat,N} | {repeat_until_ok,N} | {repeat_until_fail,N}] - GroupName = atom() - Properties = [parallel | sequence | Shuffle | {GroupRepeatType,N}] | default - SubGroups = [{GroupName,Properties} | {GroupName,Properties,SubGroups}] - Shuffle = shuffle | {shuffle,Seed} - Seed = {integer(),integer(),integer()} - GroupRepeatType = repeat | repeat_until_all_ok | repeat_until_all_fail | repeat_until_any_ok | repeat_until_any_fail - N = integer() | forever - Reason = term() - - - -

MANDATORY

- -

Returns the list of all test cases and test case groups in the - test suite module to be executed. This list also specifies the - order the cases and groups are executed by Common Test. - A test case is represented by an atom, - the name of the test case function, or a testcase tuple - indicating that the test case shall be repeated. A test case group is - represented by a group tuple, where GroupName, - an atom, is the name of the group (defined in - groups/0). - Execution properties for groups can also be specified, both - for a top-level group and for any of its subgroups. - Group execution properties specified here override - properties in the group definition (see - groups/0). - (With value default, the group definition properties - are used).

- -

If {skip,Reason} is returned, all test cases - in the module are skipped and Reason - is printed on the HTML result page.

- -

For details on groups, see section - Test Case - Groups in the User's Guide.

- - - - - - Module:groups() -> GroupDefs - Returns a list of test case group definitions. - - GroupDefs = [Group] - Group = {GroupName,Properties,GroupsAndTestCases} - GroupName = atom() - Properties = [parallel | sequence | Shuffle | {GroupRepeatType,N}] - GroupsAndTestCases = [Group | {group,GroupName} | TestCase | {testcase,TestCase,TCRepeatProps}] - TestCase = atom() - TCRepeatProps = [{repeat,N} | {repeat_until_ok,N} | {repeat_until_fail,N}] - Shuffle = shuffle | {shuffle,Seed} - Seed = {integer(),integer(),integer()} - GroupRepeatType = repeat | repeat_until_all_ok | repeat_until_all_fail | repeat_until_any_ok | repeat_until_any_fail - N = integer() | forever - - - -

OPTIONAL

- -

Defines test case groups. For details, see section - Test Case - Groups in the User's Guide.

- - - - - Module:suite() -> [Info] - Test suite info function (providing default data - for the suite). - - Info = {timetrap,Time} | {require,Required} | {require,Name,Required} | {userdata,UserData} | {silent_connections,Conns} | {stylesheet,CSSFile} | {ct_hooks, CTHs} - Time = TimeVal | TimeFunc - TimeVal = MilliSec | {seconds,integer()} | {minutes,integer()} | {hours,integer()} - TimeFunc = {Mod,Func,Args} | Fun - MilliSec = integer() - Mod = atom() - Func = atom() - Args = list() - Fun = fun() - Required = Key | {Key,SubKeys} | {Key,SubKey} | {Key,SubKey,SubKeys} - Key = atom() - SubKeys = SubKey | [SubKey] - SubKey = atom() - Name = atom() - UserData = term() - Conns = [atom()] - CSSFile = string() - CTHs = [CTHModule | -         {CTHModule, CTHInitArgs} | -         {CTHModule, CTHInitArgs, CTHPriority}] - CTHModule = atom() - CTHInitArgs = term() - - - -

OPTIONAL

- -

The test suite information function. Returns a list of tagged - tuples specifying various properties related to the execution of - this test suite (common for all test cases in the suite).

- -

Tag timetrap sets the maximum time that each - test case is allowed to execute (including - init_per_testcase/2 - and - end_per_testcase/2). - If the timetrap time is exceeded, the test case fails with reason - timetrap_timeout. A TimeFunc function can be used to - set a new timetrap by returning a TimeVal. It can also be - used to trigger a timetrap time-out by, at some point, returning a - value other than a TimeVal. For details, see section - Timetrap Time-Outs - in the User's Guide.

- -

Tag require specifies configuration variables - required by test cases (or configuration functions) - in the suite. If the required configuration variables are not found - in any of the configuration files, all test cases are skipped. - For details about the require functionality, see funtion - ct:require/1,2.

- -

With userdata, the user can - specify any test suite-related information, which can be - read by calling - ct:userdata/2.

- -

Tag ct_hooks specifies the - Common Test Hooks - to be run with this suite.

- -

Other tuples than the ones defined are ignored.

- -

For details about the test suite information function, see section - Test - Suite Information Function in the User's Guide.

- - - - - Module:init_per_suite(Config) -> NewConfig | {skip,Reason} | - {skip_and_save,Reason,SaveConfig} - Test suite initializations. - - Config = NewConfig = SaveConfig = [{Key,Value}] - Key = atom() - Value = term() - Reason = term() - - - -

OPTIONAL; if this function is defined, then end_per_suite/1 - must also be defined.

- -

This configuration function is called as the first function in the - suite. It typically contains initializations that are common for - all test cases in the suite, and that must only be done - once. Parameter Config is the configuration data - that can be modified. Whatever is returned from this - function is specified as Config to all configuration functions - and test cases in the suite.

- -

If {skip,Reason} - is returned, all test cases in the suite are skipped - and Reason is printed in the overview log for the suite.

- -

For information on save_config and skip_and_save, - see section - Saving - Configuration Data in the User's Guide.

- - - - - Module:end_per_suite(Config) -> term() | - {save_config,SaveConfig} - Test suite finalization. - - Config = SaveConfig = [{Key,Value}] - Key = atom() - Value = term() - - - -

OPTIONAL; if this function is defined, then init_per_suite/1 - must also be defined.

- -

This function is called as the last test case in the - suite. It is meant to be used for cleaning up after - init_per_suite/1.

-

For information on save_config, see section - Saving - Configuration Data in the User's Guide.

- - - - - Module:group(GroupName) -> [Info] - Test case group information function (providing default data - for a test case group, that is, its test cases and - subgroups). - - Info = {timetrap,Time} | {require,Required} | {require,Name,Required} | {userdata,UserData} | {silent_connections,Conns} | {stylesheet,CSSFile} | {ct_hooks, CTHs} - Time = TimeVal | TimeFunc - TimeVal = MilliSec | {seconds,integer()} | {minutes,integer()} | {hours,integer()} - TimeFunc = {Mod,Func,Args} | Fun - MilliSec = integer() - Mod = atom() - Func = atom() - Args = list() - Fun = fun() - Required = Key | {Key,SubKeys} | {Key,Subkey} | {Key,Subkey,SubKeys} - Key = atom() - SubKeys = SubKey | [SubKey] - SubKey = atom() - Name = atom() - UserData = term() - Conns = [atom()] - CSSFile = string() - CTHs = [CTHModule | -         {CTHModule, CTHInitArgs} | -         {CTHModule, CTHInitArgs, CTHPriority}] - CTHModule = atom() - CTHInitArgs = term() - - - -

OPTIONAL

- -

The test case group information function. It is supposed to - return a list of tagged tuples that specify various properties - related to the execution of a test case group (that is, its test - cases and subgroups). Properties set by - group/1 override - properties with the same key that have been set previously by - suite/0.

- -

Tag timetrap sets the maximum time that each - test case is allowed to execute (including - init_per_testcase/2 - and - end_per_testcase/2). - If the timetrap time is - exceeded, the test case fails with reason - timetrap_timeout. A TimeFunc function can be used to - set a new timetrap by returning a TimeVal. It can also be - used to trigger a timetrap time-out by, at some point, returning a - value other than a TimeVal. For details, see section - Timetrap - Time-Outs in the User's Guide.

- -

Tag require specifies configuration variables - required by test cases (or configuration functions) - in the suite. If the required configuration variables are not found - in any of the configuration files, all test cases in this group are - skipped. For details about the require functionality, see - function - ct:require/1,2.

- -

With userdata, the user can - specify any test case group related information that can be - read by calling - ct:userdata/2.

- -

Tag ct_hooks specifies the - Common Test Hooks - to be run with this suite.

- -

Other tuples than the ones defined are ignored.

- -

For details about the test case group information function, - see section Group - Information Function in the User's Guide.

- - - - - Module:init_per_group(GroupName, Config) -> NewConfig | - {skip,Reason} - Test case group initializations. - - GroupName = atom() - Config = NewConfig = [{Key,Value}] - Key = atom() - Value = term() - Reason = term() - - - -

OPTIONAL; if this function is defined, then end_per_group/2 - must also be defined.

- -

This configuration function is called before execution of a - test case group. It typically contains initializations that are - common for all test cases and subgroups in the group, and that - must only be performed once. GroupName is the name of the - group, as specified in the group definition (see - groups/0). - Parameter Config is the configuration data that can be - modified. - The return value of this function is given as Config - to all test cases and subgroups in the group.

- -

If {skip,Reason} - is returned, all test cases in the group are skipped and - Reason is printed in the overview log for the group.

- -

For information about test case groups, see section - Test Case - Groups in the User's Guide.

- - - - - Module:end_per_group(GroupName, Config) -> term() | - {return_group_result,Status} - Test case group finalization. - - GroupName = atom() - Config = [{Key,Value}] - Key = atom() - Value = term() - Status = ok | skipped | failed - - - -

OPTIONAL; if this function is defined, then init_per_group/2 - must also be defined.

- -

This function is called after the execution of a test case group - is finished. It is meant to be used for cleaning up after - init_per_group/2. - A status value for a nested subgroup can be returned with - {return_group_result,Status}. The status can be retrieved in - end_per_group/2 - for the group on the level above. The status is also used by - Common Test for deciding if execution of a group is to - proceed if property sequence or repeat_until_* - is set.

- -

For details about test case groups, see section - Test Case - Groups in the User's Guide.

- - - - - Module:init_per_testcase(TestCase, Config) -> NewConfig | {fail,Reason} | {skip,Reason} - Test case initializations. - - TestCase = atom() - Config = NewConfig = [{Key,Value}] - Key = atom() - Value = term() - Reason = term() - - - -

OPTIONAL; if this function is defined, - then - end_per_testcase/2 must also be - defined.

- -

This function is called before each test case. Argument - TestCase is the test case name, and - Config (list of key-value tuples) is the configuration - data that can be modified. The NewConfig list returned - from this function is given as Config to the test case. - If {fail,Reason} is returned, the test case is - marked as failed without being executed.

- -

If {skip,Reason} is returned, the test case is skipped - and Reason is printed in the overview log for the suite.

- - - - - Module:end_per_testcase(TestCase, Config) -> term() | {fail,Reason} | {save_config,SaveConfig} - Test case finalization. - - TestCase = atom() - Config = SaveConfig = [{Key,Value}] - Key = atom() - Value = term() - Reason = term() - - - -

OPTIONAL; if this function is defined, - then - init_per_testcase/2 must also be - defined.

- -

This function is called after each test case, and can be used - to clean up after - init_per_testcase/2 - and the test case. Any return value (besides {fail,Reason} - and {save_config,SaveConfig}) is ignored. By returning - {fail,Reason}, TestCase is marked as faulty (even - though it was successful in the sense that it returned - a value instead of terminating).

- -

For information on save_config, see section - Saving - Configuration Data in the User's Guide.

- - - - - Module:Testcase() -> [Info] - Test case information function. - - Info = {timetrap,Time} | {require,Required} | {require,Name,Required} | {userdata,UserData} | {silent_connections,Conns} - Time = TimeVal | TimeFunc - TimeVal = MilliSec | {seconds,integer()} | {minutes,integer()} | {hours,integer()} - TimeFunc = {Mod,Func,Args} | Fun - MilliSec = integer() - Mod = atom() - Func = atom() - Args = list() - Fun = fun() - Required = Key | {Key,SubKeys} | {Key,Subkey} | {Key,Subkey,SubKeys} - Key = atom() - SubKeys = SubKey | [SubKey] - SubKey = atom() - Name = atom() - UserData = term() - Conns = [atom()] - - - - -

OPTIONAL

- -

The test case information function. It is supposed to - return a list of tagged tuples that specify various properties - related to the execution of this particular test case. - Properties set by - Testcase/0 - override properties set previously for the test case by - group/1 or - suite/0.

- -

Tag timetrap sets the maximum time that the - test case is allowed to execute. If the timetrap time is - exceeded, the test case fails with reason timetrap_timeout. - init_per_testcase/2 - and - end_per_testcase/2 - are included in the timetrap time. - A TimeFunc function can be used to - set a new timetrap by returning a TimeVal. It can also be - used to trigger a timetrap time-out by, at some point, returning a - value other than a TimeVal. For details, see section - Timetrap - Time-Outs in the User's Guide.

- -

Tag require specifies configuration variables - that are required by the test case (or init_per_testcase/2 - or end_per_testcase/2). - If the required configuration variables are not found in any of the - configuration files, the test case is skipped. For details about - the require functionality, see function - ct:require/1,2.

- -

If timetrap or require is not set, the - default values specified by - suite/0 (or - group/1) are used.

- -

With userdata, the user can specify any test case-related - information that can be read by calling - ct:userdata/3.

- -

Other tuples than the ones defined are ignored.

- -

For details about the test case information function, see section - Test - Case Information Function in the User's Guide.

- - - - - Module:Testcase(Config) -> term() | {skip,Reason} | {comment,Comment} | {save_config,SaveConfig} | {skip_and_save,Reason,SaveConfig} | exit() - A test case. - - Config = SaveConfig = [{Key,Value}] - Key = atom() - Value = term() - Reason = term() - Comment = string() - - - -

MANDATORY

- -

The implementation of a test case. Call the functions to test and - check the result. If something fails, ensure the - function causes a runtime error or call - ct:fail/1,2 - (which also causes the test case process to terminate).

- -

Elements from the Config list can, for example, be read - with proplists:get_value/2 in STDLIB - (or the macro ?config defined in ct.hrl).

- -

If you decide not to run the test case after all, return - {skip,Reason}. Reason is then - printed in field Comment on the HTML result page.

- -

To print some information in field Comment on the HTML - result page, return {comment,Comment}.

- -

If the function returns anything else, the test case is - considered successful. The return value always gets printed - in the test case log file.

- -

For details about test case implementation, see section - Test Cases - in the User's Guide.

- -

For information on save_config and skip_and_save, - see section - Saving - Configuration Data in the User's Guide.

- - - - - diff --git a/lib/common_test/doc/src/ct.xml b/lib/common_test/doc/src/ct.xml index 0d1c8076971c..c1f638b58054 100644 --- a/lib/common_test/doc/src/ct.xml +++ b/lib/common_test/doc/src/ct.xml @@ -57,9 +57,9 @@

data_dir - Data file directory

priv_dir - Scratch file directory

Whatever added by - init_per_suite/1 + init_per_suite/1 or - init_per_testcase/2 + init_per_testcase/2 in the test suite.

 @@ -1524,7 +1524,7 @@

Returns any data specified with tag userdata in the list of tuples returned from - suite/0.

+ suite/0.

 diff --git a/lib/common_test/doc/src/ct_hooks.xml b/lib/common_test/doc/src/ct_hooks.xml index 3c38dcd439be..875efec305a4 100644 --- a/lib/common_test/doc/src/ct_hooks.xml +++ b/lib/common_test/doc/src/ct_hooks.xml @@ -130,12 +130,12 @@

OPTIONAL

This function is called after - groups/0. + groups/0. It is used to modify the test group definitions, for instance to add or remove groups or change group properties.

GroupDefs is what - groups/0 + groups/0 returned, that is, a list of group definitions.

NewGroupDefs is the possibly modified version of this list.

@@ -146,7 +146,7 @@ in the User's Guide.

Notice that for CTHs that are installed by means of the - suite/0 + suite/0 function, post_groups/2 is called before the init/2 hook function. However, for CTHs that are installed by means @@ -191,19 +191,19 @@

OPTIONAL

This function is called after - all/0. + all/0. It is used to modify the set of test cases and test group to be executed, for instance to add or remove test cases and groups, change group properties, or even skip all tests in the suite.

Return is what - all/0 + all/0 returned, that is, a list of test cases and groups to be executed, or a tuple {skip,Reason}.

GroupDefs is what - groups/0 + groups/0 or the post_groups/2 hook returned, that is, a list of group definitions.

@@ -215,7 +215,7 @@ in the User's Guide.

Notice that for CTHs that are installed by means of the - suite/0 + suite/0 function, post_all/2 is called before the init/2 hook function. However, for CTHs that are installed by means @@ -253,7 +253,7 @@

OPTIONAL

This function is called before - init_per_suite + init_per_suite if it exists. It typically contains initialization/logging that must be done before init_per_suite is called. If {skip,Reason} or {fail,Reason} is returned, @@ -270,11 +270,11 @@

Return is the result of the init_per_suite function. If it is {skip,Reason} or {fail,Reason}, - init_per_suite + init_per_suite is never called, instead the initiation is considered to be skipped or failed, respectively. If a NewConfig list is returned, - init_per_suite + init_per_suite is called with that NewConfig list. For more details, see section Pre Hooks in the User's Guide.

@@ -304,21 +304,21 @@

OPTIONAL

This function is called after - init_per_suite + init_per_suite if it exists. It typically contains extra checks to ensure that all the correct dependencies are started correctly.

Return is what - init_per_suite + init_per_suite returned, that is, {fail,Reason}, {skip,Reason}, a Config list, or a term describing how - init_per_suite + init_per_suite failed.

NewReturn is the possibly modified return value of - init_per_suite. + init_per_suite. To recover from a failure in - init_per_suite, + init_per_suite, return ConfigList with the tc_status element removed. For more details, see Post Hooks in @@ -352,11 +352,11 @@

OPTIONAL

This function is called before - init_per_group + init_per_group if it exists. It behaves the same way as pre_init_per_suite, but for function - init_per_group + init_per_group instead.

If Module:pre_init_per_group/4 is not exported, common_test @@ -385,11 +385,11 @@

OPTIONAL

This function is called after - init_per_group + init_per_group if it exists. It behaves the same way as post_init_per_suite, but for function - init_per_group + init_per_group instead.

If Module:post_init_per_group/5 is not exported, common_test @@ -418,11 +418,11 @@

OPTIONAL

This function is called before - init_per_testcase + init_per_testcase if it exists. It behaves the same way as pre_init_per_suite, but for function - init_per_testcase + init_per_testcase instead.

If Module:pre_init_per_testcase/4 is not exported, common_test @@ -455,11 +455,11 @@

OPTIONAL

This function is called after - init_per_testcase + init_per_testcase if it exists. It behaves the same way as post_init_per_suite, but for function - init_per_testcase + init_per_testcase instead.

If Module:post_init_per_testcase/5 is not exported, common_test @@ -487,11 +487,11 @@

OPTIONAL

This function is called before - end_per_testcase + end_per_testcase if it exists. It behaves the same way as pre_end_per_suite, but for function - end_per_testcase + end_per_testcase instead.

This function cannot change the result of the test case by returning skip or fail @@ -524,11 +524,11 @@

OPTIONAL

This function is called after - end_per_testcase + end_per_testcase if it exists. It behaves the same way as post_end_per_suite, but for function - end_per_testcase + end_per_testcase instead.

If Module:post_end_per_testcase/5 is not exported, common_test @@ -557,11 +557,11 @@

OPTIONAL

This function is called before - end_per_group + end_per_group if it exists. It behaves the same way as pre_init_per_suite, but for function - end_per_group + end_per_group instead.

If Module:pre_end_per_group/4 is not exported, common_test @@ -590,11 +590,11 @@

OPTIONAL

This function is called after - end_per_group + end_per_group if it exists. It behaves the same way as post_init_per_suite, but for function - end_per_group + end_per_group instead.

If Module:post_end_per_group/5 is not exported, common_test @@ -622,11 +622,11 @@

OPTIONAL

This function is called before - end_per_suite + end_per_suite if it exists. It behaves the same way as pre_init_per_suite, but for function - end_per_suite + end_per_suite instead.

@@ -649,11 +649,11 @@

OPTIONAL

This function is called after - end_per_suite + end_per_suite if it exists. It behaves the same way as post_init_per_suite, but for function - end_per_suite + end_per_suite instead.

diff --git a/lib/common_test/doc/src/ct_hooks_chapter.xml b/lib/common_test/doc/src/ct_hooks_chapter.xml index e23a8b461a58..a4c72c1fa414 100644 --- a/lib/common_test/doc/src/ct_hooks_chapter.xml +++ b/lib/common_test/doc/src/ct_hooks_chapter.xml @@ -84,10 +84,10 @@

CTHs can also be added within a test suite. This is done by returning {ct_hooks,[CTH]} in the configuration list from - suite/0, - + suite/0, + init_per_suite/1, or - + init_per_group/2.

In this case, CTH can either be only the module name of the CTH @@ -157,7 +157,7 @@ the last test suite has been run - suite/0 + suite/0 pre_init_per_suite/3 is called @@ -165,7 +165,7 @@ post_end_per_suite/4 has been called for that test suite - + init_per_suite/1 post_init_per_suite/4 is called @@ -173,7 +173,7 @@ post_end_per_suite/4 has been called for that test suite - + init_per_group/2 post_init_per_group/5 is called @@ -236,12 +236,12 @@ In a CTH, the behavior can be hooked in before the following functions:

- init_per_suite - init_per_group - init_per_testcase - end_per_testcase - end_per_group - end_per_suite + init_per_suite + init_per_group + init_per_testcase + end_per_testcase + end_per_group + end_per_suite

@@ -282,12 +282,12 @@ Post Hooks

In a CTH, behavior can be hooked in after the following functions:

- init_per_suite - init_per_group - init_per_testcase - end_per_testcase - end_per_group - end_per_suite + init_per_suite + init_per_group + init_per_testcase + end_per_testcase + end_per_group + end_per_suite

diff --git a/lib/common_test/doc/src/ct_property_test.xml b/lib/common_test/doc/src/ct_property_test.xml index 891e0b475b7c..4c5eede7584e 100644 --- a/lib/common_test/doc/src/ct_property_test.xml +++ b/lib/common_test/doc/src/ct_property_test.xml @@ -186,7 +186,7 @@ prop_ftp() -> the output from for example proper:run_commands/2 or proper:run_parallel_commands/2 Config = - the Common Test Config in test cases. + the Common Test Config in test cases. Options = [present_option()] present_option() = {print_fun, fun(Format,Args)} diff --git a/lib/common_test/doc/src/ct_suite.xml b/lib/common_test/doc/src/ct_suite.xml new file mode 100644 index 000000000000..0289a569f02b --- /dev/null +++ b/lib/common_test/doc/src/ct_suite.xml @@ -0,0 +1,619 @@ + + + + +

+ + 2020 + Ericsson AB, All Rights Reserved + + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + The Initial Developer of the Original Code is Ericsson AB. + + ct_suite + + + + +
+ ct_suite + -behaviour(ct_suite). + + +

The following section describes the mandatory and optional test suite + functions that Common Test calls during test execution. + For more details, see section + Writing Test Suites + in the User's Guide.

+ + + + + +

The name of the testcase function.

 + + + +

The name of the test group.

 + + + +

The configuration data that can be modified.

 + + + +

The status value for a nested subgroup.

 + + + ct_group_def() +

The test group definition, as returned by Module:groups/0.

 + + + ct_test_def() +

The test suite definition, as returned by Module:all/0.

 + + + ct_info() +

The test suite information, as returned by Module:suite/0, Module:group/1 and Module:Testcase/0.

 + + + + + + Callback Functions +

+ The following functions are to be exported from a + ct_suite callback module in order to define + the callback interface for a test suite. +

+ + + + Module:all() -> [ct_test_def()] | {skip, Reason} + Returns the list of all test case groups and test cases + in the module. + + ct_test_def() = TestCase | {group, GroupName} | {group, GroupName, Properties} | {group, GroupName, Properties, SubGroups} + TestCase = ct_testname() + GroupName = ct_groupname() + Properties = [parallel | sequence | Shuffle | {RepeatType, N}] | default + SubGroups = [{GroupName, Properties} | {GroupName, Properties, SubGroups}] + Shuffle = shuffle | {shuffle, Seed} + Seed = {integer(), integer(), integer()} + RepeatType = repeat | repeat_until_all_ok | repeat_until_all_fail | repeat_until_any_ok | repeat_until_any_fail + N = integer() | forever + Reason = term() + + + +

MANDATORY

+ +

Returns the list of all test cases and test case groups in the + test suite module to be executed. This list also specifies the + order the cases and groups are executed by Common Test. + A test case is represented by an atom, + the name of the test case function, or a testcase tuple + indicating that the test case shall be repeated. A test case group is + represented by a group tuple, where GroupName, + an atom, is the name of the group (defined in + Module:groups/0). + Execution properties for groups can also be specified, both + for a top-level group and for any of its subgroups. + Group execution properties specified here override + properties in the group definition (see + Module:groups/0). + (With value default, the group definition properties + are used).

+ +

If {skip, Reason} is returned, all test cases + in the module are skipped and Reason + is printed on the HTML result page.

+ +

For details on groups, see section + Test Case + Groups in the User's Guide.

+ + + + + + Module:groups() -> [ct_group_def()] + Returns a list of test case group definitions. + + ct_group_def() = {GroupName, Properties, GroupsAndTestCases} + GroupName = ct_groupname() + Properties = [parallel | sequence | Shuffle | {RepeatType, N}] + GroupsAndTestCases = [Group | {group, GroupName} | TestCase] + TestCase = ct_testname() + Shuffle = shuffle | {shuffle, Seed} + Seed = {integer(), integer(), integer()} + RepeatType = repeat | repeat_until_all_ok | repeat_until_all_fail | repeat_until_any_ok | repeat_until_any_fail + N = integer() | forever + + + +

OPTIONAL

+ +

Defines test case groups. For details, see section + Test Case + Groups in the User's Guide.

+ + + + + Module:suite() -> [ct_info()] + Test suite info function (providing default data + for the suite). + + ct_info() = {timetrap, Time} | {require, Required} | {require, Name, Required} | {userdata, UserData} | {silent_connections, Conns} | {stylesheet, CSSFile} | {ct_hooks, CTHs} + Time = TimeVal | TimeFunc + TimeVal = MilliSec | {seconds, integer()} | {minutes, integer()} | {hours, integer()} + TimeFunc = {Mod, Func, Args} | Fun + MilliSec = integer() + Mod = atom() + Func = atom() + Args = list() + Fun = fun() + Required = Key | {Key, SubKeys} | {Key, SubKey} | {Key, SubKey, SubKeys} + Key = atom() + SubKeys = SubKey | [SubKey] + SubKey = atom() + Name = atom() + UserData = term() + Conns = [atom()] + CSSFile = string() + CTHs = [CTHModule | +         {CTHModule, CTHInitArgs} | +         {CTHModule, CTHInitArgs, CTHPriority}] + CTHModule = atom() + CTHInitArgs = term() + CTHPriority = integer() + + + +

OPTIONAL

+ +

The test suite information function. Returns a list of tagged + tuples specifying various properties related to the execution of + this test suite (common for all test cases in the suite).

+ +

Tag timetrap sets the maximum time that each + test case is allowed to execute (including + Module:init_per_testcase/2 + and + Module:end_per_testcase/2). + If the timetrap time is exceeded, the test case fails with reason + timetrap_timeout. A TimeFunc function can be used to + set a new timetrap by returning a TimeVal. It can also be + used to trigger a timetrap time-out by, at some point, returning a + value other than a TimeVal. For details, see section + Timetrap Time-Outs + in the User's Guide.

+ +

Tag require specifies configuration variables + required by test cases (or configuration functions) + in the suite. If the required configuration variables are not found + in any of the configuration files, all test cases are skipped. + For details about the require functionality, see funtion + ct:require/1,2.

+ +

With userdata, the user can + specify any test suite-related information, which can be + read by calling + ct:userdata/2.

+ +

Tag ct_hooks specifies the + Common Test Hooks + to be run with this suite.

+ +

Other tuples than the ones defined are ignored.

+ +

For details about the test suite information function, see section + Test + Suite Information Function in the User's Guide.

+ + + + + Module:init_per_suite(Config) -> NewConfig | {skip, Reason} | + {skip_and_save, Reason, SaveConfig} + Test suite initializations. + + Config = NewConfig = SaveConfig = ct_config() + Reason = term() + + + +

OPTIONAL; if this function is defined, then Module:end_per_suite/1 + must also be defined.

+ +

This configuration function is called as the first function in the + suite. It typically contains initializations that are common for + all test cases in the suite, and that must only be done + once. Parameter Config is the configuration data + that can be modified. Whatever is returned from this + function is specified as Config to all configuration functions + and test cases in the suite.

+ +

If {skip, Reason} + is returned, all test cases in the suite are skipped + and Reason is printed in the overview log for the suite.

+ +

For information on save_config and skip_and_save, + see section + Saving + Configuration Data in the User's Guide.

+ + + + + Module:end_per_suite(Config) -> term() | + {save_config, SaveConfig} + Test suite finalization. + + Config = SaveConfig = ct_config() + + + +

OPTIONAL; if this function is defined, then Module:init_per_suite/1 + must also be defined.

+ +

This function is called as the last test case in the + suite. It is meant to be used for cleaning up after + Module:init_per_suite/1.

+

For information on save_config, see section + Saving + Configuration Data in the User's Guide.

+ + + + + Module:group(GroupName) -> [ct_info()] + Test case group information function (providing default data + for a test case group, that is, its test cases and + subgroups). + + GroupName = ct_groupname() + ct_info() = {timetrap, Time} | {require, Required} | {require, Name, Required} | {userdata, UserData} | {silent_connections, Conns} | {stylesheet, CSSFile} | {ct_hooks, CTHs} + Time = TimeVal | TimeFunc + TimeVal = MilliSec | {seconds, integer()} | {minutes, integer()} | {hours, integer()} + TimeFunc = {Mod, Func, Args} | Fun + MilliSec = integer() + Mod = atom() + Func = atom() + Args = list() + Fun = fun() + Required = Key | {Key, SubKeys} | {Key, SubKey} | {Key, SubKey, SubKeys} + Key = atom() + SubKeys = SubKey | [SubKey] + SubKey = atom() + Name = atom() + UserData = term() + Conns = [atom()] + CSSFile = string() + CTHs = [CTHModule | +         {CTHModule, CTHInitArgs} | +         {CTHModule, CTHInitArgs, CTHPriority}] + CTHModule = atom() + CTHInitArgs = term() + CTHPriority = integer() + + + +

OPTIONAL

+ +

The test case group information function. It is supposed to + return a list of tagged tuples that specify various properties + related to the execution of a test case group (that is, its test + cases and subgroups). Properties set by + Module:group/1 override + properties with the same key that have been set previously by + Module:suite/0.

+ +

Tag timetrap sets the maximum time that each + test case is allowed to execute (including + Module:init_per_testcase/2 + and + Module:end_per_testcase/2). + If the timetrap time is + exceeded, the test case fails with reason + timetrap_timeout. A TimeFunc function can be used to + set a new timetrap by returning a TimeVal. It can also be + used to trigger a timetrap time-out by, at some point, returning a + value other than a TimeVal. For details, see section + Timetrap + Time-Outs in the User's Guide.

+ +

Tag require specifies configuration variables + required by test cases (or configuration functions) + in the suite. If the required configuration variables are not found + in any of the configuration files, all test cases in this group are + skipped. For details about the require functionality, see + function + ct:require/1,2.

+ +

With userdata, the user can + specify any test case group related information that can be + read by calling + ct:userdata/2.

+ +

Tag ct_hooks specifies the + Common Test Hooks + to be run with this suite.

+ +

Other tuples than the ones defined are ignored.

+ +

For details about the test case group information function, + see section Group + Information Function in the User's Guide.

+ + + + + Module:init_per_group(GroupName, Config) -> NewConfig | + {skip, Reason} + Test case group initializations. + + GroupName = ct_groupname() + Config = NewConfig = ct_config() + Reason = term() + + + +

OPTIONAL; if this function is defined, then Module:end_per_group/2 + must also be defined.

+ +

This configuration function is called before execution of a + test case group. It typically contains initializations that are + common for all test cases and subgroups in the group, and that + must only be performed once. GroupName is the name of the + group, as specified in the group definition (see + Module:groups/0). + Parameter Config is the configuration data that can be + modified. + The return value of this function is given as Config + to all test cases and subgroups in the group.

+ +

If {skip, Reason} + is returned, all test cases in the group are skipped and + Reason is printed in the overview log for the group.

+ +

For information about test case groups, see section + Test Case + Groups in the User's Guide.

+ + + + + Module:end_per_group(GroupName, Config) -> term() | + {return_group_result, Status} + Test case group finalization. + + GroupName = ct_groupname() + Config = ct_config() + Status = ct_status() + + + +

OPTIONAL; if this function is defined, then Module:init_per_group/2 + must also be defined.

+ +

This function is called after the execution of a test case group + is finished. It is meant to be used for cleaning up after + Module:init_per_group/2. + A status value for a nested subgroup can be returned with + {return_group_result, Status}. The status can be retrieved in + Module:end_per_group/2 + for the group on the level above. The status is also used by + Common Test for deciding if execution of a group is to + proceed if property sequence or repeat_until_* + is set.

+ +

For details about test case groups, see section + Test Case + Groups in the User's Guide.

+ + + + + Module:init_per_testcase(TestCase, Config) -> NewConfig | {fail, Reason} | {skip, Reason} + Test case initializations. + + TestCase = ct_testname() + Config = NewConfig = ct_config() + Reason = term() + + + +

OPTIONAL; if this function is defined, + then + Module:end_per_testcase/2 must also be + defined.

+ +

This function is called before each test case. Argument + TestCase is the test case name, and + Config (list of key-value tuples) is the configuration + data that can be modified. The NewConfig list returned + from this function is given as Config to the test case. + If {fail, Reason} is returned, the test case is + marked as failed without being executed.

+ +

If {skip, Reason} is returned, the test case is skipped + and Reason is printed in the overview log for the suite.

+ + + + + Module:end_per_testcase(TestCase, Config) -> term() | {fail, Reason} | {save_config, SaveConfig} + Test case finalization. + + TestCase = ct_testname() + Config = SaveConfig = ct_config() + Reason = term() + + + +

OPTIONAL; if this function is defined, + then + Module:init_per_testcase/2 must also be + defined.

+ +

This function is called after each test case, and can be used + to clean up after + Module:init_per_testcase/2 + and the test case. Any return value (besides {fail, Reason} + and {save_config, SaveConfig}) is ignored. By returning + {fail, Reason}, TestCase is marked as faulty (even + though it was successful in the sense that it returned + a value instead of terminating).

+ +

For information on save_config, see section + Saving + Configuration Data in the User's Guide.

+ + + + + Module:Testcase() -> [ct_info()] + Test case information function. + + ct_info() = {timetrap, Time} | {require, Required} | {require, Name, Required} | {userdata, UserData} | {silent_connections, Conns} | {stylesheet, CSSFile} | {ct_hooks, CTHs} + Time = TimeVal | TimeFunc + TimeVal = MilliSec | {seconds, integer()} | {minutes, integer()} | {hours, integer()} + TimeFunc = {Mod, Func, Args} | Fun + MilliSec = integer() + Mod = atom() + Func = atom() + Args = list() + Fun = fun() + Required = Key | {Key, SubKeys} | {Key, SubKey} | {Key, SubKey, SubKeys} + Key = atom() + SubKeys = SubKey | [SubKey] + SubKey = atom() + Name = atom() + UserData = term() + Conns = [atom()] + CSSFile = string() + CTHs = [CTHModule | +         {CTHModule, CTHInitArgs} | +         {CTHModule, CTHInitArgs, CTHPriority}] + CTHModule = atom() + CTHInitArgs = term() + CTHPriority = integer() + + + + +

OPTIONAL

+ +

The test case information function. It is supposed to + return a list of tagged tuples that specify various properties + related to the execution of this particular test case. + Properties set by + Module:Testcase/0 + override properties set previously for the test case by + Module:group/1 or + Module:suite/0.

+ +

Tag timetrap sets the maximum time that the + test case is allowed to execute. If the timetrap time is + exceeded, the test case fails with reason timetrap_timeout. + Module:init_per_testcase/2 + and + Module:end_per_testcase/2 + are included in the timetrap time. + A TimeFunc function can be used to + set a new timetrap by returning a TimeVal. It can also be + used to trigger a timetrap time-out by, at some point, returning a + value other than a TimeVal. For details, see section + Timetrap + Time-Outs in the User's Guide.

+ +

Tag require specifies configuration variables + that are required by the test case (or init_per_testcase/2 + or end_per_testcase/2). + If the required configuration variables are not found in any of the + configuration files, the test case is skipped. For details about + the require functionality, see function + ct:require/1,2.

+ +

If timetrap or require is not set, the + default values specified by + Module:suite/0 (or + Module:group/1) are used.

+ +

With userdata, the user can specify any test case-related + information that can be read by calling + ct:userdata/3.

+ +

Other tuples than the ones defined are ignored.

+ +

For details about the test case information function, see section + Test + Case Information Function in the User's Guide.

+ + + + + Module:Testcase(Config) -> term() | {skip, Reason} | {comment, Comment} | {save_config, SaveConfig} | {skip_and_save, Reason, SaveConfig} | exit() + A test case. + + Config = SaveConfig = ct_config() + Reason = term() + Comment = string() + + + +

MANDATORY

+ +

The implementation of a test case. Call the functions to test and + check the result. If something fails, ensure the + function causes a runtime error or call + ct:fail/1,2 + (which also causes the test case process to terminate).

+ +

Elements from the Config list can, for example, be read + with proplists:get_value/2 in STDLIB + (or the macro ?config defined in ct.hrl).

+ +

If you decide not to run the test case after all, return + {skip, Reason}. Reason is then + printed in field Comment on the HTML result page.

+ +

To print some information in field Comment on the HTML + result page, return {comment, Comment}.

+ +

If the function returns anything else, the test case is + considered successful. The return value always gets printed + in the test case log file.

+ +

For details about test case implementation, see section + Test Cases + in the User's Guide.

+ +

For information on save_config and skip_and_save, + see section + Saving + Configuration Data in the User's Guide.

+ + + + + + diff --git a/lib/common_test/doc/src/dependencies_chapter.xml b/lib/common_test/doc/src/dependencies_chapter.xml index 53a767348928..0a0f4c562fc6 100644 --- a/lib/common_test/doc/src/dependencies_chapter.xml +++ b/lib/common_test/doc/src/dependencies_chapter.xml @@ -87,8 +87,8 @@

To avoid this, we can consider starting and stopping the server for every test. We can thus implement the start and stop action as common functions to be called from - init_per_testcase and - end_per_testcase. + init_per_testcase and + end_per_testcase. (Remember to test the start and stop functionality separately.) The configuration can also be implemented as a common function, maybe grouped with the start function. Finally, the testing of connecting and disconnecting a @@ -194,9 +194,9 @@

To pass data from one test suite to another, the same mechanism is used. The data is to be saved by finction - end_per_suite + end_per_suite and read by function - init_per_suite + init_per_suite in the suite that follows. When passing data between suites, Saver carries the name of the test suite.

@@ -306,7 +306,7 @@ any property, that is, they are not required to also be sequences. If you want the status of the subgroup to affect the sequence on the level above, return {return_group_result,Status} from - end_per_group/2, + end_per_group/2, as described in section Repeated Groups in Writing Test Suites. diff --git a/lib/common_test/doc/src/ref_man.xml b/lib/common_test/doc/src/ref_man.xml index e916fc7cec7a..b8184ed7f8ba 100644 --- a/lib/common_test/doc/src/ref_man.xml +++ b/lib/common_test/doc/src/ref_man.xml @@ -48,6 +48,7 @@ + diff --git a/lib/common_test/doc/src/specs.xml b/lib/common_test/doc/src/specs.xml index 7e40e8351d74..def8f761a544 100644 --- a/lib/common_test/doc/src/specs.xml +++ b/lib/common_test/doc/src/specs.xml @@ -2,4 +2,5 @@ + diff --git a/lib/common_test/doc/src/test_structure_chapter.xml b/lib/common_test/doc/src/test_structure_chapter.xml index 42eb3d6e236b..e041376d7f32 100644 --- a/lib/common_test/doc/src/test_structure_chapter.xml +++ b/lib/common_test/doc/src/test_structure_chapter.xml @@ -60,8 +60,8 @@ Returning {skip,Reason} from function - init_per_testcase/2 or - init_per_suite/1. + init_per_testcase/2 or + init_per_suite/1. Returning {skip,Reason} from the execution clause of the test case. The execution clause is called, so the author diff --git a/lib/common_test/doc/src/write_test_chapter.xml b/lib/common_test/doc/src/write_test_chapter.xml index 3c3790cfc530..661ba04fe887 100644 --- a/lib/common_test/doc/src/write_test_chapter.xml +++ b/lib/common_test/doc/src/write_test_chapter.xml @@ -67,7 +67,7 @@

Each test suite module must export function - all/0, + all/0, which returns the list of all test case groups and test cases to be executed in that module.

@@ -83,8 +83,8 @@ Init and End per Suite

Each test suite module can contain the optional configuration functions - init_per_suite/1 - and end_per_suite/1. + init_per_suite/1 + and end_per_suite/1. If the init function is defined, so must the end function be.

@@ -138,8 +138,8 @@ Init and End per Test Case

Each test suite module can contain the optional configuration functions - init_per_testcase/2 - and end_per_testcase/2. + init_per_testcase/2 + and end_per_testcase/2. If the init function is defined, so must the end function be.

If init_per_testcase exists, it is called before each @@ -378,7 +378,7 @@

If timetrap or require, or both, is not set specifically for a particular test case, default values specified by function - suite/0 + suite/0 are used.

@@ -404,7 +404,7 @@ Test Suite Information Function -

Function suite/0 +

Function suite/0 can, for example, be used in a test suite module to set a default timetrap value and to require external configuration data. If a test case, or a group information function also specifies any of the information tags, it @@ -445,7 +445,7 @@

A test case group is a set of test cases sharing configuration functions and execution properties. Test case groups are defined by function - groups/0 + groups/0 according to the following syntax:

  groups() -> GroupDefs
@@ -561,12 +561,12 @@
       execution is immediately stopped and the remaining cases are skipped.

 
     
Before execution of a group begins, the configuration function
-    init_per_group(GroupName, Config) 
+    init_per_group(GroupName, Config) 
     is called. The list of tuples returned from this function is passed to the 
     test cases in the usual manner by argument Config. 
     init_per_group/2 is meant to be used for initializations common 
     for the test cases in the group. After execution of the group is finished, function
-    end_per_group(GroupName, Config) 
+    end_per_group(GroupName, Config) 
     is called. This function is meant to be used for cleaning up after 
     init_per_group/2. If the init function is defined, so must the end function be.

 
@@ -1163,7 +1163,7 @@ ct:pal(?ERROR, "Error report: ~p", [Error])
environment as possible, so that subsequent test cases do not crash because of their execution order. The function - end_per_testcase + end_per_testcase is suitable for this.

diff --git a/lib/common_test/src/Makefile b/lib/common_test/src/Makefile index a20de14e0e3e..7d7b5ed203d7 100644 --- a/lib/common_test/src/Makefile +++ b/lib/common_test/src/Makefile @@ -40,6 +40,9 @@ RELSYSDIR = $(RELEASE_PATH)/lib/common_test-$(VSN) # Target Specs # ---------------------------------------------------- +BEHAVIOUR_MODULES= \ + ct_suite + MODULES= \ ct \ ct_logs \ @@ -88,9 +91,13 @@ MODULES= \ TARGET_MODULES= $(MODULES:%=$(EBIN)/%) -BEAM_FILES= $(MODULES:%=$(EBIN)/%.$(EMULATOR)) +BEAM_FILES= $(MODULES:%=$(EBIN)/%.$(EMULATOR)) \ + $(BEHAVIOUR_MODULES:%=$(EBIN)/%.$(EMULATOR)) + +ERL_FILES= \ + $(MODULES:=.erl) \ + $(BEHAVIOUR_MODULES:%=%.erl) -ERL_FILES= $(MODULES:=.erl) HRL_FILES = \ ct_util.hrl \ ct_netconfc.hrl diff --git a/lib/common_test/src/common_test.app.src b/lib/common_test/src/common_test.app.src index 07b35b71800c..8aa15efa7e67 100644 --- a/lib/common_test/src/common_test.app.src +++ b/lib/common_test/src/common_test.app.src @@ -63,7 +63,8 @@ test_server_gl, test_server_io, test_server_node, - test_server_sup + test_server_sup, + ct_suite ]}, {registered, [ct_logs, ct_util_server, diff --git a/lib/common_test/src/ct_suite.erl b/lib/common_test/src/ct_suite.erl new file mode 100644 index 000000000000..a2d23e15ef14 --- /dev/null +++ b/lib/common_test/src/ct_suite.erl @@ -0,0 +1,130 @@ +-module(ct_suite). + +%%------------------------------------------------------------------ +%% Test Suite Behaviour +%% ------------------------------------------------------------------ +-export_type([ct_testname/0, + ct_groupname/0, + ct_config/0, + ct_status/0, + ct_group_def/0, + ct_test_def/0, + ct_info/0 + ]). + +-type ct_testname() :: atom(). +-type ct_groupname() :: atom(). +-type ct_config() :: [{Key :: atom(), Value :: term()}]. +-type ct_status() :: ok | + skipped | + failed. +-type ct_group_props() :: [ + parallel | + sequence | + shuffle | + {shuffle, Seed :: {integer(), integer(), integer()}} | + {ct_group_repeat_type(), ct_test_repeat()} + ]. +-type ct_group_props_ref() :: + ct_group_props() | + default. +-type ct_group_repeat_type() :: repeat | + repeat_until_all_ok | + repeat_until_all_fail | + repeat_until_any_ok | + repeat_until_any_fail. +-type ct_test_repeat() :: integer() | + forever. +-type ct_group_def() :: {ct_groupname(), ct_group_props(), [ + ct_testname() | + ct_group_def() | + {group, ct_groupname()} | + ct_testcase_ref() + ]}. +-type ct_subgroups_def() :: {ct_groupname(), ct_group_props_ref()} | + {ct_groupname(), ct_group_props_ref(), ct_subgroups_def()}. +-type ct_group_ref() :: {group, ct_groupname()} | + {group, ct_groupname(), ct_group_props_ref()} | + {group, ct_groupname(), ct_group_props_ref(), ct_subgroups_def()}. +-type ct_testcase_ref() :: {testcase, ct_testname(), ct_testcase_repeat_prop()}. +-type ct_testcase_repeat_prop() :: {repeat, ct_test_repeat()} | + {repeat_until_ok, ct_test_repeat()} | + {repeat_until_fail, ct_test_repeat()}. +-type ct_info() :: {timetrap, ct_info_timetrap()} | + {require, ct_info_required()} | + {require, Name :: atom(), ct_info_required()} | + {userdata, UserData :: term()} | + {silent_connections, Conns :: [atom()]} | + {stylesheet, CSSFile :: string()} | + {ct_hooks, CTHs :: ct_hooks()}. +-type ct_info_timetrap() :: MilliSec :: integer() | + {seconds, integer()} | + {minutes, integer()} | + {hours, integer()} | + {Mod :: atom(), Func :: atom(), Args :: list()} | + ct_info_timetrap_fun(). +-type ct_info_timetrap_fun() :: fun(). +-type ct_info_required() :: Key :: atom() | + {Key :: atom(), SubKeys :: ct_info_required_subkeys()} | + {Key :: atom(), SubKey :: atom()} | + {Key :: atom(), SubKey :: atom(), SubKeys :: ct_info_required_subkeys()}. +-type ct_info_required_subkeys() :: SubKey :: atom() | + [SubKey :: atom()]. +-type ct_hooks() :: [ + CTHModule :: atom() | + {CTHModule :: atom(), CTHInitArgs :: term()} | + {CTHModule :: atom(), CTHInitArgs :: term(), CTHPriority :: integer()} + ]. +-type ct_test_def() :: ct_testname() | ct_group_ref() | ct_testcase_ref(). + +-callback all() -> + [TestDef :: ct_test_def()] | + {skip, Reason :: term()}. + +-callback groups() -> + [GroupDef :: ct_group_def()]. + +-callback suite() -> + [Info :: ct_info()]. + +-callback init_per_suite(Config :: ct_config()) -> + NewConfig :: ct_config() | + {skip, Reason :: term()} | + {skip_and_save, Reason :: term(), SaveConfig :: ct_config()}. + +-callback end_per_suite(Config :: ct_config()) -> + term() | + {save_config, SaveConfig :: ct_config()}. + +-callback group(GroupName :: ct_groupname()) -> + [Info :: ct_info()]. + +-callback init_per_group(GroupName :: ct_groupname(), Config :: ct_config()) -> + NewConfig :: ct_config() | + {skip, Reason :: term()}. + +-callback end_per_group(GroupName :: ct_groupname(), Config :: ct_config()) -> + term() | + {return_group_result, Status :: ct_status()}. + +-callback init_per_testcase(TestCase :: ct_testname(), Config :: ct_config()) -> + NewConfig :: ct_config() | + {fail, Reason :: term()} | + {skip, Reason :: term()}. + +-callback end_per_testcase(TestCase :: ct_testname(), Config :: ct_config()) -> + term() | + {fail, Reason :: term()} | + {save_config, SaveConfig :: ct_config()}. + +%% only all/0 is mandatory +-optional_callbacks([groups/0, + suite/0, + init_per_suite/1, + end_per_suite/1, + group/1, + init_per_group/2, + end_per_group/2, + init_per_testcase/2, + end_per_testcase/2 + ]). diff --git a/lib/ssl/test/ssl_ECC.erl b/lib/ssl/test/ssl_ECC.erl index 56b67361e175..73dafcd1fc93 100644 --- a/lib/ssl/test/ssl_ECC.erl +++ b/lib/ssl/test/ssl_ECC.erl @@ -23,6 +23,8 @@ -module(ssl_ECC). +-behaviour(ct_suite). + -include_lib("common_test/include/ct.hrl"). -include_lib("public_key/include/public_key.hrl"). diff --git a/lib/ssl/test/ssl_ECC_SUITE.erl b/lib/ssl/test/ssl_ECC_SUITE.erl index 496cb38fd98d..f834ac410573 100644 --- a/lib/ssl/test/ssl_ECC_SUITE.erl +++ b/lib/ssl/test/ssl_ECC_SUITE.erl @@ -22,6 +22,8 @@ -module(ssl_ECC_SUITE). +-behaviour(ct_suite). + -include_lib("common_test/include/ct.hrl"). -include_lib("public_key/include/public_key.hrl"). diff --git a/lib/ssl/test/ssl_alert_SUITE.erl b/lib/ssl/test/ssl_alert_SUITE.erl index ffdc9ea91677..3b32d3ece15e 100644 --- a/lib/ssl/test/ssl_alert_SUITE.erl +++ b/lib/ssl/test/ssl_alert_SUITE.erl @@ -20,6 +20,8 @@ -module(ssl_alert_SUITE). +-behaviour(ct_suite). + -include_lib("common_test/include/ct.hrl"). -include_lib("public_key/include/public_key.hrl"). diff --git a/lib/ssl/test/ssl_alpn_SUITE.erl b/lib/ssl/test/ssl_alpn_SUITE.erl index f0be74071658..9cc3303604b7 100644 --- a/lib/ssl/test/ssl_alpn_SUITE.erl +++ b/lib/ssl/test/ssl_alpn_SUITE.erl @@ -21,6 +21,8 @@ %% -module(ssl_alpn_SUITE). +-behaviour(ct_suite). + -include_lib("common_test/include/ct.hrl"). %% Callback functions diff --git a/lib/ssl/test/ssl_api_SUITE.erl b/lib/ssl/test/ssl_api_SUITE.erl index d52645cc52fb..95e3c0a058fc 100644 --- a/lib/ssl/test/ssl_api_SUITE.erl +++ b/lib/ssl/test/ssl_api_SUITE.erl @@ -21,6 +21,8 @@ %% -module(ssl_api_SUITE). +-behaviour(ct_suite). + -include_lib("common_test/include/ct.hrl"). -include_lib("ssl/src/ssl_api.hrl"). diff --git a/lib/ssl/test/ssl_app_env_SUITE.erl b/lib/ssl/test/ssl_app_env_SUITE.erl index ee9c3ad74552..98e96023b247 100644 --- a/lib/ssl/test/ssl_app_env_SUITE.erl +++ b/lib/ssl/test/ssl_app_env_SUITE.erl @@ -21,6 +21,8 @@ %% -module(ssl_app_env_SUITE). +-behaviour(ct_suite). + -include_lib("common_test/include/ct.hrl"). -include_lib("ssl/src/ssl_api.hrl"). diff --git a/lib/ssl/test/ssl_basic_SUITE.erl b/lib/ssl/test/ssl_basic_SUITE.erl index 0039b0244f51..1426c7ef49b8 100644 --- a/lib/ssl/test/ssl_basic_SUITE.erl +++ b/lib/ssl/test/ssl_basic_SUITE.erl @@ -22,6 +22,8 @@ -module(ssl_basic_SUITE). +-behaviour(ct_suite). + -include_lib("common_test/include/ct.hrl"). -include_lib("ssl/src/ssl_api.hrl"). diff --git a/lib/ssl/test/ssl_bench_SUITE.erl b/lib/ssl/test/ssl_bench_SUITE.erl index fe30a83a7e46..f78ac9c9cc80 100644 --- a/lib/ssl/test/ssl_bench_SUITE.erl +++ b/lib/ssl/test/ssl_bench_SUITE.erl @@ -19,6 +19,8 @@ %% -module(ssl_bench_SUITE). +-behaviour(ct_suite). + -include_lib("common_test/include/ct_event.hrl"). %% Callback functions diff --git a/lib/ssl/test/ssl_bench_test_lib.erl b/lib/ssl/test/ssl_bench_test_lib.erl index cbb9cfe47abd..74ab142993e1 100644 --- a/lib/ssl/test/ssl_bench_test_lib.erl +++ b/lib/ssl/test/ssl_bench_test_lib.erl @@ -19,6 +19,8 @@ %% -module(ssl_bench_test_lib). +-behaviour(ct_suite). + %% API -export([setup/1]). diff --git a/lib/ssl/test/ssl_cert_SUITE.erl b/lib/ssl/test/ssl_cert_SUITE.erl index bec6057d9123..655d29b6ae76 100644 --- a/lib/ssl/test/ssl_cert_SUITE.erl +++ b/lib/ssl/test/ssl_cert_SUITE.erl @@ -21,6 +21,8 @@ %% -module(ssl_cert_SUITE). +-behaviour(ct_suite). + -include_lib("common_test/include/ct.hrl"). -include_lib("public_key/include/public_key.hrl"). diff --git a/lib/ssl/test/ssl_cert_tests.erl b/lib/ssl/test/ssl_cert_tests.erl index 5422ff7fe4fe..d2bf39d88c65 100644 --- a/lib/ssl/test/ssl_cert_tests.erl +++ b/lib/ssl/test/ssl_cert_tests.erl @@ -21,6 +21,8 @@ %% -module(ssl_cert_tests). +-behaviour(ct_suite). + -include_lib("public_key/include/public_key.hrl"). %% Test cases diff --git a/lib/ssl/test/ssl_cipher_SUITE.erl b/lib/ssl/test/ssl_cipher_SUITE.erl index eee8d8078be1..31e60269f8bb 100644 --- a/lib/ssl/test/ssl_cipher_SUITE.erl +++ b/lib/ssl/test/ssl_cipher_SUITE.erl @@ -20,6 +20,8 @@ -module(ssl_cipher_SUITE). +-behaviour(ct_suite). + -include_lib("common_test/include/ct.hrl"). -include("tls_record.hrl"). -include("ssl_cipher.hrl"). diff --git a/lib/ssl/test/ssl_cipher_suite_SUITE.erl b/lib/ssl/test/ssl_cipher_suite_SUITE.erl index 6952472300c6..99911af3706a 100644 --- a/lib/ssl/test/ssl_cipher_suite_SUITE.erl +++ b/lib/ssl/test/ssl_cipher_suite_SUITE.erl @@ -22,6 +22,8 @@ -module(ssl_cipher_suite_SUITE). +-behaviour(ct_suite). + -include_lib("common_test/include/ct.hrl"). %% Callback functions -export([all/0, diff --git a/lib/ssl/test/ssl_crl_SUITE.erl b/lib/ssl/test/ssl_crl_SUITE.erl index ad75d29a78f3..f9fac829629e 100644 --- a/lib/ssl/test/ssl_crl_SUITE.erl +++ b/lib/ssl/test/ssl_crl_SUITE.erl @@ -21,6 +21,8 @@ -module(ssl_crl_SUITE). +-behaviour(ct_suite). + -include_lib("common_test/include/ct.hrl"). -include_lib("public_key/include/public_key.hrl"). diff --git a/lib/ssl/test/ssl_dist_SUITE.erl b/lib/ssl/test/ssl_dist_SUITE.erl index 20463114f4fb..019e22eaa842 100644 --- a/lib/ssl/test/ssl_dist_SUITE.erl +++ b/lib/ssl/test/ssl_dist_SUITE.erl @@ -20,6 +20,8 @@ -module(ssl_dist_SUITE). +-behaviour(ct_suite). + -include_lib("common_test/include/ct.hrl"). -include_lib("public_key/include/public_key.hrl"). -include("ssl_dist_test_lib.hrl"). diff --git a/lib/ssl/test/ssl_dist_bench_SUITE.erl b/lib/ssl/test/ssl_dist_bench_SUITE.erl index 67944c74d27c..2216c6b04b71 100644 --- a/lib/ssl/test/ssl_dist_bench_SUITE.erl +++ b/lib/ssl/test/ssl_dist_bench_SUITE.erl @@ -19,6 +19,8 @@ %% -module(ssl_dist_bench_SUITE). +-behaviour(ct_suite). + -include_lib("common_test/include/ct_event.hrl"). -include_lib("public_key/include/public_key.hrl"). diff --git a/lib/ssl/test/ssl_dist_test_lib.erl b/lib/ssl/test/ssl_dist_test_lib.erl index fa1f4217feb1..bf010e6ad4b4 100644 --- a/lib/ssl/test/ssl_dist_test_lib.erl +++ b/lib/ssl/test/ssl_dist_test_lib.erl @@ -20,6 +20,8 @@ -module(ssl_dist_test_lib). +-behaviour(ct_suite). + -include_lib("common_test/include/ct.hrl"). -include_lib("public_key/include/public_key.hrl"). -include("ssl_dist_test_lib.hrl"). diff --git a/lib/ssl/test/ssl_engine_SUITE.erl b/lib/ssl/test/ssl_engine_SUITE.erl index 4e88c6e46cfe..c0e28120bee7 100644 --- a/lib/ssl/test/ssl_engine_SUITE.erl +++ b/lib/ssl/test/ssl_engine_SUITE.erl @@ -21,6 +21,8 @@ %% -module(ssl_engine_SUITE). +-behaviour(ct_suite). + -include_lib("common_test/include/ct.hrl"). -include_lib("public_key/include/public_key.hrl"). diff --git a/lib/ssl/test/ssl_eqc_SUITE.erl b/lib/ssl/test/ssl_eqc_SUITE.erl index 52edb39d9cca..b663597835ed 100644 --- a/lib/ssl/test/ssl_eqc_SUITE.erl +++ b/lib/ssl/test/ssl_eqc_SUITE.erl @@ -20,6 +20,7 @@ -module(ssl_eqc_SUITE). +-behaviour(ct_suite). %% Common test -export([all/0, diff --git a/lib/ssl/test/ssl_handshake_SUITE.erl b/lib/ssl/test/ssl_handshake_SUITE.erl index f990b9328ae4..6da6da4051f7 100644 --- a/lib/ssl/test/ssl_handshake_SUITE.erl +++ b/lib/ssl/test/ssl_handshake_SUITE.erl @@ -22,6 +22,8 @@ -module(ssl_handshake_SUITE). +-behaviour(ct_suite). + -include_lib("common_test/include/ct.hrl"). -include("ssl_alert.hrl"). -include("ssl_handshake.hrl"). diff --git a/lib/ssl/test/ssl_key_update_SUITE.erl b/lib/ssl/test/ssl_key_update_SUITE.erl index 2816f1a39e22..aba6a02ddc7a 100644 --- a/lib/ssl/test/ssl_key_update_SUITE.erl +++ b/lib/ssl/test/ssl_key_update_SUITE.erl @@ -19,6 +19,8 @@ %% -module(ssl_key_update_SUITE). +-behaviour(ct_suite). + %% Callback functions -export([all/0, groups/0, diff --git a/lib/ssl/test/ssl_mfl_SUITE.erl b/lib/ssl/test/ssl_mfl_SUITE.erl index e8fa0ddf5264..0f9aeb1c674a 100644 --- a/lib/ssl/test/ssl_mfl_SUITE.erl +++ b/lib/ssl/test/ssl_mfl_SUITE.erl @@ -18,6 +18,9 @@ %% %CopyrightEnd% %% -module(ssl_mfl_SUITE). + +-behaviour(ct_suite). + -include_lib("common_test/include/ct.hrl"). %% Common test diff --git a/lib/ssl/test/ssl_npn_SUITE.erl b/lib/ssl/test/ssl_npn_SUITE.erl index addf105e592b..81c75ecff04a 100644 --- a/lib/ssl/test/ssl_npn_SUITE.erl +++ b/lib/ssl/test/ssl_npn_SUITE.erl @@ -21,6 +21,8 @@ %% -module(ssl_npn_SUITE). +-behaviour(ct_suite). + -include_lib("common_test/include/ct.hrl"). %% Callback functions diff --git a/lib/ssl/test/ssl_npn_hello_SUITE.erl b/lib/ssl/test/ssl_npn_hello_SUITE.erl index ee84383c612d..dae07aae6309 100644 --- a/lib/ssl/test/ssl_npn_hello_SUITE.erl +++ b/lib/ssl/test/ssl_npn_hello_SUITE.erl @@ -22,6 +22,8 @@ -module(ssl_npn_hello_SUITE). +-behaviour(ct_suite). + -include_lib("ssl/src/tls_record.hrl"). -include_lib("ssl/src/tls_handshake.hrl"). -include_lib("ssl/src/ssl_cipher.hrl"). diff --git a/lib/ssl/test/ssl_packet_SUITE.erl b/lib/ssl/test/ssl_packet_SUITE.erl index bd390d67bf06..a65173f1723e 100644 --- a/lib/ssl/test/ssl_packet_SUITE.erl +++ b/lib/ssl/test/ssl_packet_SUITE.erl @@ -21,6 +21,8 @@ %% -module(ssl_packet_SUITE). +-behaviour(ct_suite). + -include_lib("common_test/include/ct.hrl"). %% Callback functions diff --git a/lib/ssl/test/ssl_payload_SUITE.erl b/lib/ssl/test/ssl_payload_SUITE.erl index df8394ab148a..e594c745b694 100644 --- a/lib/ssl/test/ssl_payload_SUITE.erl +++ b/lib/ssl/test/ssl_payload_SUITE.erl @@ -20,6 +20,8 @@ -module(ssl_payload_SUITE). +-behaviour(ct_suite). + -include_lib("common_test/include/ct.hrl"). %% Common test -export([all/0, diff --git a/lib/ssl/test/ssl_pem_cache_SUITE.erl b/lib/ssl/test/ssl_pem_cache_SUITE.erl index 2469b47040ab..50d808905485 100644 --- a/lib/ssl/test/ssl_pem_cache_SUITE.erl +++ b/lib/ssl/test/ssl_pem_cache_SUITE.erl @@ -22,6 +22,8 @@ -module(ssl_pem_cache_SUITE). +-behaviour(ct_suite). + -include_lib("common_test/include/ct.hrl"). -include_lib("kernel/include/file.hrl"). diff --git a/lib/ssl/test/ssl_renegotiate_SUITE.erl b/lib/ssl/test/ssl_renegotiate_SUITE.erl index 18ca56b6f976..4b4686341585 100644 --- a/lib/ssl/test/ssl_renegotiate_SUITE.erl +++ b/lib/ssl/test/ssl_renegotiate_SUITE.erl @@ -22,6 +22,8 @@ -module(ssl_renegotiate_SUITE). +-behaviour(ct_suite). + -include_lib("common_test/include/ct.hrl"). -include_lib("public_key/include/public_key.hrl"). diff --git a/lib/ssl/test/ssl_rfc_5869_SUITE.erl b/lib/ssl/test/ssl_rfc_5869_SUITE.erl index d350dd099da8..497a05107b4c 100644 --- a/lib/ssl/test/ssl_rfc_5869_SUITE.erl +++ b/lib/ssl/test/ssl_rfc_5869_SUITE.erl @@ -21,6 +21,8 @@ %% -module(ssl_rfc_5869_SUITE). +-behaviour(ct_suite). + -include_lib("common_test/include/ct.hrl"). %% Common test diff --git a/lib/ssl/test/ssl_session_SUITE.erl b/lib/ssl/test/ssl_session_SUITE.erl index d123b38d5597..0a614f8b8c76 100644 --- a/lib/ssl/test/ssl_session_SUITE.erl +++ b/lib/ssl/test/ssl_session_SUITE.erl @@ -21,6 +21,8 @@ %% -module(ssl_session_SUITE). +-behaviour(ct_suite). + -include("tls_handshake.hrl"). -include("ssl_record.hrl"). diff --git a/lib/ssl/test/ssl_session_cache_SUITE.erl b/lib/ssl/test/ssl_session_cache_SUITE.erl index c53161f16868..7c182ee063de 100644 --- a/lib/ssl/test/ssl_session_cache_SUITE.erl +++ b/lib/ssl/test/ssl_session_cache_SUITE.erl @@ -22,6 +22,8 @@ -module(ssl_session_cache_SUITE). +-behaviour(ct_suite). + -include_lib("common_test/include/ct.hrl"). %% Callback functions diff --git a/lib/ssl/test/ssl_session_ticket_SUITE.erl b/lib/ssl/test/ssl_session_ticket_SUITE.erl index 218c4a156471..16791e2c3691 100644 --- a/lib/ssl/test/ssl_session_ticket_SUITE.erl +++ b/lib/ssl/test/ssl_session_ticket_SUITE.erl @@ -20,6 +20,8 @@ -module(ssl_session_ticket_SUITE). +-behaviour(ct_suite). + %% Callback functions -export([all/0, groups/0, diff --git a/lib/ssl/test/ssl_sni_SUITE.erl b/lib/ssl/test/ssl_sni_SUITE.erl index 74200c7ab00b..9e2f9fddb1e7 100644 --- a/lib/ssl/test/ssl_sni_SUITE.erl +++ b/lib/ssl/test/ssl_sni_SUITE.erl @@ -21,6 +21,8 @@ -module(ssl_sni_SUITE). +-behaviour(ct_suite). + -include_lib("common_test/include/ct.hrl"). -include_lib("public_key/include/public_key.hrl"). -include_lib("kernel/include/inet.hrl"). diff --git a/lib/ssl/test/ssl_socket_SUITE.erl b/lib/ssl/test/ssl_socket_SUITE.erl index a6bce2a414ae..e10ec5afafa1 100644 --- a/lib/ssl/test/ssl_socket_SUITE.erl +++ b/lib/ssl/test/ssl_socket_SUITE.erl @@ -20,6 +20,8 @@ -module(ssl_socket_SUITE). +-behaviour(ct_suite). + -include_lib("common_test/include/ct.hrl"). -include_lib("public_key/include/public_key.hrl"). diff --git a/lib/ssl/test/ssl_test_lib.erl b/lib/ssl/test/ssl_test_lib.erl index 03c8c710ba7c..fc5291b326fe 100644 --- a/lib/ssl/test/ssl_test_lib.erl +++ b/lib/ssl/test/ssl_test_lib.erl @@ -21,6 +21,8 @@ %% -module(ssl_test_lib). +-behaviour(ct_suite). + -include_lib("common_test/include/ct.hrl"). -include_lib("public_key/include/public_key.hrl"). diff --git a/lib/ssl/test/ssl_upgrade_SUITE.erl b/lib/ssl/test/ssl_upgrade_SUITE.erl index b258b9b05709..cc5db6978873 100644 --- a/lib/ssl/test/ssl_upgrade_SUITE.erl +++ b/lib/ssl/test/ssl_upgrade_SUITE.erl @@ -19,6 +19,8 @@ %% -module(ssl_upgrade_SUITE). +-behaviour(ct_suite). + -include_lib("common_test/include/ct.hrl"). %% Common test