feat: Improve error reporting in test case execution (#111)

Co-authored-by: Julien THOMAS <external.julien.thomas@de.bosch.com>

CHANGELOG.md

@@ -9,10 +9,41 @@ This Changelog can be updated by calling:
 invoke changelog
 ```
 
-## Version 0.18.0 (2022-06-07)
+## Version Unreleased (2022-07-11)
 
 #### New Features
 
+* agnostic tags
+* make DuT DT-aux capable ([#77](https://github.com/eclipse/kiso-testing/issues/77))
+* Implement double threaded auxiliary interface ([#74](https://github.com/eclipse/kiso-testing/issues/74))
+* detect test collection errors and abort execution
+* add show tag script for test information analysis
+#### Fixes
+
+* multiple yaml logging
+#### Refactorings
+
+* adapt acroname aux to dt interface ([#84](https://github.com/eclipse/kiso-testing/issues/84))
+* migrate recorder to DT-aux interface ([#80](https://github.com/eclipse/kiso-testing/issues/80))
+#### Docs
+
+* add tools section and remove tools from coverage
+#### Others
+
+* rename show-tag CLI to pykiso-tags
+* fix docstring
+* fix issues induced by tmp_path fixture
+* try to fix get_yaml_files test
+* increase coverage
+* add tests for show-tag
+
+Full set of changes: [`0.18.0...0ee4fb2`](https://github.com/eclipse/kiso-testing/compare/0.18.0...0ee4fb2)
+
+## Version 0.18.0 (2022-06-08)
+
+#### New Features
+
+* make release 0.18.0 ([#72](https://github.com/eclipse/kiso-testing/issues/72))
 * create base class for test ([#61](https://github.com/eclipse/kiso-testing/issues/61))
 * make the proxy agnostic of transitioning messages
 * add pykiso to pytest tool ([#62](https://github.com/eclipse/kiso-testing/issues/62))

README.md

@@ -114,9 +114,10 @@ Once installed the application is bound to `pykiso`, it can be called with the f
 ```bash
 Usage: pykiso [OPTIONS] [PATTERN]
 
-  Embedded Integration Test Framework - CLI Entry Point.
+  TAG Filters: Just add pairs of arguments to the pykiso call. Multiple values
+  for a key must be seperate via a comma.
 
-  PATTERN: overwrite the test filter pattern from the YAML file (optional)
+  For example: pykiso -c your_config.yaml --branch dev,master --variant delta
 
 Options:
   -c, --test-configuration-file FILE
@@ -130,10 +131,6 @@ Options:
                                   set the verbosity of the logging
   --version                       Show the version and exit.
   -h, --help                          Show this message and exit.
-  --variant                       allow the user to execute a subset of tests
-                                  based on variants
-  --branch-level                  allow the user to execute a subset of tests
-                                  based on branch levels
   --junit                         enables the generation of a junit report
   --text                          default, test results are only displayed in
                                   the console

docs/getting_started/config_file.rst

@@ -85,11 +85,11 @@ The test suite definition is a list of key-value pairs.
 Each test suite consists of a `test_suite_id`, a `suite_dir` and a `test_filter_pattern`.
 
 For fast test development, the `test_filter_pattern` can be overwritten from the command line
-in order to e.g. execute a single test file inside the `suite_dir` using the CLI argument `PATTERN`:
+in order to e.g. execute a single test file inside the `suite_dir` using the CLI argument `-p` or `--pattern`:
 
 ..code:: bash
 
-    pykiso -c dummy.yaml test_suite_1.py
+    pykiso -c dummy.yaml -p test_suite_1.py
 
 
 Requirements specification

docs/introduction/integration_test.rst

@@ -184,7 +184,7 @@ an additional reference can be added into the test_run decorator:
 
 In order to run only a subset of tests, an additional reference can be added to the test_run decorator:
 
-- variant : optional test variants can be defined like:
+- tag : [optional] the variant can be defined like:
 
 .. code:: python
 
@@ -193,12 +193,27 @@ In order to run only a subset of tests, an additional reference can be added to
 Both parameters (variant/branch_level), will play the role of filter to fine
 tune the test collection and at the end ensure the execution of very specific tests subset.
 
-.. note:: branch_level parameter is also part of the CLI and both (variant/branch_level)
-    accept multiple values.
+.. note:: cli tags must be given in pairs. If one key has multiple values seperate them with a comma
 
 .. code:: bash
 
-    pykiso -c configuration_file --variant var1 --variant var2 --branch-level daily --branch-level nightly
+    pykiso -c configuration_file --variant var1,var2 --branch-level daily,nightly
+
+.. table:: Exectuion table for test case tags and cli tag arguments
+   :widths: auto
+
+   =======================================================  ===============================  ========
+   test case tags                                           cli tags                         executed
+   =======================================================  ===============================  ========
+   "branch_level": ["daily","nightly"]                      branch_level nightly             🗸
+   "branch_level": ["daily","nightly"]                      branch_level nightly,daily       🗸
+   "branch_level": ["daily","nightly"]                      branch_level master              ✗
+   "branch_level": ["daily","nightly"],"variant":["var1"]   branch_level nightly             🗸
+   "branch_level": ["daily","nightly"],"variant":["var1"]   variant var1                     🗸
+   "branch_level": ["daily","nightly"],"variant":["var1"]   variant var2                     ✗
+   "branch_level": ["daily","nightly"],"variant":["var1"]   branch_level daily variant var1  ✗
+   =======================================================  ===============================  ========
+
 
 In order to utilise the SetUp/TearDown test-suite feature, users have to define a class inheriting from
 :py:class:`~pykiso.test_coordinator.test_suite.BasicTestSuiteSetup` or

docs/whats_new/version_ongoing.rst

@@ -52,3 +52,23 @@ The collected messages by the Communication auxiliary can still be cleared with
 :py:meth`~pykiso.lib.auxiliaries.communication_auxiliary.CommunicationAuxiliary.clear_buffer`
 
 See :ref:`communication_auxiliary`
+
+DUT Auxiliary adaption
+^^^^^^^^^^^^^^^^^^^^^^
+refactor/redesign of the device under test auxiliary to fit with the brand new double
+threaded auxiliary interface
+
+Record Auxiliary adaption
+^^^^^^^^^^^^^^^^^^^^^^^^^
+adapt the record auxiliary to fit with the brand new double threaded auxiliary interface
+
+Acroname Auxiliary adaption
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+adapt the acroname auxiliary to fit with the brand new double threaded auxiliary interface
+
+Agnostic tag call
+^^^^^^^^^^^^^^^^^
+Instead of having only the 2 tags "variant" and "branch_level" to select tests, users
+can now set any tagname.
+
+See: :ref:`define_test_information` for more details.

pyproject.toml

@@ -99,7 +99,7 @@ omit = [
 ]
 
 [tool.coverage.report]
-exclude_lines = ["pass", "def __repr__"]
+exclude_lines = ["pass", "def __repr__" ,"pragma: no cover","if TYPE_CHECKING:"]
 
 [build-system]
 requires = ["poetry-core>=1.0.0"]

src/pykiso/cli.py

@@ -20,16 +20,17 @@
 
 """
 import collections
-import itertools
 import logging
 import pprint
 import sys
 import time
 from pathlib import Path
-from typing import NamedTuple, Optional, Tuple
+from typing import Dict, List, NamedTuple, Optional, Tuple
 
 import click
 
+from pykiso.test_coordinator.test_execution import ExitCode
+
 from . import __version__
 from .config_parser import parse_config
 from .global_config import Grabber
@@ -39,6 +40,8 @@
 
 LogOptions = collections.namedtuple("LogOptions", "log_path log_level report_type")
 
+click.UsageError.exit_code = ExitCode.BAD_CLI_USAGE
+
 # use to store the selected logging options
 log_options: Optional[NamedTuple] = None
 
@@ -119,7 +122,47 @@ def get_logging_options() -> LogOptions:
     return log_options
 
 
-@click.command(context_settings={"help_option_names": ["-h", "--help"]})
+def eval_user_tags(click_context: click.Context) -> Dict[str, List[str]]:
+    """Evaluate commandline args for user tags and raise exceptions for invalid
+    arguments.
+
+    :param click_context: click context
+    :raises click.NoSuchOption: if key doesnt start with "--" or has an invalid
+      character like "_"
+    :raises click.BadOptionUsage: no value specfied for user tag
+    :return: user tags with values
+    """
+    user_tags = {}
+    user_args = click_context.args.copy()
+    if not user_args:
+        return user_tags
+    while user_args:
+        try:
+            key = user_args.pop(0)
+            if not key.startswith("--") or "_" in key:
+                correct_key = (
+                    f'{"" if key.startswith("--") else "--" }{key.replace("_","-")}'
+                )
+                raise click.NoSuchOption(option_name=key, possibilities=[correct_key])
+
+            value = user_args.pop(0)
+            user_tags[key[2:]] = value.split(",")
+        except IndexError:
+            raise click.BadOptionUsage(
+                option_name=key,
+                message=f"No value specified for tag {key}",
+                ctx=click_context,
+            )
+    return user_tags
+
+
+@click.command(
+    context_settings={
+        "help_option_names": ["-h", "--help"],
+        "ignore_unknown_options": True,
+        "allow_extra_args": True,
+    }
+)
 @click.option(
     "-c",
     "--test-configuration-file",
@@ -161,43 +204,39 @@ def get_logging_options() -> LogOptions:
     default=True,
     help="default, test results are only displayed in the console",
 )
-@click.option(
-    "--variant",
-    multiple=True,
-    type=str,
-    default=None,
-    help="allow the user to execute a subset of tests based on variants",
-)
-@click.option(
-    "--branch-level",
-    multiple=True,
-    type=str,
-    default=None,
-    help="allow the user to execute a subset of tests based on branch levels",
-)
 @click.option(
     "--failfast",
     is_flag=True,
     help="stop the test run on the first error or failure",
 )
-@click.argument("pattern", required=False)
+@click.option(
+    "-p",
+    "--pattern",
+    type=click.STRING,
+    required=False,
+    help="test filter pattern, e.g. 'test_suite_1.py' or 'test_*.py'. It will be applied to all defined test suites.",
+)
 @click.version_option(__version__)
 @Grabber.grab_cli_config
+@click.pass_context
 def main(
+    click_context: click.Context,
     test_configuration_file: Tuple[PathType],
     log_path: PathType = None,
     log_level: str = "INFO",
     report_type: str = "text",
-    variant: Optional[tuple] = None,
-    branch_level: Optional[tuple] = None,
     pattern: Optional[str] = None,
     failfast: bool = False,
 ):
     """Embedded Integration Test Framework - CLI Entry Point.
 
-    PATTERN: overwrite the test filter pattern from the YAML file (optional)
+    TAG Filters: any additional option to be passed to the test as tag through
+    the pykiso call. Multiple values must be separated with a comma.
+
+    For example: pykiso -c your_config.yaml --branch-level dev,master --variant delta
 
     \f
+    :param click_context: click context
     :param test_configuration_file: path to the YAML config file
     :param log_path: path to an existing directory or file to write logs to
     :param log_level: any of DEBUG, INFO, WARNING, ERROR
@@ -218,8 +257,10 @@ def main(
 
         ConfigRegistry.register_aux_con(cfg_dict)
 
+        user_tags = eval_user_tags(click_context)
+
         exit_code = test_execution.execute(
-            cfg_dict, report_type, variant, branch_level, pattern, failfast
+            cfg_dict, report_type, user_tags, pattern, failfast
         )
         ConfigRegistry.delete_aux_con()
         for handler in logging.getLogger().handlers: