CTM Guidebook

The picture below shows all possible configuration elements for CTM:

1. Requirements

You can have your requirements as issues in one of the following tools

• Atlassian JIRA
• GitHub
• Enterprise GitHub

2. Tracing Methods

CTM provides two ways for connecting requirements with the test code. Either via parsing your source code for annotated/commented tests (language dependent) or via a json requirement mapping file (not language dependent)

Code Annotations

You can use a comment directly in your test code to connect it with specific requirements using the syntax below:

// Trace(<RequirementSystem>:<RequirementId>, <RequirementSystem>:<RequirementId>, ...)

Example:
Mapping requirement 13 from Jira project MYPROJECT and issue no. 42 from GitHub repository myOrg/myRepo to the following (Python) test method

   # Trace(Jira:MYPROJECT-13, GitHub:myOrg/myRepo#42)
   def test_isupper(self):
      # Test something meaningful here

Supported languages:

• Java: parser - example
• JavaScript: parser
• Python: parser - example
• Gauge Specs: parser - example

If you miss your language of choice, please use the requirement mapping file or write a parser on your own (we're welcoming pull-requests 😉)

Configuration setting:
In order to use code annotations, use the sourcecode setting in the CTM configuration

Requirement Mapping File

A json file containing a list of tests and the requirements they are traced to. A json entry consists of the following elements:

• source_reference: The full name of the test class or method

  Example:

  • Tracing the entire test class

  "source_reference": "myOrg.myRepo.myBigTest"

  • Tracing a single test method (test methods always have to end with empty brackets ())

   "source_reference": "myOrg.myRepo.myBigTest.myTestMethod()

• github_keys: A list of GitHub issues (requirements) that are mapped to the test

  Example:

   "github_keys": [
        "myOrg/myRepo#42",
        "myOrg/myRepo#43"
   ]

• jira_keys: A list of JIRA issues (requirements) that are mapped to the test

  Example:

   "jira_keys": [
        "MYJIRAPROJECT-1",
        "MYJIRAPROJECT-1"
   ]

• filelocation (optional): If provided the CTM will add links to the test source code into the HTML and json output reports

  Example:

    "filelocation": {
        "git": {
          "organization": "myOrg",
          "repository": "mySourcecodeRepo",
          "branch": "master"
        },
        "relativePath": "./myOrg/mySourcecodeRepo/myBigTest.java"
    }

  git contains the GitHub repository coordinates while relativePath contains the relative location of the automated test source code inside the GitHub repository.

See a complete json requirement mapping file example.

Configuration setting:
In order to use a requirement mapping file, use the mapping setting in the CTM configuration

3. Types of Test Results

CTM can connect only certain type of test results to your tests and requirements.
Supported types:

• xunit

Tell us what other types of test results are relevant for your project!

Configuration setting:
In order to set the local path to your test result, use the testReport setting in the CTM configuration

4. Configuration File

The json file that configures CTM. The following json elements are allowed:

• github: (optional) In order to point to the right (Enterprise) GitHub, CTM needs the GitHub base URL. Optional only for rare use cases e.g. using other source code systems. You might see broken links in reports. Also the traceability repository feature will not work.

  Example:

  "github": {
    "base_url": "https://github.com",
    "access_token": "<YOUR_GITHUB_ACCESSTOKEN>",
    "createLinksInBacklogItems": true
  }

  • base_url The base URL of your Enterprise GitHub instance or the public GitHub base URL

  • access_token (optional) Only required in case you want to use a traceability repository.
    The access token is used to create a GitHub release for each configured delivery. The scope of the access token must be (at least) repo.

  • createLinksInBacklogItems (optional) Only required in case you want to use a traceability repository.
    If set to true CTM will add a comment to the GitHub issue (requirement) which links to the corresponding entry in the traceability repository
    Ensure to also set the field access_token if you set createLinksInBacklogItems to true

• jira: (optional) In order to point to your JIRA project, CTM needs the JIRA base URL.

  Example:

  "jira": {
    "base_url": "https://myjira.myCompany.org",
    "basicAuth": {
            "user": "myJiraUser",
            "password": "myJiraPassword"
    },
    "createLinksInBacklogItems": true
  }

  • base_url The base URL of your JIRA instance

  • basicAuth (optional) Only required in case you want to use a traceability repository.
    The credentials are used to add a comment to the JIRA issue (requirement) which links to the corresponding entry in the traceability repository

  • createLinksInBacklogItems (optional) Only required in case you want to use a traceability repository.
    If set to true CTM will add a comment to the JIRA issue (requirement) which links to the corresponsing entry in the traceability repository
    Ensure to also set the field basicAuth if you set createLinksInBacklogItems to true

• workDir: A place where CTM will store its work

  Example:

  "workDir": "/tmp/ctm_workdir"

• outputDir: Location were CTM will write its HTML and json report files to.

  Example:

  "outputDir": "/tmp/ctm"

• log: CTM logging level.

  Example:

  "log": {
    "level": "INFO"
  }

  Possible values: INFO, WARNING, ERROR, FATAL

• mapping: (optional) The json requirement mapping file that maps tests to requirements.

  Example:

  "mapping": {
    "local": "/tmp/mapping.json"
  }

  ❗ Use either the mapping or sourcecode element in your configuration file, not both.

• sourcecode: (optional) Use this element if you want to use code annotations for mapping tests to requirements. Multiple source code locations can be given.

  Example:

  "sourcecode": [{
    "git": {
      "organization": "myOrg",
      "repository": "myRepo",
      "branch": "master"
    },
    "local": "/tmp/src/",
    "language": "java"
  }]

  • language Either gaugespec, java, javascript or python
  • local (optional) Local path to your source code. Either local or git must be given.
    If local is given, CTM will parse the source code from the local source
  • git (optional) GitHub repository that stores your source code. Either local or git must be given.
    If git is given, CTM will clone the GitHub repository into the workDir directory and parse the file from it. Also it will create a link in the traceability reports which points to the source code location at GitHub.

  If local and git are given and local already contains a clone of the git GitHub repository, CTM will call a git pull to ensure the source code is up to date and then parse the source code.
  If local and git are given and local is an empty folder, CTM will try to clone the git GitHub repository into the local location and parse it from there.

  ❗ Use either the mapping or sourcecode element in your configuration file, not both.

• testReport: Location (and corresponding type) of your test results. Multiple locations can be given.

  Example:

  "testReport": [{
    "type": "xunit-xml",
    "local": "/tmp/surefire-reports"
  }]   

  • type Currently only xunit-xml is supported
  • local Local path to your test results

• traceabilityRepo: (optional) GitHub repository (traceability repository) to store a continuous trace of your automated test metrics.

  "traceabilityRepo": {
    "git": {
      "organization": "myOrg",
      "repository": "myRepo",
      "branch": "master"
    }
  }

---

Below you can find examples of complete configuration files:

• Requirement mapping file configuration: Reporting on all requirements listed in the json requirement mapping file.
• Code annotations configuration: Reporting on all requirements annotated in a Java source code.

5. Delivery Mapping File (Optional)

You can select the requirements that are relevant for a certain delivery of your product and create a Delivery Traceability Report just for them.

• program: The name of your product.

  Example:

  "program": "My program name"

• delivery: The name of the delivery.

  Example:

  "delivery": "1.0.0"

• jira_keys: A list of the delivery requirements maintained in JIRA.

  Example:

  "jira_keys": [ 
	"MYJIRAPROJECT-1", 
   	"MYJIRAPROJECT-2", 
        "MYJIRAPROJECT-3" 
  ] 

• github_keys: A list of new requirements maintained in GitHub.

  Example:

  "github_keys": [ 
      "myOrg/mySourcecodeRepo#42", 
      "myOrg/mySourcecodeRepo#43"
  ]

Have a look at an example of a delivery mapping file to get started.

❗ You can choose not to create a separate delivery mapping file but rather pass its values directly as parameters as described below.

6. Parameters

The following command line paratemers can be pass to CTM. Some of the parameters specifiy configuration values which can also be set in the configuration file. In case a parameter is given via the command line and via the configuration file the value from the command line will overrule the value from the configuration file.

• -c <PATH_TO_CONFIG>: Path to your local configuration file

  Example: -c /tmp/myConfig.json

• -df <YOUR_DELIVERY_JSON_FILE> (optional): Path to your local delivery mapping file.

  Example: -df /tmp/delivery.json

• -sp <YOUR_PROGRAM_NAME> (optional): The name of your program.
  This value corresponds to the parameter program in the delivery mapping file

  Example: -sp My_Product

• -sd <YOUR_DELIVERY_NAME_OR_VERSION> (optional): The name of the delivery.
  This value corresponds to the parameter delivery in the delivery mapping file

  Example: -sd 1.2.0

• -bi <COMMA_SEPERATED_LIST_OF_REQUIREMENTS> (optional): List of requirements that are relevant for a certain delivery of your product.
  This value corresponds to the parameter github_keys and jira_keys in the delivery mapping file

  Format: <RequirementSystem>:<RequirementId>, ...

  Example: -bi GitHub:myOrg/myRepo#5, Jira:MYPROJECT-42

• -erm (optional): Export a requirement mapping file, if your requirement mapping was done via source code annotations, and you want to have all mappings exported to one requirement mapping file.

  Example: -erm

7. Overall Traceability Report

CTM creates a report which shows the test status of all requirements that have been annotated in the source code or mapped in the the requirement mapping file. The report is generated in two formats json and HTML and stored in the outputDir defined in the configuration file.
The report files are named:

• ctm_report_all.html
• ctm_report_all.json

Example:

• ctm_report_all.html

• ctm_report_all.json

{
    "myorg/myRepo#1": {
        "link": "https://github.com/myorg/myRepo/issues/1",
        "test_cases": [
            {
                "test_fullname": "simple.TestStringMethods.test_isupper",
                "test_name": "test_isupper",
                "test_class": "simple.TestStringMethods",
                "test_source": "https://github.com/myOrg/myRepo/blob/master/simple.py",
                "passed": true,
                "skipped": false
            }
        ]
    },
    "MYPROJECT-1": {
        "link": "https://myjira.myCompany.org/browse/MYPROJECT-1",
        "test_cases": [
            {
                "test_fullname": "simple.TestStringMethods.test_upper",
                "test_name": "test_upper",
                "test_class": "simple.TestStringMethods",
                "test_source": "https://github.com/myOrg/myRepo/blob/master/simple.py",
                "passed": true,
                "skipped": false
            },
            {
                "test_fullname": "simple.TestStringMethods.test_isupper",
                "test_name": "test_isupper",
                "test_class": "simple.TestStringMethods",
                "test_source": "https://github.com/myOrg/myRepo/blob/master/simple.py",
                "passed": true,
                "skipped": false
            },
            {
                "test_fullname": "simple.TestStringMethods.test_split",
                "test_name": "test_split",
                "test_class": "simple.TestStringMethods",
                "test_source": "https://github.com/myOrg/myRepo/blob/master/simple.py",
                "passed": true,
                "skipped": false
            }
        ]
    }
}

This report helps you understand what the impact of failing tests is on your business requirements as well as what areas of your functionality requires more testing.

8. Delivery Traceability Report (optional)

In case a delivery mapping file or the delivery parameters were provided, CTM additionally creates a second report (Delivery Traceability Report) which shows the test status of the selected requirements that have been marked for this release. This report is generated in two formats jsonand HTML and stored in the outputDir defined in the configuration file.
The report files are named:

• ctm_report_<delivery>.html
• ctm_report_<delivery>.json

<delivery> is taken from the corresponding value in the delivery mapping file or the delivery parameters

Example:

• ctm_report_1.42.0.html

• ctm_report_1.42.0.json

{
    "MYPROJECT-1": {
        "link": "https://myjira.myCompany.org/browse/MYPROJECT-1",
        "test_cases": [
            {
                "test_fullname": "simple.TestStringMethods.test_upper",
                "test_name": "test_upper",
                "test_class": "simple.TestStringMethods",
                "test_source": "https://github.com/myOrg/myRepo/blob/master/simple.py",
                "passed": true,
                "skipped": false
            },
            {
                "test_fullname": "simple.TestStringMethods.test_isupper",
                "test_name": "test_isupper",
                "test_class": "simple.TestStringMethods",
                "test_source": "https://github.com/myOrg/myRepo/blob/master/simple.py",
                "passed": true,
                "skipped": false
            },
            {
                "test_fullname": "simple.TestStringMethods.test_split",
                "test_name": "test_split",
                "test_class": "simple.TestStringMethods",
                "test_source": "https://github.com/myOrg/myRepo/blob/master/simple.py",
                "passed": true,
                "skipped": false
            }
        ]
    }
}

This report provides transparency on the amount and type of automated checks being provided for your new business requirements and aids in identifying possible gaps.

9. Traceability Repository

Motivation

The traceability repository allows you to continuously monitor the health of your product's requirements.
Test results often miss a link to the project's requirements so that it's cumbersome to understand which requirement is affected by a broken test.

Adding CTM to your Continuous Delivery pipeline will give the complete team an always up to date view on the quality status of your project's requirements, as CTM will constantly commit your automated test results into the traceability repository. Figure 9.1: Always up to date information on the automated test status of all product requirements

Requirements status

Figure 9.2: A folder is created in the traceability repository for each product requirement. The folder contains:

• a link to the requirement
• the current overall requirement test status
• a detailed list of all automated test results belonging to the requirement
• latest automated test result files (e.g. xunit test result XML files)

Delivery status

Figure 9.3: In case a delivery mapping file or the delivery parameters were provided to CTM, a folder is created for each delivery. The folder contains:

• a link to all requirements related to the delivery
• the overall test status per requirement
• a detailed list of all automated test results per requirement

GitHub release

Figure 9.4: In case a delivery mapping file or the delivery parameters were provided to CTM, a GitHub release is created for each delivery, containing the same information as the delivery page

Requirements

• An empty GitHub repository (on GitHub or on your Enterprise GitHub instance)
• A working git command line client which can be called by CTM and which has access to the above given repository

Linking the requirement to its automated test status

If you use one of the supported tools to store your requirements, CTM can be configured to add a link from a requirement to the automated test status in the traceability repository. This link will be included in a comment that CTM will add at the GitHub or Jira issue that represents the requirement.

Figure 9.5: (GitHub based) Requirement definition with CTM added link to traceability repository

Set the configuration parameter createLinksInBacklogItems under the github or jira element to true to enable the requirement link creation.

Please note, if enabled, CTM will create and maintain a file called .commentCache in the configured workDir. This file is used to track which requirements have already been linked. In case this file gets lost (e.g. because you deleted the workDir), CTM will add a new link to the requirement, which might causes doublicate links/comments.