Implementation Agnostic Testing

decisions/0007-implementation-agnostic-tests.md

@@ -0,0 +1,80 @@
+# Implementation-agnostic Testing and Test Harness
+
+Date: 10/06/2023
+
+## Status
+
+Proposed
+
+## Context
+
+In order to support the development of OSCAL tooling, it was decided prototype a unified tool responsible for validating OSCAL implementations against specification requirements.
+
+Currently, only profile resolution has been [formalized into a draft specification](../src/specifications/profile-resolution/profile-resolution-specml.xml).
+
+### Existing Infrastructure
+
+The profile resolver specification currently leverages an in-house XML format known as SpecML, which breaks down a specification into a collection of **sections**, which contain in turn a collection of **requirements**.
+Each `<section/>` and `<requirement/>` has a unique `@id` attribute.
+
+The sections and requirements are mirrored in the XSLT implementation's profile resolution unit tests.
+Although crucial to the XSLT implementation, these tests are not portable and it would not be simple to use the tests in their current state to validate other implementations.
+
+### Specification Tests
+
+Some specifications such as [CommonMark](https://commonmark.org/) include a [test suite and testing harness](https://github.com/commonmark/commonmark-spec/tree/master/test) to make it possible for implementors to "score" their implementation's conformance to the specification.
+
+## Decision
+
+### SpecML
+
+The specification format will remain unchanged for now.
+There is an argument for the format to be replaced or simplified in the future, but the use of `@id` attributes for sections and requirements make linking a test to a example simple.
+
+### Test Suite Data Format
+
+The test suite will be described using a JSON file with a simple data format.
+
+This file will contain a collection of objects that map to a given spec requirement via `section_id` and `requirement_id` fields.
+These objects will further contain a collection of "scenario" objects, each of which containing a `description`, `source_profile_path`, `expected_catalog_path`, and a collection of `selection_expressions`.
+
+For a given scenario, a test runner would be expected to perform profile resolution with the `source_profile_path` and compare selections of the resulting document with the `expected_catalog_path`.
+The `selection_expressions` are XPath expressions, though the [test harness](#test-harness) may further constrain the XPath expression's capabilities.
+
+Here is an example test suite made up of one requirement:
+
+```json
+[
+    {
+        "section_id": "import",
+        "requirement_id": "req-uri-resolve",
+        "scenarios": [
+            {
+                "description": "Check that group and control titles match, signalling that URIs have been resolved",
+                "source_profile_path": "requirement-tests/req-include-all-asis.xml",
+                "expected_catalog_path": "requirement-tests/output-expected/req-include-all-asis_RESOLVED.xml",
+                "selection_expressions": [
+                    "./oscal:group/oscal:title",
+                    "./oscal:group/oscal:control/oscal:title"
+                ]
+            }
+        ]
+    }
+]
+```
+
+The development of a JSON schema for this format is left as future work.
+
+### Test Harness
+
+A prototype testing harness has been developed, with the capability to report a given profile resolver's compliance to a specification given a [test suite JSON file](#test-suite-data-format).
+
+The prototype harness is built to be as simple as possible, avoiding external libraries.
+Python's native XPath capabilities are limited, further constraining the capabilities of the test suite.
+
+## Consequences
+
+Writing specification tests for profile resolution will require significant resources, but will make profile resolution more approachable for implementors and will make changes to the specification more maintainable.
+
+Due to the "requirement based" approach of the specification test suite, new tests can be added gradually.
+Test coverage can be determined by determining which requirements do not have tests.

src/specifications/profile-resolution/profile-resolution-specml-requirements.xspec

@@ -132,7 +132,7 @@
    </x:scenario>
    <x:scenario label="6.1.5 Multiple imports | Note that this occurs even if the same catalog is imported multiple times: each distinct import collects controls into a separate selection">
       <x:scenario label="Example req-chained-deepA.xml - PENDING&#xA;              chained profiles"
-                  pending="chained profiles"><?requirement rq-mulitple-imports ?>
+                  pending="chained profiles"><?requirement rq-multiple-imports ?>
          <x:context href="requirement-tests/req-chained-deepA.xml"/>
          <x:expect label="Resolution of req-chained-deepA.xml"
                    select="opr:scrub(.)"

src/specifications/profile-resolution/profile-resolution-specml.xml

@@ -412,7 +412,7 @@ profile:
       <section id="URI-multiple">
         <head>Multiple imports</head>
         <p>Each import directive is processed to produce a set of controls. <req
-            id="rq-mulitple-imports">Note that this occurs even if the same catalog is imported
+            id="rq-multiple-imports" level="must">Note that this occurs even if the same catalog is imported
             multiple times: each distinct import collects controls into a separate
               <int>selection</int></req>: </p>
         <tagging whose="source_profile">
@@ -445,7 +445,7 @@ intermediate:
           - ac-3
           - ac-4 
         </tagging>
-        <p><req id="rq-multiple-merge">The control inclusions are combined and collapsed in the next
+        <p><req id="rq-multiple-merge" level="must">The control inclusions are combined and collapsed in the next
             phase of processing</req>, <term>merge</term>(see <xref rid="merge-phase"/>) . </p>
         <p>Multiple imports against the same resource are allowed, and would most commonly occur when the profile author is using <xref rid="mapping"/> to create very specific output. 
           Multiple imports may result in outputs with clashing control IDs if mapping or the merge directive is not set correctly.</p>

src/specifications/profile-resolution/profile-resolution-unit-tests.xml

@@ -87,7 +87,7 @@
          <addresses-requirement requirement-id="req-resolve-profile">When a profile imports a profile,
             the subordinate profile SHOULD be resolved first into a catalog using this
             specification, before it is imported. </addresses-requirement>
-         <addresses-requirement requirement-id="rq-mulitple-imports">Note that this occurs even if the same catalog is imported
+         <addresses-requirement requirement-id="rq-multiple-imports">Note that this occurs even if the same catalog is imported
             multiple times: each distinct import collects controls into a separate
               selection</addresses-requirement>
          <addresses-requirement requirement-id="rq-multiple-merge">The control inclusions are combined and collapsed in the next

src/specifications/profile-resolution/resolution-testing.xml

@@ -67,7 +67,7 @@
       <statement>If a processor encounters a circular import as described above (self-imports are inherently circular), the processor MUST cease processing and generate an error. </statement>
       <test file="requirement-tests/req-circular_import.xml" status="pending">PENDING circular import detection</test>
    </requirement>
-   <requirement id="rq-mulitple-imports">
+   <requirement id="rq-multiple-imports">
       <statement>Note that this occurs even if the same catalog is imported multiple times: each distinct import collects controls into a separate selection</statement>
       <test file="requirement-tests/req-chained-deepA.xml" status="pending">PENDING chained profiles</test>
    </requirement>