From aaec9f98bbff81727c284f3613e832ecc5a73371 Mon Sep 17 00:00:00 2001
From: Nikita Wootten <nikita.wootten@nist.gov>
Date: Fri, 6 Oct 2023 16:23:58 -0400
Subject: [PATCH] Added spec test adr and prototype spec test file

---
 .../0007-implementation-agnostic-tests.md     | 49 +++++++++++++++++++
 .../profile-resolution/spec-tests.json        |  7 +++
 2 files changed, 56 insertions(+)
 create mode 100644 decisions/0007-implementation-agnostic-tests.md
 create mode 100644 src/specifications/profile-resolution/spec-tests.json

diff --git a/decisions/0007-implementation-agnostic-tests.md b/decisions/0007-implementation-agnostic-tests.md
new file mode 100644
index 0000000000..4549439494
--- /dev/null
+++ b/decisions/0007-implementation-agnostic-tests.md
@@ -0,0 +1,49 @@
+# Implementation-agnostic Testing and Test Harness
+
+Date: 10/06/2023
+
+## Status
+
+Proposed
+
+## Context
+
+In order to support the development of OSCAL tooling, it was decided prototype a unified tool responsible for validating OSCAL implementations against specification requirements.
+
+Currently, only profile resolution has been [formalized into a draft specification](../src/specifications/profile-resolution/profile-resolution-specml.xml).
+
+### Existing Infrastructure
+
+The profile resolver specification currently leverages an in-house XML format known as SpecML, which breaks down a specification into a collection of **sections**, which contain in turn a collection of **requirements**.
+Each `<section/>` and `<requirement/>` has a unique `@id` attribute.
+
+The sections and requirements are mirrored in the XSLT implementation's profile resolution unit tests.
+Although crucial to the XSLT implementation, these tests are not portable and it would not be simple to use the tests in their current state to validate other implementations.
+
+### Specification Tests
+
+Some specifications such as [CommonMark](https://commonmark.org/) include a [test suite and testing harness](https://github.com/commonmark/commonmark-spec/tree/master/test) to make it possible for implementors to "score" their implementation's conformance to the specification.
+
+## Decision
+
+<!-- TODO -->
+
+### SpecML
+
+The specification format will remain unchanged for now.
+There is an argument for the format to be replaced or simplified in the future, but the use of `@id` attributes for sections and requirements make linking a test to a example simple.
+
+### Test Suite Data Format
+
+<!-- TODO -->
+
+### Test Harness
+
+<!-- TODO -->
+
+## Consequences
+
+Writing specification tests for profile resolution will require significant resources, but will make profile resolution more approachable for implementors and will make changes to the specification more maintainable.
+
+Due to the "requirement based" approach of the specification test suite, new tests can be added gradually.
+Test coverage can be determined by determining which requirements do not have tests.
diff --git a/src/specifications/profile-resolution/spec-tests.json b/src/specifications/profile-resolution/spec-tests.json
new file mode 100644
index 0000000000..1cdb1afe8b
--- /dev/null
+++ b/src/specifications/profile-resolution/spec-tests.json
@@ -0,0 +1,7 @@
+[
+    {
+        "section_id": "import",
+        "requirement_id": "req-uri-resolve",
+        "scenarios": []
+    }
+]
\ No newline at end of file