Correctly pruning unused definitions in metaschema composition (usnis…

…tgov#199)

* Metaschema composition XSpecs
* Adding 'a8' prune-unused-definitions filter for comparison (most advanced yet).
* Added new prune filter with slight adjustments to interface; new test for running it standalone
* Removing noisy XSpec result
* More tuning up v9 pruning step for usnistgov#198
* Cleanup; restoring top-level compose XSLT
* Updating files before merge
* First draft of testing approach with current WIP prune phase transform.
* Fix tests to align with Wendell exception msg check.
* Move pruning tests back to their own folder.
* Refactor test input XML docs into separate files
* Make the tests more reusable across implementations.
* Relocate input files to subdir per convo with @wendellpiez.
* Slight improvement to prototype
* Much cleanup and rearrangement; readme docs
* Updating readme plus entry (shell) XSLT and XProc to current
* Removing obsolete XSpec reports
* Adding .gitignore to exclude XSpec results
* More rearrangement and cleanup; adding step 4 XSpec
* New filter detecting and removing unused definitions passes unit tests
* Removing early copy of remedied step 4 XSLT
* Updating Metaschema Schematron XSpecs to functional state: they are now stable, addressing #47
* More adjustments prepping to M4 unit tests - usnistgov#81 usnistgov#186 #201 - now everything is green or grey (pending) no pink
* Added more clarification to pruning XSpec
* Edits in response to feedback from @aj-stein-nist thanks AJ!
* adding blank lines at end of files

Co-authored-by: Alexander Stein <alexander.stein@nist.gov>
Co-authored-by: David Waltermire <david.waltermire@nist.gov>

test-suite/README.md

@@ -8,17 +8,15 @@ An example metaschema for mocking up documentation generation.
 
 ### `schema-generation`
 
+Unit testing over handmade mini metaschemas. Due for maintenance.
+
 To use: under Bash, run ./run-tests.sh in folder schema-generation
 
 ./run-tests --help gives help.
 
 ./run-tests.sh [testdir] runs tests only in a particular test directory.
 
-### `schema-generation`
-
-Unit testing over handmade mini metaschemas. Due for maintenance.
-
-### `worked-exaples`
+### `worked-examples`
 
 With one more testing metaschemas exercising different features. 
 

test-suite/metaschema-xspec/composition/expected/nestedmessy_metaschema-composed.xml

@@ -0,0 +1,151 @@
+<METASCHEMA xmlns="http://csrc.nist.gov/ns/oscal/metaschema/1.0"
+            abstract="no"
+            module="nestedmessy"
+            _base-uri="...">
+   <INFO info-type="objects-used">Seeing assembly#nestedmessy:box, assembly#nestedmessy:envelope, field#nestedmessy:page, flag#nestedmessy:no, assembly#nestedmessy:folder</INFO>
+   <schema-name>Nested Reference Test</schema-name>
+   <schema-version>1.0</schema-version>
+   <short-name>nestedmessy</short-name>
+   <namespace>http://csrc.nist.gov/metaschema/ns/nested</namespace>
+   <json-base-uri>http://csrc.nist.gov/metaschema/ns/nested</json-base-uri>
+   <remarks>
+      <p>If any 'unused' definition appears in a schema, we have a bug.</p>
+   </remarks>
+   <EXCEPTION problem-type="unused-definition">REMOVING unused assembly definition for 'unused-assembly' from nestedmessy.</EXCEPTION>
+   <define-field _metaschema-xml-id="/field/nestedmessy/page"
+                 _metaschema-json-id="/field/nestedmessy/page"
+                 collapsible="no"
+                 as-type="string"
+                 scope="global"
+                 name="page"
+                 module="nestedmessy"
+                 _base-uri="..."
+                 _key-name="nestedmessy:page">
+      <formal-name>PAGE</formal-name>
+      <description>PAGE</description>
+      <flag _step="no"
+            _key="no"
+            _metaschema-xml-id="/field/nestedmessy/page/no"
+            _metaschema-json-id="/field/nestedmessy/page/no"
+            as-type="positiveInteger"
+            required="no"
+            ref="no"
+            _key-ref="nestedmessy:no"
+            _using-name="no"
+            _in-xml-name="no"
+            _in-json-name="no"/>
+   </define-field>
+   <define-flag _metaschema-xml-id="/flag/nestedmessy/no"
+                _metaschema-json-id="/flag/nestedmessy/no"
+                as-type="positiveInteger"
+                scope="global"
+                name="no"
+                module="nestedmessy"
+                _base-uri="..."
+                _key-name="nestedmessy:no"/>
+   <EXCEPTION problem-type="unused-definition">REMOVING unused field definition for 'unused-field' from nestedmessy.</EXCEPTION>
+   <define-assembly _metaschema-xml-id="/assembly/nestedmessy/envelope"
+                    _metaschema-json-id="/assembly/nestedmessy/envelope"
+                    scope="global"
+                    name="envelope"
+                    module="nestedmessy"
+                    _base-uri="..."
+                    _key-name="nestedmessy:envelope">
+      <formal-name>ENVELOPE</formal-name>
+      <description>ENVELOPE</description>
+      <model>
+         <field _step="page"
+                _key="page"
+                _metaschema-xml-id="/assembly/nestedmessy/envelope/page"
+                _metaschema-json-id="/assembly/nestedmessy/envelope/page"
+                max-occurs="unbounded"
+                min-occurs="0"
+                ref="page"
+                _key-ref="nestedmessy:page"
+                _using-name="page"
+                _in-xml-name="page"
+                _in-json-name="page"/>
+      </model>
+   </define-assembly>
+   <define-assembly _metaschema-xml-id="/assembly/nestedmessy/box"
+                    _metaschema-json-id="/assembly/nestedmessy/box"
+                    scope="global"
+                    name="box"
+                    module="nestedmessy"
+                    _base-uri="..."
+                    _key-name="nestedmessy:box"
+                    _using-root-name="BOX">
+      <formal-name>BOX</formal-name>
+      <description>BOX</description>
+      <root-name>BOX</root-name>
+      <model>
+         <assembly _step="envelope"
+                   _key="envelope"
+                   _metaschema-xml-id="/assembly/nestedmessy/box/envelope"
+                   _metaschema-json-id="/assembly/nestedmessy/box/envelope"
+                   max-occurs="unbounded"
+                   min-occurs="0"
+                   ref="envelope"
+                   _key-ref="nestedmessy:envelope"
+                   _using-name="envelope"
+                   _in-xml-name="envelope"
+                   _in-json-name="envelope"/>
+         <assembly _step="folder"
+                   _key="folder"
+                   _metaschema-xml-id="/assembly/nestedmessy/box/folder"
+                   _metaschema-json-id="/assembly/nestedmessy/box/folder"
+                   max-occurs="unbounded"
+                   min-occurs="0"
+                   ref="folder"
+                   _key-ref="nestedmessy:folder"
+                   _using-name="folder"
+                   _in-xml-name="folder"
+                   _in-json-name="folder"/>
+      </model>
+   </define-assembly>
+   <define-assembly _metaschema-xml-id="/assembly/nestedmessy/folder"
+                    _metaschema-json-id="/assembly/nestedmessy/folder"
+                    scope="global"
+                    name="folder"
+                    module="nestedmessy"
+                    _base-uri="..."
+                    _key-name="nestedmessy:folder">
+      <formal-name>FOLDER</formal-name>
+      <description>FOLDER</description>
+      <model>
+         <assembly _step="envelope"
+                   _key="envelope"
+                   _metaschema-xml-id="/assembly/nestedmessy/folder/envelope"
+                   _metaschema-json-id="/assembly/nestedmessy/folder/envelope"
+                   max-occurs="unbounded"
+                   min-occurs="0"
+                   ref="envelope"
+                   _key-ref="nestedmessy:envelope"
+                   _using-name="envelope"
+                   _in-xml-name="envelope"
+                   _in-json-name="envelope"/>
+         <assembly _step="folder"
+                   _key="folder"
+                   _metaschema-xml-id="/assembly/nestedmessy/folder/folder"
+                   _metaschema-json-id="/assembly/nestedmessy/folder/folder"
+                   max-occurs="unbounded"
+                   min-occurs="0"
+                   ref="folder"
+                   _key-ref="nestedmessy:folder"
+                   _using-name="folder"
+                   _in-xml-name="folder"
+                   _in-json-name="folder"/>
+         <field _step="page"
+                _key="page"
+                _metaschema-xml-id="/assembly/nestedmessy/folder/page"
+                _metaschema-json-id="/assembly/nestedmessy/folder/page"
+                max-occurs="unbounded"
+                min-occurs="0"
+                ref="page"
+                _key-ref="nestedmessy:page"
+                _using-name="page"
+                _in-xml-name="page"
+                _in-json-name="page"/>
+      </model>
+   </define-assembly>
+</METASCHEMA>

test-suite/metaschema-xspec/composition/input/nestedmessy_metaschema.xml

@@ -0,0 +1,64 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<?xml-stylesheet type="text/css" href="../../../toolchains/xslt-M4/lib/metaschema-author.css"?>
+<?xml-model href="../validate/metaschema-composition-check.sch" type="application/xml" schematypens="http://purl.oclc.org/dsdl/schematron"?>
+<METASCHEMA xmlns="http://csrc.nist.gov/ns/oscal/metaschema/1.0">
+    <schema-name>Nested Reference Test</schema-name>
+    <schema-version>1.0</schema-version>
+    <short-name>nestedmessy</short-name>
+    <namespace>http://csrc.nist.gov/metaschema/ns/nested</namespace>
+    <json-base-uri>http://csrc.nist.gov/metaschema/ns/nested</json-base-uri>
+
+    <remarks>
+        <p>If any 'unused' definition appears in a schema, we have a bug.</p>
+    </remarks>
+
+    <define-assembly name="unused-assembly">
+        <formal-name>Unused</formal-name>
+        <description>Unused</description>
+        <model>
+            <field ref="unused-field"/>
+        </model>
+    </define-assembly>
+
+    <define-field name="page">
+        <formal-name>PAGE</formal-name>
+        <description>PAGE</description>
+        <flag ref="no"/>
+    </define-field>
+
+    <define-flag name="no" as-type="positiveInteger"/>
+
+    <define-field name="unused-field">
+        <formal-name>Unused field</formal-name>
+        <description>Unused field</description>
+    </define-field>
+
+    <define-assembly name="envelope">
+        <formal-name>ENVELOPE</formal-name>
+        <description>ENVELOPE</description>
+        <model>
+            <field ref="page" max-occurs="unbounded"/>
+        </model>
+
+    </define-assembly>
+
+    <define-assembly name="box">
+        <formal-name>BOX</formal-name>
+        <description>BOX</description>
+        <root-name>BOX</root-name>
+        <model>
+            <assembly ref="envelope" max-occurs="unbounded"/>
+            <assembly ref="folder"   max-occurs="unbounded"/>
+        </model>
+    </define-assembly>
+
+    <define-assembly name="folder">
+        <formal-name>FOLDER</formal-name>
+        <description>FOLDER</description>
+        <model>
+            <assembly ref="envelope" max-occurs="unbounded"/>
+            <assembly ref="folder"   max-occurs="unbounded"/>
+            <field ref="page" max-occurs="unbounded"/>
+        </model>
+    </define-assembly>
+</METASCHEMA>

test-suite/metaschema-xspec/composition/input/pruning001.xml

@@ -0,0 +1,9 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<METASCHEMA xmlns="http://csrc.nist.gov/ns/oscal/metaschema/1.0"
+    module="pruning001">
+    <define-assembly name="box">
+        <formal-name>BOX</formal-name>
+        <description>Just a box, no relationship to anything else.</description>
+        <root-name>BOX</root-name>
+    </define-assembly>
+</METASCHEMA>

test-suite/metaschema-xspec/composition/input/pruning002.xml

@@ -0,0 +1,16 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<METASCHEMA xmlns="http://csrc.nist.gov/ns/oscal/metaschema/1.0"
+    module="pruning002">
+    <define-assembly name="box">
+        <formal-name>BOX</formal-name>
+        <description>Just a box, no relationship to anything else.</description>
+        <root-name>BOX</root-name>
+        <model>
+            <assembly ref="folder"/>
+        </model>
+    </define-assembly>
+    <define-assembly name="folder">
+        <formal-name>Folder</formal-name>
+        <description>One or more folders belong in a box.</description>
+    </define-assembly>
+</METASCHEMA>

test-suite/metaschema-xspec/composition/input/pruning003.xml

@@ -0,0 +1,14 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<METASCHEMA xmlns="http://csrc.nist.gov/ns/oscal/metaschema/1.0" 
+    module="pruning003">
+    <define-assembly name="box">
+        <formal-name>BOX</formal-name>
+        <description>Just a box, no relationship to anything else.</description>
+        <root-name>BOX</root-name>
+        <model/>
+    </define-assembly>
+    <define-assembly name="folder">
+        <formal-name>Folder</formal-name>
+        <description>One or more folders belong in a box.</description>
+    </define-assembly>
+</METASCHEMA>

test-suite/metaschema-xspec/composition/metaschema-composition.xspec

@@ -0,0 +1,20 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<x:description xmlns:x="http://www.jenitennison.com/xslt/xspec"
+               stylesheet="../../../toolchains/xslt-M4/nist-metaschema-COMPOSE.xsl"
+               run-as="external"
+               xmlns="http://csrc.nist.gov/ns/oscal/metaschema/1.0"
+               xmlns:m="http://csrc.nist.gov/ns/oscal/metaschema/1.0"
+               xmlns:ov="http://csrc.nist.gov/ns/oscal/test/variable">
+
+   <x:helper stylesheet="../metaschema-test-helper.xsl"/>
+
+   <x:scenario label="Metaschema with unused definitions">
+      <!-- Scrubbing both the result and the expected result for comparison -->
+      <x:context href="input/nestedmessy_metaschema.xml"/>
+      <x:variable name="ov:expected-output"
+         href="expected/nestedmessy_metaschema-composed.xml"/>
+      <x:expect label="Base test - fully resolved"
+         test="m:scrub($x:result)" select="m:scrub($ov:expected-output)"/>
+   </x:scenario>
+
+</x:description>

test-suite/metaschema-xspec/composition/metaschema-prune-unused-definitions.xspec

@@ -0,0 +1,41 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<x:description 
+    xmlns:xs="http://www.w3.org/2001/XMLSchema"
+    xmlns:x="http://www.jenitennison.com/xslt/xspec"
+    xmlns:m="http://csrc.nist.gov/ns/oscal/metaschema/1.0"
+    xmlns="http://csrc.nist.gov/ns/oscal/metaschema/1.0"
+    stylesheet="../../../toolchains/xslt-M4/nist-metaschema-COMPOSE.xsl"
+    run-as="external">
+    <x:scenario label="In Metaschema composition pruning phase">
+        <x:scenario label="during the pruning phase">
+            <x:scenario label="a single root assembly with no references">
+                <x:context href="./input/pruning001.xml"/>
+                <x:expect label="should not throw an exception"
+                    test="$x:result//m:EXCEPTION => empty()"/>
+            </x:scenario>
+            <x:scenario label="top-level definitions with correct references.">
+                <x:context href="./input/pruning002.xml"/>
+                <x:expect label="should not throw an exception"
+                    test="$x:result//m:EXCEPTION => empty()"/>
+            </x:scenario>
+            <x:scenario label="top-level definitions with an unused definition.">
+                <x:context href="./input/pruning003.xml"/>                
+                <x:expect label="throws only exception for an unused definition."
+                    test="count($x:result//m:EXCEPTION) eq 1"/>
+                <x:expect label="and should be of type unused-definition with the proper message."
+                    test="$x:result//m:EXCEPTION[@problem-type = 'unused-definition'] => exists()"/>
+                <x:variable name="m:ec" as="xs:string">REMOVING unused assembly definition for 'folder' from pruning003.</x:variable>
+                <x:expect label="and the error message indicates the specific definition that's unused."
+                    test="$x:result//m:EXCEPTION[@problem-type='unused-definition'] = $m:ec"/>
+            </x:scenario>
+            <x:scenario label="top-level definitions with an unused definition and an orphan (not used by anything used).">
+                <x:context href="./input/pruning004.xml"/>                
+                <x:expect label="throws two exceptions for unused definitions"
+                    test="count($x:result//m:EXCEPTION) eq 2"/>
+                <x:expect label="and has neither unused model defined."
+                    test="($x:result//m:define-assembly[@name='folder'] |
+                           $x:result//m:define-field[@name='page']) => empty()"/>
+            </x:scenario>
+        </x:scenario>
+    </x:scenario>
+</x:description>

test-suite/metaschema-xspec/composition/readme.md

@@ -0,0 +1,15 @@
+# Metaschema processing XSpec testing - Metaschema Composition end-to-end (E2E)
+
+To run XSpec, we suggest either an XML IDE or a command line tool as described in [XSpec documentation in Github](https://github.com/xspec/xspec/wiki).
+
+* `composition/metaschema-composition.xspec`
+
+Tests XSLT `../../../toolchains/xslt-M4/nist-metaschema-COMPOSE.xsl`
+
+Currently checks if unused definitions are correctly filtered away from a metaschema source instance, in composition.
+
+(This prevents result schemas from including unused models.)
+
+* `composition/metaschema-prune-unused-definitions.xspec`
+
+Also testing the same bug, but using standoff instances, with a start on more comprehensive tests.