Skip to content

Commit

Permalink
Correctly pruning unused definitions in metaschema composition (#199)
Browse files Browse the repository at this point in the history 
* Metaschema composition XSpecs
* Adding 'a8' prune-unused-definitions filter for comparison (most advanced yet).
* Added new prune filter with slight adjustments to interface; new test for running it standalone
* Removing noisy XSpec result
* More tuning up v9 pruning step for #198
* Cleanup; restoring top-level compose XSLT
* Updating files before merge
* First draft of testing approach with current WIP prune phase transform.
* Fix tests to align with Wendell exception msg check.
* Move pruning tests back to their own folder.
* Refactor test input XML docs into separate files
* Make the tests more reusable across implementations.
* Relocate input files to subdir per convo with @wendellpiez.
* Slight improvement to prototype
* Much cleanup and rearrangement; readme docs
* Updating readme plus entry (shell) XSLT and XProc to current
* Removing obsolete XSpec reports
* Adding .gitignore to exclude XSpec results
* More rearrangement and cleanup; adding step 4 XSpec
* New filter detecting and removing unused definitions passes unit tests
* Removing early copy of remedied step 4 XSLT
* Updating Metaschema Schematron XSpecs to functional state: they are now stable, addressing #47
* More adjustments prepping to M4 unit tests - #81 #186 #201 - now everything is green or grey (pending) no pink
* Added more clarification to pruning XSpec
* Edits in response to feedback from @aj-stein-nist thanks AJ!
* adding blank lines at end of files

Co-authored-by: Alexander Stein <alexander.stein@nist.gov>
Co-authored-by: David Waltermire <david.waltermire@nist.gov>
  • Loading branch information
@wendellpiez @aj-stein-nist @david-waltermire
3 people committed Jun 21, 2022
1 parent 8513965 commit 8d628a8
Show file tree
Hide file tree
Showing 43 changed files with 795 additions and 7,445 deletions.
8 changes: 3 additions & 5 deletions test-suite/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,17 +8,15 @@ An example metaschema for mocking up documentation generation.

### `schema-generation`

Unit testing over handmade mini metaschemas. Due for maintenance.

To use: under Bash, run ./run-tests.sh in folder schema-generation

./run-tests --help gives help.

./run-tests.sh [testdir] runs tests only in a particular test directory.

### `schema-generation`

Unit testing over handmade mini metaschemas. Due for maintenance.

### `worked-exaples`
### `worked-examples`

With one more testing metaschemas exercising different features.

Expand Down
151 changes: 151 additions & 0 deletions test-suite/metaschema-xspec/composition/expected/nestedmessy_metaschema-composed.xml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,151 @@
<METASCHEMA xmlns="http://csrc.nist.gov/ns/oscal/metaschema/1.0"
abstract="no"
module="nestedmessy"
_base-uri="...">
<INFO info-type="objects-used">Seeing assembly#nestedmessy:box, assembly#nestedmessy:envelope, field#nestedmessy:page, flag#nestedmessy:no, assembly#nestedmessy:folder</INFO>
<schema-name>Nested Reference Test</schema-name>
<schema-version>1.0</schema-version>
<short-name>nestedmessy</short-name>
<namespace>http://csrc.nist.gov/metaschema/ns/nested</namespace>
<json-base-uri>http://csrc.nist.gov/metaschema/ns/nested</json-base-uri>
<remarks>
<p>If any 'unused' definition appears in a schema, we have a bug.</p>
</remarks>
<EXCEPTION problem-type="unused-definition">REMOVING unused assembly definition for 'unused-assembly' from nestedmessy.</EXCEPTION>
<define-field _metaschema-xml-id="/field/nestedmessy/page"
_metaschema-json-id="/field/nestedmessy/page"
collapsible="no"
as-type="string"
scope="global"
name="page"
module="nestedmessy"
_base-uri="..."
_key-name="nestedmessy:page">
<formal-name>PAGE</formal-name>
<description>PAGE</description>
<flag _step="no"
_key="no"
_metaschema-xml-id="/field/nestedmessy/page/no"
_metaschema-json-id="/field/nestedmessy/page/no"
as-type="positiveInteger"
required="no"
ref="no"
_key-ref="nestedmessy:no"
_using-name="no"
_in-xml-name="no"
_in-json-name="no"/>
</define-field>
<define-flag _metaschema-xml-id="/flag/nestedmessy/no"
_metaschema-json-id="/flag/nestedmessy/no"
as-type="positiveInteger"
scope="global"
name="no"
module="nestedmessy"
_base-uri="..."
_key-name="nestedmessy:no"/>
<EXCEPTION problem-type="unused-definition">REMOVING unused field definition for 'unused-field' from nestedmessy.</EXCEPTION>
<define-assembly _metaschema-xml-id="/assembly/nestedmessy/envelope"
_metaschema-json-id="/assembly/nestedmessy/envelope"
scope="global"
name="envelope"
module="nestedmessy"
_base-uri="..."
_key-name="nestedmessy:envelope">
<formal-name>ENVELOPE</formal-name>
<description>ENVELOPE</description>
<model>
<field _step="page"
_key="page"
_metaschema-xml-id="/assembly/nestedmessy/envelope/page"
_metaschema-json-id="/assembly/nestedmessy/envelope/page"
max-occurs="unbounded"
min-occurs="0"
ref="page"
_key-ref="nestedmessy:page"
_using-name="page"
_in-xml-name="page"
_in-json-name="page"/>
</model>
</define-assembly>
<define-assembly _metaschema-xml-id="/assembly/nestedmessy/box"
_metaschema-json-id="/assembly/nestedmessy/box"
scope="global"
name="box"
module="nestedmessy"
_base-uri="..."
_key-name="nestedmessy:box"
_using-root-name="BOX">
<formal-name>BOX</formal-name>
<description>BOX</description>
<root-name>BOX</root-name>
<model>
<assembly _step="envelope"
_key="envelope"
_metaschema-xml-id="/assembly/nestedmessy/box/envelope"
_metaschema-json-id="/assembly/nestedmessy/box/envelope"
max-occurs="unbounded"
min-occurs="0"
ref="envelope"
_key-ref="nestedmessy:envelope"
_using-name="envelope"
_in-xml-name="envelope"
_in-json-name="envelope"/>
<assembly _step="folder"
_key="folder"
_metaschema-xml-id="/assembly/nestedmessy/box/folder"
_metaschema-json-id="/assembly/nestedmessy/box/folder"
max-occurs="unbounded"
min-occurs="0"
ref="folder"
_key-ref="nestedmessy:folder"
_using-name="folder"
_in-xml-name="folder"
_in-json-name="folder"/>
</model>
</define-assembly>
<define-assembly _metaschema-xml-id="/assembly/nestedmessy/folder"
_metaschema-json-id="/assembly/nestedmessy/folder"
scope="global"
name="folder"
module="nestedmessy"
_base-uri="..."
_key-name="nestedmessy:folder">
<formal-name>FOLDER</formal-name>
<description>FOLDER</description>
<model>
<assembly _step="envelope"
_key="envelope"
_metaschema-xml-id="/assembly/nestedmessy/folder/envelope"
_metaschema-json-id="/assembly/nestedmessy/folder/envelope"
max-occurs="unbounded"
min-occurs="0"
ref="envelope"
_key-ref="nestedmessy:envelope"
_using-name="envelope"
_in-xml-name="envelope"
_in-json-name="envelope"/>
<assembly _step="folder"
_key="folder"
_metaschema-xml-id="/assembly/nestedmessy/folder/folder"
_metaschema-json-id="/assembly/nestedmessy/folder/folder"
max-occurs="unbounded"
min-occurs="0"
ref="folder"
_key-ref="nestedmessy:folder"
_using-name="folder"
_in-xml-name="folder"
_in-json-name="folder"/>
<field _step="page"
_key="page"
_metaschema-xml-id="/assembly/nestedmessy/folder/page"
_metaschema-json-id="/assembly/nestedmessy/folder/page"
max-occurs="unbounded"
min-occurs="0"
ref="page"
_key-ref="nestedmessy:page"
_using-name="page"
_in-xml-name="page"
_in-json-name="page"/>
</model>
</define-assembly>
</METASCHEMA>
64 changes: 64 additions & 0 deletions test-suite/metaschema-xspec/composition/input/nestedmessy_metaschema.xml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/css" href="../../../toolchains/xslt-M4/lib/metaschema-author.css"?>
<?xml-model href="../validate/metaschema-composition-check.sch" type="application/xml" schematypens="http://purl.oclc.org/dsdl/schematron"?>
<METASCHEMA xmlns="http://csrc.nist.gov/ns/oscal/metaschema/1.0">
<schema-name>Nested Reference Test</schema-name>
<schema-version>1.0</schema-version>
<short-name>nestedmessy</short-name>
<namespace>http://csrc.nist.gov/metaschema/ns/nested</namespace>
<json-base-uri>http://csrc.nist.gov/metaschema/ns/nested</json-base-uri>

<remarks>
<p>If any 'unused' definition appears in a schema, we have a bug.</p>
</remarks>

<define-assembly name="unused-assembly">
<formal-name>Unused</formal-name>
<description>Unused</description>
<model>
<field ref="unused-field"/>
</model>
</define-assembly>

<define-field name="page">
<formal-name>PAGE</formal-name>
<description>PAGE</description>
<flag ref="no"/>
</define-field>

<define-flag name="no" as-type="positiveInteger"/>

<define-field name="unused-field">
<formal-name>Unused field</formal-name>
<description>Unused field</description>
</define-field>

<define-assembly name="envelope">
<formal-name>ENVELOPE</formal-name>
<description>ENVELOPE</description>
<model>
<field ref="page" max-occurs="unbounded"/>
</model>

</define-assembly>

<define-assembly name="box">
<formal-name>BOX</formal-name>
<description>BOX</description>
<root-name>BOX</root-name>
<model>
<assembly ref="envelope" max-occurs="unbounded"/>
<assembly ref="folder" max-occurs="unbounded"/>
</model>
</define-assembly>

<define-assembly name="folder">
<formal-name>FOLDER</formal-name>
<description>FOLDER</description>
<model>
<assembly ref="envelope" max-occurs="unbounded"/>
<assembly ref="folder" max-occurs="unbounded"/>
<field ref="page" max-occurs="unbounded"/>
</model>
</define-assembly>
</METASCHEMA>
9 changes: 9 additions & 0 deletions test-suite/metaschema-xspec/composition/input/pruning001.xml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
<?xml version="1.0" encoding="UTF-8"?>
<METASCHEMA xmlns="http://csrc.nist.gov/ns/oscal/metaschema/1.0"
module="pruning001">
<define-assembly name="box">
<formal-name>BOX</formal-name>
<description>Just a box, no relationship to anything else.</description>
<root-name>BOX</root-name>
</define-assembly>
</METASCHEMA>
16 changes: 16 additions & 0 deletions test-suite/metaschema-xspec/composition/input/pruning002.xml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
<?xml version="1.0" encoding="UTF-8"?>
<METASCHEMA xmlns="http://csrc.nist.gov/ns/oscal/metaschema/1.0"
module="pruning002">
<define-assembly name="box">
<formal-name>BOX</formal-name>
<description>Just a box, no relationship to anything else.</description>
<root-name>BOX</root-name>
<model>
<assembly ref="folder"/>
</model>
</define-assembly>
<define-assembly name="folder">
<formal-name>Folder</formal-name>
<description>One or more folders belong in a box.</description>
</define-assembly>
</METASCHEMA>
14 changes: 14 additions & 0 deletions test-suite/metaschema-xspec/composition/input/pruning003.xml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
<?xml version="1.0" encoding="UTF-8"?>
<METASCHEMA xmlns="http://csrc.nist.gov/ns/oscal/metaschema/1.0"
module="pruning003">
<define-assembly name="box">
<formal-name>BOX</formal-name>
<description>Just a box, no relationship to anything else.</description>
<root-name>BOX</root-name>
<model/>
</define-assembly>
<define-assembly name="folder">
<formal-name>Folder</formal-name>
<description>One or more folders belong in a box.</description>
</define-assembly>
</METASCHEMA>
20 changes: 20 additions & 0 deletions test-suite/metaschema-xspec/composition/metaschema-composition.xspec
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
<?xml version="1.0" encoding="UTF-8"?>
<x:description xmlns:x="http://www.jenitennison.com/xslt/xspec"
stylesheet="../../../toolchains/xslt-M4/nist-metaschema-COMPOSE.xsl"
run-as="external"
xmlns="http://csrc.nist.gov/ns/oscal/metaschema/1.0"
xmlns:m="http://csrc.nist.gov/ns/oscal/metaschema/1.0"
xmlns:ov="http://csrc.nist.gov/ns/oscal/test/variable">

<x:helper stylesheet="../metaschema-test-helper.xsl"/>

<x:scenario label="Metaschema with unused definitions">
<!-- Scrubbing both the result and the expected result for comparison -->
<x:context href="input/nestedmessy_metaschema.xml"/>
<x:variable name="ov:expected-output"
href="expected/nestedmessy_metaschema-composed.xml"/>
<x:expect label="Base test - fully resolved"
test="m:scrub($x:result)" select="m:scrub($ov:expected-output)"/>
</x:scenario>

</x:description>
41 changes: 41 additions & 0 deletions test-suite/metaschema-xspec/composition/metaschema-prune-unused-definitions.xspec
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
<?xml version="1.0" encoding="UTF-8"?>
<x:description
xmlns:xs="http://www.w3.org/2001/XMLSchema"
xmlns:x="http://www.jenitennison.com/xslt/xspec"
xmlns:m="http://csrc.nist.gov/ns/oscal/metaschema/1.0"
xmlns="http://csrc.nist.gov/ns/oscal/metaschema/1.0"
stylesheet="../../../toolchains/xslt-M4/nist-metaschema-COMPOSE.xsl"
run-as="external">
<x:scenario label="In Metaschema composition pruning phase">
<x:scenario label="during the pruning phase">
<x:scenario label="a single root assembly with no references">
<x:context href="./input/pruning001.xml"/>
<x:expect label="should not throw an exception"
test="$x:result//m:EXCEPTION => empty()"/>
</x:scenario>
<x:scenario label="top-level definitions with correct references.">
<x:context href="./input/pruning002.xml"/>
<x:expect label="should not throw an exception"
test="$x:result//m:EXCEPTION => empty()"/>
</x:scenario>
<x:scenario label="top-level definitions with an unused definition.">
<x:context href="./input/pruning003.xml"/>
<x:expect label="throws only exception for an unused definition."
test="count($x:result//m:EXCEPTION) eq 1"/>
<x:expect label="and should be of type unused-definition with the proper message."
test="$x:result//m:EXCEPTION[@problem-type = 'unused-definition'] => exists()"/>
<x:variable name="m:ec" as="xs:string">REMOVING unused assembly definition for 'folder' from pruning003.</x:variable>
<x:expect label="and the error message indicates the specific definition that's unused."
test="$x:result//m:EXCEPTION[@problem-type='unused-definition'] = $m:ec"/>
</x:scenario>
<x:scenario label="top-level definitions with an unused definition and an orphan (not used by anything used).">
<x:context href="./input/pruning004.xml"/>
<x:expect label="throws two exceptions for unused definitions"
test="count($x:result//m:EXCEPTION) eq 2"/>
<x:expect label="and has neither unused model defined."
test="($x:result//m:define-assembly[@name='folder'] |
$x:result//m:define-field[@name='page']) => empty()"/>
</x:scenario>
</x:scenario>
</x:scenario>
</x:description>
15 changes: 15 additions & 0 deletions test-suite/metaschema-xspec/composition/readme.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# Metaschema processing XSpec testing - Metaschema Composition end-to-end (E2E)

To run XSpec, we suggest either an XML IDE or a command line tool as described in [XSpec documentation in Github](https://github.com/xspec/xspec/wiki).

* `composition/metaschema-composition.xspec`

Tests XSLT `../../../toolchains/xslt-M4/nist-metaschema-COMPOSE.xsl`

Currently checks if unused definitions are correctly filtered away from a metaschema source instance, in composition.

(This prevents result schemas from including unused models.)

* `composition/metaschema-prune-unused-definitions.xspec`

Also testing the same bug, but using standoff instances, with a start on more comprehensive tests.
Loading

0 comments on commit 8d628a8

Please sign in to comment.