Skip to content

Commit

Permalink
Metaschema / XSLT implementation alignment (#197)
Browse files Browse the repository at this point in the history 
* Relocate schema resources (#191)

- Moved schema resources out of XSLT implementation
- Relocated XSD and datatype XML schemas.
Removed junk and generated files.
Many datatype adjustments
   - Tweeks to schema data types to address unicode issues. Resolves usnistgov/OSCAL#1127. Resolves usnistgov/OSCAL#956.
   - Adjusted type names of Metaschema types to be purely hyphenated.
   - Removed extra character ranges that are causing C# problems. Resolves usnistgov/OSCAL#1127.
- Adjusted URI data type names to be more consistent between XML and JSON.
Many metaschema.xsd adjustments
- Alignment of data types used with new Metschema datatype names.
- Additional repairs of tests, including naming consistency.
   - adding new unit test for a valid json-value key using a label
   - many fixes to broken tests
   - Removed duplicate uuid test scenario.
   - Many fixes to make unit test Metaschemas valid.
- Added JSON schema schema.
- Some schema generation refactoring to support new data-driven test harness.

* Fixed URI of choice unit test.

* Update metaschema-datatypes.xsd

Adjusting whitespace handling for URI types.

* adding leading and trailing whitespace checks to all XML schema datatype derived types

* Adjustments to debug charstrings.

* Update metaschema-datatypes.json

Minor adjustments to data types for comments on #1260.

* refactored metaschema schema to support external constraint definitions

* Added constraint extensibility configuration.

* added formal-name and description to definition references

* XSLT M4 Metaschema pipeline updates (#214)

* Composition unit tests now valid to updated schemas; added (missing) tests.

* Adding XSpec for schema generation; nominal schema target examples

* Adding XSpec tests (testing schema generation as transforms) and initial set of targets for testing

* Major reorganization and archiving (some temporary, prior to defenestration); new tests; readme documents in each folder for schema-generation unit tests to help trace efforts
* More details in json-value-key readme; updated top-level JSON schema generation xspec
* Metaschema Schematron rule intercepting a json-key setting with no BY_KEY in the grouping logic
* Cleaned up extra JSON Schema file
* Updated metaschema model wrt json-key, json-value-key flag-ref (no longer flag-name); other Metaschema touchups
* Rewiring and simplifying XSD production pipeline with datatypes now acquired from the normative metaschema /schema subdirectory - removed the namespace fixup step, no longer needed.
* Updated anthology ('menagerie') metaschema example, with cleanup
* In XSD generation, now emitting datatype definitions only for datatypes actually needed by a schema
* Updating schema TODO notes on synching unit tests; bit of cleanup
* Addressing #212 - XML to JSON converter no longer chokes on edge cases (array items of SINGLETON_OR_ARRAY groups)
* Updating top-level composition unit tests
* .gitignore covering HTML reports from test runs
* Removing outdated testing results
* Adding back support for old datatype names as indicated in metaschema.xsd (cf line 1252)
* Restoring deprecated datatypes per #195
* Bit of cleanup; utility maintenance
* Removing 'INFO' level comment in XSD
* Tweakage to align XSD out (dropping comment; cleaning up namespaces)
* Patching hole in JSON datatype assignment also
* Emitting cleaner namespaces in XSD; slight refactoring of JSON Schema
* Adjusting JSON Schema type definitions to produce valid constraints over Metaschema atomic types
* Extending atomic data type acquisition to collect one level of indirect references
* Cleaning up obsolete datatype support
* Bit more cleanup; updated readme
* Adjusting handling of warnings and exception messages in pipelines
* Removing outdated artifacts from schema generation unit testing.md

Co-authored-by: Wendell Piez <wendell.piez@nist.gov>
  • Loading branch information
@david-waltermire @wendellpiez
david-waltermire and wendellpiez committed Dec 7, 2022
1 parent c19ef72 commit abe0ad2
Show file tree
Hide file tree
Showing 283 changed files with 10,209 additions and 23,041 deletions.
20 changes: 14 additions & 6 deletions ...schema-gen/metaschema-v090-datatypes.json → schema/json/metaschema-datatypes.json
View file
Original file line number Diff line number Diff line change
Expand Up @@ -44,8 +44,10 @@
"pattern": "^.+@.+$"
},
"HostnameDatatype": {
"$ref": "#/definitions/StringDatatype",
"format": "idn-hostname"
"allOf": [
{"$ref": "#/definitions/StringDatatype"},
{"format": "idn-hostname"}
]
},
"IntegerDatatype": {
"type": "integer"
Expand All @@ -68,12 +70,18 @@
"type": "string"
},
"NonNegativeIntegerDatatype": {
"$ref": "#/definitions/IntegerDatatype",
"minimum": 0
"allOf": [
{"$ref": "#/definitions/IntegerDatatype"},
{"minimum": 0,
"type": "number"}
]
},
"PositiveIntegerDatatype": {
"$ref": "#/definitions/IntegerDatatype",
"minimum": 1
"allOf": [
{"$ref": "#/definitions/IntegerDatatype"},
{"minimum": 1,
"type": "number"}
]
},
"StringDatatype": {
"type": "string",
Expand Down
241 changes: 241 additions & 0 deletions schema/xml/metaschema-datatypes.xsd
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,241 @@
<?xml version="1.0" encoding="utf-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
elementFormDefault="qualified">

<xs:simpleType name="Base64Datatype">
<xs:restriction base="xs:base64Binary">
<xs:pattern value="\S(.*\S)?">
<xs:annotation>
<xs:documentation>A trimmed string, at least one character with no
leading or trailing whitespace.</xs:documentation>
</xs:annotation>
</xs:pattern>
</xs:restriction>
</xs:simpleType>

<xs:simpleType name="BooleanDatatype">
<xs:restriction base="xs:boolean">
<xs:pattern value="\S(.*\S)?">
<xs:annotation>
<xs:documentation>A trimmed string, at least one character with no
leading or trailing whitespace.</xs:documentation>
</xs:annotation>
</xs:pattern>
</xs:restriction>
</xs:simpleType>

<xs:simpleType name="DateDatatype">
<xs:restriction base="xs:date">
<xs:pattern value="(((2000|2400|2800|(19|2[0-9](0[48]|[2468][048]|[13579][26])))-02-29)|(((19|2[0-9])[0-9]{2})-02-(0[1-9]|1[0-9]|2[0-8]))|(((19|2[0-9])[0-9]{2})-(0[13578]|10|12)-(0[1-9]|[12][0-9]|3[01]))|(((19|2[0-9])[0-9]{2})-(0[469]|11)-(0[1-9]|[12][0-9]|30)))(Z|[+-][0-9]{2}:[0-9]{2})?"/>
</xs:restriction>
</xs:simpleType>

<xs:simpleType name="DateWithTimezoneDatatype">
<xs:annotation>
<xs:documentation>The xs:date with a required timezone.</xs:documentation>
</xs:annotation>
<xs:restriction base="DateDatatype">
<xs:pattern value="(((2000|2400|2800|(19|2[0-9](0[48]|[2468][048]|[13579][26])))-02-29)|(((19|2[0-9])[0-9]{2})-02-(0[1-9]|1[0-9]|2[0-8]))|(((19|2[0-9])[0-9]{2})-(0[13578]|10|12)-(0[1-9]|[12][0-9]|3[01]))|(((19|2[0-9])[0-9]{2})-(0[469]|11)-(0[1-9]|[12][0-9]|30)))(Z|[+-][0-9]{2}:[0-9]{2})"/>
</xs:restriction>
</xs:simpleType>

<xs:simpleType name="DateTimeDatatype">
<xs:restriction base="xs:dateTime">
<xs:pattern value="(((2000|2400|2800|(19|2[0-9](0[48]|[2468][048]|[13579][26])))-02-29)|(((19|2[0-9])[0-9]{2})-02-(0[1-9]|1[0-9]|2[0-8]))|(((19|2[0-9])[0-9]{2})-(0[13578]|10|12)-(0[1-9]|[12][0-9]|3[01]))|(((19|2[0-9])[0-9]{2})-(0[469]|11)-(0[1-9]|[12][0-9]|30)))T((2[0-3]|[01][0-9]):([0-5][0-9]):([0-5][0-9])(\.[0-9]+)?)(Z|[+-][0-9]{2}:[0-9]{2})?"/>
</xs:restriction>
</xs:simpleType>

<xs:simpleType name="DateTimeWithTimezoneDatatype">
<xs:annotation>
<xs:documentation>The xs:dateTime with a required timezone.</xs:documentation>
</xs:annotation>
<xs:restriction base="DateTimeDatatype">
<xs:pattern value="(((2000|2400|2800|(19|2[0-9](0[48]|[2468][048]|[13579][26])))-02-29)|(((19|2[0-9])[0-9]{2})-02-(0[1-9]|1[0-9]|2[0-8]))|(((19|2[0-9])[0-9]{2})-(0[13578]|10|12)-(0[1-9]|[12][0-9]|3[01]))|(((19|2[0-9])[0-9]{2})-(0[469]|11)-(0[1-9]|[12][0-9]|30)))T((2[0-3]|[01][0-9]):([0-5][0-9]):([0-5][0-9])(\.[0-9]+)?)(Z|[+-][0-9]{2}:[0-9]{2})"/>
</xs:restriction>
</xs:simpleType>

<xs:simpleType name="DayTimeDurationDatatype">
<xs:restriction base="xs:duration">
<xs:pattern value="[-+]?P([-+]?[0-9]+D)?(T([-+]?[0-9]+H)?([-+]?[0-9]+M)?([-+]?[0-9]+([.,][0-9]{0,9})?S)?)?"/>
</xs:restriction>
</xs:simpleType>

<xs:simpleType name="DecimalDatatype">
<xs:restriction base="xs:decimal">
<xs:pattern value="\S(.*\S)?">
<xs:annotation>
<xs:documentation>A trimmed string, at least one character with no
leading or trailing whitespace.</xs:documentation>
</xs:annotation>
</xs:pattern>
</xs:restriction>
</xs:simpleType>

<xs:simpleType name="EmailAddressDatatype">
<xs:annotation>
<xs:documentation>An email address</xs:documentation>
</xs:annotation>
<xs:restriction base="StringDatatype">
<xs:pattern value="\S.*@.*\S">
<xs:annotation>
<xs:documentation>Need a better pattern.</xs:documentation>
</xs:annotation>
</xs:pattern>
</xs:restriction>
</xs:simpleType>

<xs:simpleType name="HostnameDatatype">
<xs:annotation>
<xs:documentation>A host name</xs:documentation>
</xs:annotation>
<xs:restriction base="StringDatatype">
<!-- TODO: Need a good hostname pattern -->
</xs:restriction>
</xs:simpleType>

<xs:simpleType name="IntegerDatatype">
<xs:restriction base="xs:integer">
<xs:pattern value="\S(.*\S)?">
<xs:annotation>
<xs:documentation>A trimmed string, at least one character with no
leading or trailing whitespace.</xs:documentation>
</xs:annotation>
</xs:pattern>
</xs:restriction>
</xs:simpleType>

<xs:simpleType name="IPV4AddressDatatype">
<xs:annotation>
<xs:documentation>The ip-v4-address type specifies an IPv4 address in
dot decimal notation.</xs:documentation>
</xs:annotation>
<xs:restriction base="StringDatatype">
<xs:pattern value="((25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9][0-9]|[0-9]).){3}(25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9][0-9]|[0-9])" />
</xs:restriction>
</xs:simpleType>

<xs:simpleType name="IPV6AddressDatatype">
<xs:annotation>
<xs:documentation>The ip-v6-address type specifies an IPv6 address
represented in 8 hextets separated by colons.</xs:documentation>
<xs:documentation>This is based on the pattern provided here:
https://stackoverflow.com/questions/53497/regular-expression-that-matches-valid-ipv6-addresses
with some customizations.</xs:documentation>
</xs:annotation>
<xs:restriction base="xs:string">
<xs:whiteSpace value="collapse" />
<xs:pattern
value="(([0-9a-fA-F]{1,4}:){7,7}[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,7}:|([0-9a-fA-F]{1,4}:){1,6}:[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,5}(:[0-9a-fA-F]{1,4}){1,2}|([0-9a-fA-F]{1,4}:){1,4}(:[0-9a-fA-F]{1,4}){1,3}|([0-9a-fA-F]{1,4}:){1,3}(:[0-9a-fA-F]{1,4}){1,4}|([0-9a-fA-F]{1,4}:){1,2}(:[0-9a-fA-F]{1,4}){1,5}|[0-9a-fA-F]{1,4}:((:[0-9a-fA-F]{1,4}){1,6})|:((:[0-9a-fA-F]{1,4}){1,7}|:)|[fF][eE]80:(:[0-9a-fA-F]{0,4}){0,4}%[0-9a-zA-Z]{1,}|::([fF]{4}(:0{1,4}){0,1}:){0,1}((25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9][0-9]|[0-9]).){3,3}(25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9][0-9]|[0-9])|([0-9a-fA-F]{1,4}:){1,4}:((25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9][0-9]|[0-9]).){3,3}(25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9][0-9]|[0-9]))" />
</xs:restriction>
</xs:simpleType>

<xs:simpleType name="NonNegativeIntegerDatatype">
<xs:restriction base="xs:nonNegativeInteger">
<xs:pattern value="\S(.*\S)?">
<xs:annotation>
<xs:documentation>A trimmed string, at least one character with no
leading or trailing whitespace.</xs:documentation>
</xs:annotation>
</xs:pattern>
</xs:restriction>
</xs:simpleType>

<xs:simpleType name="PositiveIntegerDatatype">
<xs:restriction base="xs:positiveInteger">
<xs:pattern value="\S(.*\S)?">
<xs:annotation>
<xs:documentation>A trimmed string, at least one character with no
leading or trailing whitespace.</xs:documentation>
</xs:annotation>
</xs:pattern>
</xs:restriction>
</xs:simpleType>

<xs:simpleType name="StringDatatype">
<xs:annotation>
<xs:documentation>A string, but not empty and not whitespace-only
(whitespace is U+9, U+10, U+32 or [ \n\t]+ )</xs:documentation>
</xs:annotation>
<xs:restriction base="xs:string">
<xs:annotation>
<xs:documentation>The OSCAL 'string' datatype restricts the XSD type by prohibiting leading
and trailing whitespace, and something (not only whitespace) is required.</xs:documentation>
</xs:annotation>
<xs:whiteSpace value="preserve" />
<xs:pattern value="\S(.*\S)?">
<xs:annotation>
<xs:documentation>A trimmed string, at least one character with no
leading or trailing whitespace.</xs:documentation>
</xs:annotation>
</xs:pattern>
</xs:restriction>
</xs:simpleType>

<xs:simpleType name="TokenDatatype">
<xs:annotation>
<!--<xs:documentation>Matching XSD NCName, except whitespace is not collapsed.</xs:documentation> -->
<xs:documentation>A string token following the rules of XML "no
colon" names, with no whitespace. (XML names are single alphabetic
characters followed by alphanumeric characters, periods, underscores or dashes.)
</xs:documentation>
</xs:annotation>
<xs:restriction base="StringDatatype">
<!-- value="(\p{L}|_)(\p{L}|\p{N}|[.\-_]|\xb7|[#x0300-#x036F#x203F-#x2040])*">-->
<xs:pattern
value="(\p{L}|_)(\p{L}|\p{N}|[.\-_])*">
<xs:annotation>
<!--<xs:documentation>An XML initial character (but not colon), followed
by any XML name character (but not colon).</xs:documentation> -->
<xs:documentation>A single token may not contain whitespace.
</xs:documentation>
</xs:annotation>
</xs:pattern>
</xs:restriction>
</xs:simpleType>

<xs:simpleType name="URIDatatype">
<xs:annotation>
<xs:documentation>A URI</xs:documentation>
</xs:annotation>
<xs:restriction base="xs:anyURI">
<xs:pattern value="[a-zA-Z][a-zA-Z0-9+\-.]+:.*\S">
<xs:annotation>
<xs:documentation>Requires a scheme with colon per RFC 3986.</xs:documentation>
</xs:annotation>
</xs:pattern>
</xs:restriction>
</xs:simpleType>

<xs:simpleType name="URIReferenceDatatype">
<xs:annotation>
<xs:documentation>A URI reference, such as a relative URL
</xs:documentation>
</xs:annotation>
<xs:restriction base="xs:anyURI">
<xs:pattern value="\S(.*\S)?">
<xs:annotation>
<xs:documentation>A trimmed URI, at least one character with no
leading or trailing whitespace.</xs:documentation>
</xs:annotation>
</xs:pattern>
</xs:restriction>
</xs:simpleType>

<xs:simpleType name="UUIDDatatype">
<xs:annotation>
<xs:documentation>A type 4 ('random' or 'pseudorandom') or type 5 UUID per RFC
4122.</xs:documentation>
</xs:annotation>
<xs:restriction base="StringDatatype">
<xs:pattern
value="[0-9A-Fa-f]{8}-[0-9A-Fa-f]{4}-[45][0-9A-Fa-f]{3}-[89ABab][0-9A-Fa-f]{3}-[0-9A-Fa-f]{12}">
<xs:annotation>
<xs:documentation>A sequence of 8-4-4-4-12 hex digits, with extra
constraints in the 13th and 17-18th places for version 4 and 5
</xs:documentation>
</xs:annotation>
</xs:pattern>
</xs:restriction>
</xs:simpleType>
</xs:schema>

11 changes: 11 additions & 0 deletions schema/xml/metaschema-markup-line.xsd
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" elementFormDefault="qualified">

<xs:include schemaLocation="metaschema-prose-base.xsd"/>

<xs:complexType name="MarkupLineDatatype" mixed="true">
<xs:choice minOccurs="0" maxOccurs="unbounded">
<xs:group ref="inlineMarkupGroup"/>
</xs:choice>
</xs:complexType>
</xs:schema>
85 changes: 4 additions & 81 deletions ...xslt-M4/schema-gen/oscal-prose-module.xsd → schema/xml/metaschema-markup-multiline.xsd
View file
Original file line number Diff line number Diff line change
@@ -1,14 +1,10 @@
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" elementFormDefault="qualified">

<xs:include schemaLocation="metaschema-prose-base.xsd"/>

<xs:complexType name="markupLineType" mixed="true">
<xs:choice minOccurs="0" maxOccurs="unbounded">
<xs:group ref="inlineMarkupGroup"/>
</xs:choice>
</xs:complexType>

<xs:complexType name="markupMultilineType">
<xs:choice minOccurs="0" maxOccurs="unbounded">
<xs:complexType name="MarkupMultilineDatatype">
<xs:choice minOccurs="1" maxOccurs="unbounded">
<xs:group ref="blockElementGroup"/>
</xs:choice>
</xs:complexType>
Expand Down Expand Up @@ -103,77 +99,4 @@
<xs:enumeration value="right"/>
</xs:restriction>
</xs:simpleType>

<xs:group name="inlineMarkupGroup">
<xs:choice>
<xs:element name="a" type="anchorType"/>
<xs:element name="insert" type="insertType"/>
<xs:group ref="phraseMarkupGroup"/>
<xs:group ref="imageMarkupGroup"/>
</xs:choice>
</xs:group>
<xs:group name="imageMarkupGroup">
<xs:choice>
<xs:element name="img" type="imageType"/>
</xs:choice>
</xs:group>
<xs:group name="phraseMarkupGroup">
<xs:choice>
<xs:element name="code" type="inlineMarkupType" minOccurs="1"/>
<xs:element name="em" type="inlineMarkupType" minOccurs="1"/>
<xs:element name="i" type="inlineMarkupType" minOccurs="1"/>
<xs:element name="b" type="inlineMarkupType" minOccurs="1"/>
<xs:element name="strong" type="inlineMarkupType" minOccurs="1"/>
<xs:element name="sub" type="inlineMarkupType" minOccurs="1"/>
<xs:element name="sup" type="inlineMarkupType" minOccurs="1"/>
<xs:element name="q" type="inlineMarkupType" minOccurs="1"/>
</xs:choice>
</xs:group>
<xs:complexType name="inlineMarkupType" mixed="true">
<xs:choice minOccurs="0" maxOccurs="unbounded">
<xs:group ref="inlineMarkupGroup"/>
</xs:choice>
</xs:complexType>
<xs:complexType name="imageType">
<xs:attribute name="alt" type="xs:string"/>
<xs:attribute name="src" use="required" type="xs:anyURI"/>
</xs:complexType>

<xs:complexType name="anchorType" mixed="true">
<xs:choice minOccurs="0" maxOccurs="unbounded">
<xs:group ref="phraseMarkupGroup"/>
<xs:group ref="imageMarkupGroup"/>
</xs:choice>
<xs:attribute name="href" type="xs:anyURI"/>
</xs:complexType>

<!--
A value to be assigned by responder via parameter
(a parameter must be available)
-->
<xs:complexType name="insertType">
<xs:annotation>
<xs:documentation>An insert can be used to identify a placeholder for dynamically inserting text related to a specific object, which is referenced by the object's identifier using an <code>id-ref</code>. This insert mechanism allows the selection of which text value from the object to dynamically include based on the application's display requirements.</xs:documentation>
</xs:annotation>
<xs:attribute name="type" use="required" type="xs:NCName">
<xs:annotation>
<xs:documentation>The type of object to include from (e.g., parameter, control, component, role, etc.)</xs:documentation>
</xs:annotation>
</xs:attribute>
<xs:attribute name="id-ref" use="required" type="xs:NCName">
<xs:annotation>
<xs:documentation>The identity of the object to insert a value for. The identity will be selected from the index of objects of the specified <code>type</code>. The specific value to include is based on the application's display requirements, which will likely use a specific data element associated with the <code>type</code> (e.g., title, identifier, value, etc.) that is appropriate for the application.</xs:documentation>
</xs:annotation>
</xs:attribute>
</xs:complexType>

<!--
Within a selection, a choice - tbd
unlike 'value', choice may have inline stuff as well as 'assign' elements
choice = element choice { (assign | mix)* }
-->
<!--
IDs are required everywhere for now
idAttr = attribute id { xsd:ID }
-->
</xs:schema>
Loading

0 comments on commit abe0ad2

Please sign in to comment.