Requirements gathering and design for next-gen Metaschema

[wendellpiez]

Once we have the development platform stable (#15) we can start working on requirements gathering and design for more better Metaschema functionality as well as refactoring and improving current functionality.

Important work areas (these could be split apart):

Modularity - the current Metaschema design permits assembly of metaschemas from modules including modules shared with other metaschemas. Issues in the past have revealed some confusion about this (usnistgov/OSCAL#406), although it works.

Extensibility - escape hatch functionality (extension by restriction) works well but may not be ideal or even adequate for all purposes. Users and developers need options and guidance. See usnistgov/OSCAL#546 among others.

Higher-level validation (need a better name?) we are talking about validating data against sets of constraints that go beyond what is supported by XSD or JSON schema ootb. We need to be able both to express the constraints we wish to impose, and implement checking against those constraints in XML and JSON contexts. For example, an XML-based implementation could work by producing a Schematron or a Schematron-like XSLT which could be applied to data and return status reports regarding conformance.

A metaschema - either a core metaschema, or another layer applied to a core metaschema, is the reasonable place for constraint declaration and specification. Tooling applied to a metaschema can then produce the applicable code.

Problems to be addressed in these constraint declarations:

• Defining scope for validation of referential integrity
  • not just document-wide but within a particular branch in a document, or across nodes dispersed among a set of documents
• Key-based retrieval - retrieving nodes based on values, nominal types, or associated values, not only location
• Data comparisons - starting with string comparison and matching (regex) but also including typed operations such as numeric, date or duration comparison (X is later than Y, etc.)
• Uniqueness (distinctiveness) testing - should fall out naturally from the foregoing. (x is distinct within set Y if no member of Y apart from x, equals x.)

Any others? Please comment

[wendellpiez]

Feb 6 2020

This issue has been open a month and no one has commented.

This suggests to me that our current thinking about the Metaschema refactor is adequate and we should not pause for requirements gathering at this point. Instead, we should push to implement the current design with planned extensions and modification only (see #19, #31), on a new basis that makes a strong foundation for later work.

Can we close this ticket or plan to revisit later after phase 1 refactor is complete?

[aj-stein-nist]

We in the OSCAL Team have been working on this design work elsewhere in specific issues addressing individual items, like #171. We will focus on those and close this.