Metaschema constraints definition and enforcement

[wendellpiez]

User Story:

Metaschema needs new features to help developers and potentially users (content developers) to validate arbitrary constraints over their data, sometimes defined locally, frequently over and above the constraints introduced by the models themselves. So, not a new field (for example), but a new rule about values permitted on a field with a given flag.

This bears importantly on customization, as many customizations if not most can take the form of new constraints over generic models (such as part and prop) rather than new models.

Goals:

Extend the features of Metaschema to support more and better constraint checking, including node occurrence and value constraints as well as compound and co-occurrence constraints.

We start with customizing allowed-values but we go from there. # 2 on the list is probably support for checking referential integrity, for example of internal links, or whether every insert points to a parameter.

An implemention will be able to read the constraint definition and produce appropriate XSD and/or Schematron code, for XML, and for JSON, the functional equivalent. (Perhaps another Schematron, or a single "ambidextrous" one.)

Dependencies:

A stable Metaschema. This work is being done in conjunction with usnistgov/metaschema#39.

In its branch there are (rough) design notes exploring a number of features we wish to support: see https://github.com/wendellpiez/metaschema/blob/issue39-pipeline-refactor/toolchains/xslt-proto-v04/definition-notes.md.

Acceptance Criteria

• All website and readme documentation affected by the changes in this issue have been updated. Changes to the website can be made in the docs/content directory of your branch.
• A Pull Request (PR) is submitted that fully addresses the goals of this User Story. This issue is referenced in the PR.
• The CI-CD build process runs without any reported errors on the PR. This can be confirmed by reviewing that all checks have passed in the PR.

[wendellpiez]

April 8 Update

Considerable progress on this in my working branch. An implementation is now nominally feature complete. It works by producing a Schematron to be used alongside the XSD for a Metaschema, which reports errors for constraints violated in data conforming to its model. Testing and bug fixing proceeds.

https://github.com/wendellpiez/metaschema/tree/issue39-pipeline-refactor

[wendellpiez]

April 30 Update

After demonstrating the basic idea of Metaschema-based Schematron production for constraints enforcement, including the necessary path mapping (#45), we have been iterating the design, extending to support local extensions of constraints on data objects, defined on ancestors of those objects.

At this point there are still two or three questions of design/specification in this feature set, Most lately, I have created a couple of test metaschemas illustrating these issues. @david-waltermire-nist and I will continue to work on this with all deliberate speed.

Cf usnistgov/OSCAL#633

[wendellpiez]

August 4 Update

This has been stable while we built out other features including bidirectional conversion.

Remaining to do: build XSLT to read an XML-oriented Metatron and produce the JSONatron variant (a Schematron that can be run over JSON). Also, testing.

[david-waltermire]

The Metaschema constraints have been implemented in the Java-based toolchain. The XSLT implementation needs to be brought up to the same level of support.

[wendellpiez]

WIP on this is in the repository at https://github.com/usnistgov/metaschema/blob/main/toolchains/xslt-M4/schema-gen/make-metaschema-metatron.xsl. It is stable although not fully tested and possibly not fully current.

Moving forward then depends on external specs and unit tests for constraints definition and use.

This could be done informally against OSCAL but I am not ready to plan that.

Advice? Advancing on this depends on how we refactor the implementations and unit tests for Metaschema 'constraint' features.