Tracker Issue for Metaschema work

[wendellpiez]

User Story:

Proceeding with integration of next-generation Metaschema.

• Stand up and test next-generation XSD and JSON Schema generation
  • update deployed Metaschema schema and Schematron to new version
    • update OSCAL metaschemas
  • integrate new XSLTs into scripting
    • produce new schemas using XSLTs
    • validate all contents against new schemas
  • Integrate and test XSLTs under CI/CD
• Build, test and deploy constraints enforcement model (Metatron)
  • Schematron generation (for enforcing constraints over XML)
    • testing
  • JSONatron generation (for enforcing constraints over JSON)
    • testing
  • Integration of Metatron production
• Plan, complete, test and deploy the two-way (XML/JSON) content conversion
  • XML -> supermodel tranform
  • JSON -> supermodel transform
  • supermodel -> XML transform (serializer)
  • supermodel -> JSON transform (serializer)
  • combining these pairs into XML->JSON and JSON->XML capabilities (for any model)
  • Integrate this capability into docs production?
• (Re)design, build out and deploy updated schema documentation
  • Including both documentation pages and model maps
  • Support for local definitions
  • Mockup or placeholder for constraints definitions

To do: make Issues for these in Metaschema repo and add them here?

In the usnistgov/metaschema repo, a number of Issues track development of Metaschema features:

• Bugs or small enhancements:
  • Generated JSON Schema does not enforce root element metaschema#11,
  • Field collapse is not properly handled in schema generation metaschema-xslt#11,
  • Bug: Identify Broken Imports in Metaschema metaschema-xslt#8
• Requirements gathering and design for next-gen Metaschema metaschema#17 describes functional requirements for checking allowed values
• Extend to support multiple designated (valid) root nodes metaschema#31 follows on Generated JSON Schema does not enforce root element metaschema#11 and addresses Valid OSCAL File Fragments #590 in this repo
• Metaschema refactor phase 1: towards v2 metaschema#39 provides high-level tracking of redesign and implementation
• Following on Requirements gathering and design for next-gen Metaschema metaschema#17, Metaschema constraints definition and enforcement metaschema-xslt#12 works on requirements analysis and design/development of constraint-checking features

[wendellpiez]

Ongoing work is here: https://github.com/wendellpiez/metaschema/tree/issue39-pipeline-refactor

[wendellpiez]

Sprint 28 Update April 2

We are making good headway on this in metaschema repository fork (linked above). There is running code plus a working test metaschema with data. A pipeline produces a Schematron from the metaschema that detects errors in the data. The data shows both "good" and "known bad" tagging, which functions correctly and permits demonstration.

Now working: co-occurrence tests including allowed (enumerated) values, string matching, cardinality constraints. There are a couple of semantics issues to work out with @david-waltermire-nist. Need a few more days for a v1 of the Metaschema->Schematron. Then back to XSD and JSON Schema.

[wendellpiez]

Sprint 28 Update April 8

Nominally feature complete on an implementation that can enforce constraints over XML defined by a Metaschema. An XSLT pipeline produces a Schematron that detects and reports constraint violations in data.

To do this we had to produce an early solution to the XPath parsing question (cf usnistgov/metaschema-xslt#13), which is now in place.

Some demonstration and unit testing is in place, more is coming. Bug fixes as we expose remaining issues.

Next up on this item is probably the Metaschema-aware path rewriting so we can produce a JSONatron version or alternative (enforcing constraints over JSON directly).

It needs to be able to take a path such as prop and return the appropriate alternative, in this case *[@key='properties']/* where "properties" is the grouping name for prop.

It could also make sense to produce JSONpointer ? (Any application to target there?)

[wendellpiez]

Update April 16

Making steady progress. See usnistgov/metaschema#39 usnistgov/metaschema-xslt#12 usnistgov/metaschema-xslt#13.

Functionality is now complete to a nominal v1 feature set, except for full datatype support.

Much testing and then documentation remains to be done. See also usnistgov/metaschema-xslt#35.

Metaschema now supports constraint definition and testing against a range of different constraint types including co-occurrence constraints (when X is a, Y must be b) and key references. It does this through a parallel pipeline (to XSD and JSON Schema production), which produces a Schematron that can be applied to instances to achieve validation.

In support of this, we are now parsing Metapath inside the XSLT implementation of Metaschema, which will also enable other features such as validating JSON directly (without conversion first).

[wendellpiez]

Update May 7 2020

I added check boxes for progress tracking in the description above.

Progress is good. Remaining to do:

• Extending Metatron (Metaschema-based Schematron implementing constraints checking) to support JSON as well as XML;
• JSON->XML conversion. (XML->JSON conversion is done and looks good.)

Both of these should be in place given a few more hours of effort. Then testing and bug fixing.

[wendellpiez]

Update May 14 2020

More progress in the working branch. We are nearly feature-complete for all the necessary functionality for Metaschema-aware XML->JSON path mapping, needed for both the JSON-to-XML converter and the JSONatron side of Metatron (Schematron over JSON). Testing, however, is showing a few bugs remaining to be fixed.

Following this there remains significant work integrating all the pieces and providing functional interfaces so these routines can be run independently and under CI/CD.

[wendellpiez]

Update May 21 2020

Despite discovery of new requirements, we are even closer to being feature complete with the XML->JSON path mapping capability, with one known remaining loose end to implement (retrieving values of fields with dynamic value keys) and much testing. Then I will turn to integrating this functionality into Metatron (constraint checking functionality) and Metaschema-based bidirectional XML-JSON conversion.

[wendellpiez]

Update May 22 2020

There are no known bugs in the core path rewriting functionality. However, much testing remains to be done while we move towards integration. XSpec tests are not current either.

Next steps (addressing usnistgov/metaschema#39):

• continue testing/debugging XML (metapath)->JSON path conversion
• Metatron
  • pick up testing schematron over XML
  • implement jsonatron to run over JSON
  • test it
  • build out and test (both) schematron generation
  • bundle and provide interfaces for all schema and usable docs generation support
  • PR (addresses Requirements gathering and design for next-gen Metaschema metaschema#17, Metaschema constraints definition and enforcement metaschema-xslt#12)

Then:

• implement JSON->supermodel converter
  • test it
• build out production of transformations supporting conversion, both ways
  • test it
  • deploy single bidirectional converters per metaschema, or pairs of one-way converters?

[wendellpiez]

Sprint 31 Update June 10

Significant progress, maybe half way through the first item. OSCAL schemas are now valid to the new Metaschema schema and usage conventions. I have also added support for the uuid datatype. However I am hitting a wall on running the schema-generation unit testing (will discuss with @david-waltermire-nist).

This Issue tracks large items. Maybe we need to track them in more detail in the Metaschema repo.

[wendellpiez]

Sprint 31 Update June 18

Bullet point 1, integrating the schemas, is paused. Needs attention from @david-waltermire-nist to address the scripting to set up and test new metaschemas with new generation tools out of the OSCAL repo. OSCAL metaschemas have been converted and look good so far.

Meanwhile, I've made a bit more progress on bullet point 2, docs generation, repairing small bugs found in the JSON map. Will also need assistance on the design aspect of this work item.

[wendellpiez]

With help from @david-waltermire-nist the unit tests are now running.

Unit tests have been updated to the new Metaschema model, and currently run successfully. Further testing is always in order.

PR usnistgov/metaschema#61 integrates all the Metaschema work to date including:

• Working XSD and JSON Schema production from Metaschemas (subject to testing)
• Old (ported) docs production (needing redesign and rework as described above)
• Provisional (alpha) implementation of constraints enforcement functionality (needs testing)
• Provisional (alpha) implementation of XML->JSON conversion pipeline
• Misc goodies including metaschema-analytic XSLT, CSS production....

It needs further work integrating, developing and testing, which I am continuing in the feature branch. It can also be pulled in order to make the work available to other devs.

[wendellpiez]

Sprint 31 Update July 1

I have continued with testing and bug fixes; this week's commits address the constraints-checking functionality on the XML side as well as mapping XPaths addressing the XML into XPaths addressing the JSON.

Moving forward to implement the last remaining piece of functionality -- the JSON-to-XML conversion -- may be in order, even before integrating everything else. It is both necessary for a complete package (including the CI/CD testing of round-trip conversions), and could be useful for producing samples for further testing.

I also have a provisional design in mind for implementing code generation logic that can be parameterized to produce transformations that operate on either JSON or XML or both (for example in support of the conversions or the constraints checking). Since the logic for these Metaschema productions is exactly parallel, such an implementation could save significantly on maintenance. Will discuss with @david-waltermire-nist. Since the logic on the XML side is already built, this approach could also save work the JSON side.

[wendellpiez]

Sprint 32 Update July 8 2020

This week has seen significant progress in the Metaschema-based JSON->supermodel conversion capability. An XSLT now produces a conversion utility from a Metaschema. It parallels the functioning XML conversion utility (and shares a code base, as suggested above), except with all the path expressions mapped over (using Metapath casting logic) to address a JSON representation.

In this process, I have also debugged the Metapath casting in several outlier cases such as fields and assemblies grouped explicitly in the XML (group-as/@in-xml='GROUPED'); values hidden in JSON keys (e.g., where a key in the JSON represents a flag value, not a name); etc. XSpec-, XSLT- and XProc-based regression tests are helping keep things on track.

Sequencing this transformation with the XML->supermodel and supermodel serialization transformations, a testing pipeline now executes a round trip conversion end to end -- it is still lossy until the handling is fully adjusted to support all the features, but able to expose any errors.

When this conversion is working, pulling all the steps into a set of pipelining XSLTs will be next.

[wendellpiez]

Sprint 32 Update July 14

Support for bi-directional conversion between XML and JSON alternative notations (of metaschema-valid data) is now feature complete and ready for more exhaustive testing.

OSCAL metaschemas are also available and ready for testing, deployment and further development.

Assuming testing holds up with real data/schemas, we are ready to discuss docs redesign/refurb.

Summary of goals (v M4 Metaschema):

• Schema production - XSD and JSON schema
• Converter production - both ways
• Docs production
  • Stabilize refactored design
  • Integrate new pipeline
  • Build out for new features
• Constraints enforcement (ready to test)
  • over XML
  • over JSON

[wendellpiez]

Further comment: our path mapping has also exposed a bug in Saxon. See

https://saxonica.plan.io/issues/4645

We can avoid this bug in our (JSON-supporting) transformations by not using SINGLETON_OR_ARRAY in the JSON syntax.

Interestingly, the bug appears in recent Saxon-HE but not (reportedly) in Saxon-JS.

[wendellpiez]

Update July 23

Working with @david-waltermire-nist to get new scripts (transformations) integrated into a dev branch of the OSCAL repo.

In related work, we implemented an XSLT runtime shell for Schematron, which should make it easier to apply Schematrons in general with less overhead. So scripts that invoke Schematron may need to be adjusted.

[wendellpiez]

To be superseded by #721

[david-waltermire]

Closing this issue based on work merged in December 2020. Continuing constraint work in #856.