Build out and test bidirectional XML and JSON conversion

[wendellpiez]

User Story:

• As an OSCAL user, I should be able to see both XML and JSON examples for catalogs and profiles;
• As a producer and publisher of catalog and profile data, I need a way to produce JSON equivalents for my XML source (or vice versa);
• As a developer using OSCAL, I need access to tools for making JSON from my catalog or profile XML and XML from my JSON; these tools need to have been tested and shown to be feature-complete for a v1;
• In particular, supported functionality includes conversion into XML of prose from markdown-embedded-in-JSON (in OSCAL formats) as well as the (relatively trivial) conversion from XML into Markdown.

Goals:

• Demonstrate and test conversion of XML into JSON and correspondingly back again using metaschema-driven tools (XSLTs produced from a metaschema);
• Use both mockup/pathological data and real/canonical data for demonstration of this feature;
• Be positioned to integrate automated conversions into processing pipelines including CI/CD.

Dependencies:

Current metaschema work should be stable and merged in (#339), with any downstream branches updated. We can start using the demo metaschema for testing including unit testing.

At some point we need to address whether JSON emitted from this pipeline is adequate (with respect to syntax conventions) or whether some post-processing might be called for. (Seek advice from stakeholders with JSON requirements.)

For examples of profile and catalog data we need a list of canonical files to be converted (for production of JSON variants from XML source) plus any test/mockup files for testing catalog and profile conversions specifically (inside or outside CI/CD).

Supporting bidirectional conversion outside the catalog and profile layers is not a goal, but it should be enabled implicitly by the metaschema work, and ready to test as soon as data is available for any given metaschema.

Acceptance Criteria

1. XSLTs can be produced programmatically to convert XML into JSON for any given OSCAL model, from its metaschema, and to convert JSON back into XML. This can be either two XSLTs per model, or a single XSLT that can detect which direction to go from the input given.
2. We have unit tests that show success in bidirectional conversion. (We can use XSpec for unit tests.) Any lapses in bidirectional conversion are exposed by the tests. The unit tests are reasonably comprehensive of metaschema structures (assemblies, fields etc.); in particular they include prose, with examples of all OSCAL-valid block-level and inline-level elements (structures) within prose.
3. We consider conversions ready to integrate into the CI/CD handling data on the catalog and profile layers (for a new Issue).
4. We have a plan for addressing remaining issues in bidirectional conversion, with new Issues as appropriate. This may include documentation and/or scripts for users to run the conversions on their data.

[howieavp76]

I now have the scripts in place for testing that can round trip XML->JSON->XML. It can also do the inverse for JSON->XML->JSON. Testing for "loss" is more difficult than expected and some of the newer content is not coming over in the conversions. Still a lot of work to do to troubleshoot differences and make the process repeatable and reliable.

[iMichaela]

5/2/2019

@howieavp76 made good progress on thsi issue. @wendellpiez is looking at the lower level details. @howieavp76 finished the round-trip test script, but needs to work on a script to run the process to creat new artifacts for testing - this will be integrated with CircleCI.

[wendellpiez]

Next to do: build out examples and unit test for low-level conversions including prose. Since none of the deployed catalogs or profiles contain mixed content (face markup) in prose, this should not affect deployed conversions (but it will be needed as soon as we have any).

[david-waltermire]

5/16/19

Most of this functionality is working now, except tables. The bi-directional testing that @howieavp76 is working (#355) on will be used to test the accuracy of these conversions.

As follow-on work, we need mini-testing content to exercise the markdown <> html conversion to make sure it works and does not regress.

[david-waltermire]

This will be implemented in the CI/CD process in PR #358 using the existing converters.

[wendellpiez]

Code to convert OSCAL Markdown to XML via XSLT is now feature complete, including tables and images. It needs testing and integration into the current JSON-to-XML convertor generator pipeline.

[iMichaela]

5/23/2019

This issue depends on #283. Once #283 is done, this can move forward. @howieavp76 and @david-waltermire-nist will work on integrating it into CI/CD on Friday (tomorrow).

[david-waltermire]

We are going to hold this at "reviewer approved" until #283 is done and then close this issue.

[howieavp76]

See PR #405 for test results. Some of the conversions have errors that we should address.

[david-waltermire]

This is now being worked in PR #410. Round-trip errors need to be addressed before that PR can be pulled.

[david-waltermire]

PR #410 is now merged. Closing this issue.