Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Complete Metaschema Implementation #186

Closed
wendellpiez opened this issue May 24, 2018 · 8 comments
Closed

Complete Metaschema Implementation #186

wendellpiez opened this issue May 24, 2018 · 8 comments
Assignees
@anweiss @david-waltermire
Labels
LoE: Large Scope: Metaschema Issues targeted at the metaschema pipeline User Story
Milestone
OSCAL 1.0 Milesto...

Comments

@wendellpiez
Copy link
Contributor

wendellpiez commented May 24, 2018 •
edited by david-waltermire
Loading

User Story:

As an OSCAL stakeholder, I can validate that OSCAL metaschema components work correctly by consulting unit tests.

Goals:

OSCAL metaschema functionality has (at least) three discrete areas:

  • Produce and publish functioning and correct XSD and JSON schemas for OSCAL from metaschema source. (@wendellpiez)
  • Support bidirectional conversion between XML and JSON via metaschema configuration. Inasmuch as metaschema contains all the info needed for a complete XML-JSON element-to-object mapping, a conversion tool can be dynamically generated or configured and doesn't have to be hard wired. (@wendellpiez)
  • Document a user story for supporting bidirectional conversion of text markup (e.g., bold, italics, lists) as follow-on work. (@david-waltermire-nist)

The first two of these would benefit from unit testing to ensure requirements are met and functionality is stable under maintenance. We now have unit testing in place for the XSD production (first half of item 1).

Non-Goals

  • Support for bidirectional conversion of text markup (e.g., bold, italics, lists) will not be addressed in this issue for XML or JSON.

Dependencies:

  • Parameter insertion JSON template syntax #194 is about designing an approach for parameter insertion in JSON Markdown OSCAL content. Documenting this design is a precursor for this issue.
  • Build out the functionality to the point where testing makes sense. (At time of writing this is in progress.)

Acceptance Criteria:

  • Schema generation for Catalog and Profile XML and JSON schemas can be scripted, and up-to-date schemas are published.
  • Conversion of XML to JSON, and JSON to XML can be demonstrated, also using a scripted approach.
  • Unit tests are in place (1) for production of XSD and JSON Schema from metaschema source and (2) for bidirectional conversion of OSCAL-conformant XML or JSON into its "counterpart" format.

Note: the unit tests do not all have to pass for acceptance of this user story.
@wendellpiez wendellpiez mentioned this issue May 25, 2018
Closed
@david-waltermire david-waltermire changed the title User Story: metaschema unit tests Complete Metaschema Implementation May 25, 2018
This was referenced May 29, 2018
Closed
Merged
@david-waltermire david-waltermire self-assigned this May 31, 2018
@wendellpiez wendellpiez mentioned this issue Jun 7, 2018
Closed
@wendellpiez
Copy link
Contributor Author

wendellpiez commented Jun 7, 2018 •
edited
Loading

Sprint 11 Progress Report June 7 2018

Progress on goals

  1. Produce XSD and JSON schemas reflecting metaschema models - 100%
  2. Develop support for bidirectional XML-JSON conversion on basis of metaschema mapping - 5%

Ongoing work is here:

@david-waltermire
Copy link
Contributor

2018-08-07 Meeting Notes

@wendellpiez will create a new feature branch and PR to post the update schema files ASAP.

@wendellpiez
Copy link
Contributor Author

The feature-metaschema branch was so close, that I opted to clean it up and post the PR from there.

This was referenced Jun 7, 2018
Merged
Closed
@wendellpiez
Copy link
Contributor Author

wendellpiez commented Jun 13, 2018 •
edited
Loading

Sprint 11 Progress Report June 13

Since last Friday when a PR was posted (#201), there have been many changes and improvements in this branch. It probably needs to be squashed again.

  • XSD and JSON Schema generation from Metaschema source: 90%, with some smaller features remaining in JSON Schema but more testing also required (see Finishing and testing the edges of metaschema support for JSON #202)
  • Metaschema-based production of respective conversion utilities (two separate XSLTs): getting close to 100% but even less tested
  • Unit tests: 50% (many unit tests are in place but not built out)
  • OSCAL Metaschemas: 80%: Catalog and Profile metaschemas exist and can be tested; the OSCAL Declarations model has no metaschema yet. (One consequence being that catalogs with their declarations included are not valid to the metaschema catalog XSD. This could be better however, tbd)
  • Scripts: a single (bash shell) script supports production of all (5) artifacts (two schemas, two transformations, a docs file) from a single metaschema instance.

Summary: there is some remaining work to be done on the metaschema infrastructure but it will really improve only under use, and we also have much documentation to write / examples to produce.

Need help updating docs / producing examples. Will happily assist anyone else editing the metaschema. We also have XSLT/CSS to help support an editing cycle.

@david-waltermire david-waltermire mentioned this issue Jun 19, 2018
Closed
@david-waltermire
Copy link
Contributor

6/21/2018 Status

  • The README in [WIP] Feature metaschema #201 build/metaschema needs to be updated on how to use the stylesheets for this.
  • We also need to document the features of the metaschema.

@wendellpiez
Copy link
Contributor Author

Sprint 11 Progress Report, June 28 2018

The readme.md file looks better than it did. Also, there is a new markdown file with an initial cut at some metaschema documentation.

https://github.com/wendellpiez/OSCAL/blob/feature-metaschema/build/metaschema/readme.md
https://github.com/wendellpiez/OSCAL/blob/feature-metaschema/schema/metaschema/design-notes.md

Both are included in PR #201. However, there is (plenty) more work to do on documentation, perhaps with a new User Story (since docs are not actually mentioned as acceptance criteria on this one).

Percent complete: ? - depending on how you read the Issue 😄

wendellpiez added a commit to wendellpiez/OSCAL that referenced this issue Jun 28, 2018
@wendellpiez
More improvements to readme see usnistgov#186
381b095
@iMichaela
Copy link
Contributor

July 18 Status Update (Sprint 12 Acceptance Meeting)
The issue is waiting for a review from @david-waltermire-nist and/or @iMichaela

iMichaela pushed a commit that referenced this issue Jul 26, 2018
@wendellpiez @iMichaela
[WIP] Feature metaschema (#201)
4165610 
* Creation of initial OSCAL documentation.
Migration to Slate.

* Update README.md

Corrected a typo

* Docs branch (sprint 9) (#134)

Updated documentation as part of Sprint 9. Changes include implementation of the Slate framework,  many improvements to the XML documentation, and creation of JSON documentation.

* fix relative linking and code sample alignment for pages.nist.gov (#141)

* dev layout, styling fixes and README updates (#149)

* Moved the GitHub link to a a more prominent place in the left navigation (#161)

* Updating OSCAL component image #140 (#170)

A content change with an updated components image and the base Visio artifacts. This addresses #140.

* This is a combination of 29 commits, one of which combined 50 earlier commits. In addition to the metaschema feature, this commits provides major reorganization, including removal of much extraneous material (dev clutter).

All core features of metaschema format are now supported, including schema generation (XSD and JSON Schema), conversion utility generation (two XSLTs, one for each direction) and docs are all generated from metaschema sources. Metaschemas supporting OSCAL catalogs and profiles are offered.

* Now adding profile converters

* Cosmetic touches

* Adjustments to metaschema functionality

* Fixed readme

* Started to merge documentation for the pages site.

* Improvements to metaschema XSD and its display XSLT

* More rearranging subdirectories

* Completed merge of catalog documentation from the pages.nist.gov/OSCAL site

* Completed merge of catalog documentation from the pages.nist.gov/OSCAL site

* Added environment variable for SAXON_HOME

* More restructuring resources; some improvements / enhancements and a couple of bug fixes / extensions

* Making room for new shell script

* Removed leftovers; edited a readme

* Metaschema enhancements

* More adjustments to metaschema

* Further enhancements

* Still making corrections

* This is a combination of 29 commits, one of which combined 50 earlier commits. In addition to the metaschema feature, this commits provides major reorganization, including removal of much extraneous material (dev clutter).

All core features of metaschema format are now supported, including schema generation (XSD and JSON Schema), conversion utility generation (two XSLTs, one for each direction) and docs are all generated from metaschema sources. Metaschemas supporting OSCAL catalogs and profiles are offered.

* Now adding profile converters

* Cosmetic touches

* Fixed readme

* Adjustments to metaschema functionality

* Started to merge documentation for the pages site.

* Improvements to metaschema XSD and its display XSLT

* Completed merge of catalog documentation from the pages.nist.gov/OSCAL site

* Completed merge of catalog documentation from the pages.nist.gov/OSCAL site

* Added environment variable for SAXON_HOME

* More restructuring resources; some improvements / enhancements and a couple of bug fixes / extensions

* Making room for new shell script

* Removed leftovers; edited a readme

* Metaschema enhancements

* More adjustments to metaschema

* Further enhancements

* Still making corrections

* Starting to develop parameter documentaiton

* Added parameter documentation.

* Addressing #175

* synching up

* Removing lib files now in util directory

* Now Issue #175

* Restoring full names (labels) on subcontrols

* Further adjustments for #175

* Metaschema schema changes to make yesterday's extensions to the catalog valid; also a couple of small corrections to the catalog metaschema.

* Metaschema schema changes to make yesterday's extensions to the catalog valid; also a couple of small corrections to the catalog metaschema.

* Fixing little glitch

* Now supporting a separate JSON-oriented documentation track (markdown next to the JSON schema)

* More extensions / improvements including JSON Schema XSpec

* More updates and improvements to unit tests and to converter generators, to current metaschema design

* Further updates and refinements to JSON docs

* Further refinement and glitch removal

* Readme updates

* Preliminary implementation of documentation presenting JSON examples

* Filling in missing edges; new readme for converters

* Delete README-old.md

* Create readme.md

* Added front note

* Adjusting metaschema production script and components

* Removing labels from selection parameters; added note to param documentation regarding label usage.

* More enhancements mostly of markdown docs formatting

* Relocated SP800-53 and FedRAMP examples to address #181.

* updating the remediated FedRAMP profiles.

* Updated paths in FedRamp profiles to be relative

* Removing unwanted subdirectory

* Removing unwanted (testing) files

* More rationalizing of structure this time inside build/metaschema

* Further restructuring: now a new 'test' subdir for any/all testing; also adding XSpec HTML results to .gitignore

* Refresh and cleanup

* Fixing typo in gitignore

* Now, the broken paths on the unit test docs

* More metaschema design notes; replacing errant XSLT to its correct location

* Updated SP800-53 declarations file; the catalog is valid to the revision.

* Small extensions / bug fixes

* Improved wording on warning msg

* More incremental improvements especially to build/metaschema/readme

* More improvements to readme see #186

* Improvements and refreshments

* Refreshing profile docs now to include defs from the imported catalog

* Relocated all files to the docs subdirectory in preperation for migrating to master branch.

* Small metaschema improvements

* New XML and JSON schema docs in docs subdirectory

* Refreshing documentation for new structure

* Refresh and cleanup

* More refinements to paths

* Refreshed, this time with whitespace in the JSON

* Refresh with minor bug correction

* Documentation Updates

* Correcting syntax errors.

* Now refreshing profile schema artifacts

* Ensuring text format outputs of XSLT producing markdown

* Adjusting paths in schema documentation includes

* Missed one

* Propagating XML docs glitch correction to produce markdown not html

* Update README.md
@david-waltermire
Copy link
Contributor

Need to document a user story for text conversion before completing this issue.

@anweiss anweiss mentioned this issue Aug 2, 2018
Closed
@david-waltermire david-waltermire mentioned this issue Aug 2, 2018
Closed
This was referenced Aug 2, 2018
Closed
Closed
@david-waltermire david-waltermire added Scope: Modeling Issues targeted at development of OSCAL formats Scope: Metaschema Issues targeted at the metaschema pipeline and removed Scope: Modeling Issues targeted at development of OSCAL formats labels May 9, 2019
@david-waltermire david-waltermire added this to the OSCAL 1.0 M1 milestone May 9, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@anweiss anweiss

@david-waltermire david-waltermire

Labels
LoE: Large Scope: Metaschema Issues targeted at the metaschema pipeline User Story
Projects
None yet
Milestone
OSCAL 1.0 Milestone 1
Development

No branches or pull requests
4 participants
@wendellpiez @anweiss @david-waltermire @iMichaela