Develop OSCAL Modular Approach Strategy

[brian-ruf]

User Story:

As an OSCAL Specification Designer, I need to define the OSCAL specification such that I can re-use model information in more than one schema, as well as allow stakeholders to extend OSCAL without violating the core syntax. To accomplish this approach, we need to design a strategy around modularizing models.

Goals:

• Ensure features can be developed separately, integrated easily, and reused without unnecessary duplication.
• Ensure an organizational implementing OSCAL can extend the specification to meet information needs that satisfy the organization's unique compliance needs.
• OSCAL specifications such as prose, metadata, and attachments can be easily re-used across OSCAL layers without duplication.
• Review issues OSCAL Implementation: Core System Metadata #213, OSCAL Implementation: System Specification #214, OSCAL Implementation: Component Specification #215, OSCAL Implementation: Component Definition #216, and OSCAL Implementation: Capability Specification #217 and either close or modify these issues based on the developed strategy.

Dependencies:

This depends on a cleaner understanding of what is "in-scope" for the OSCAL specification.
This also depends on establishing a formal baseline for the OSCAL metaschema capabilities.

Acceptance Criteria

• A strategy is documented that describes specific modularity features to be developed in OSCAL.
• Organizations adopting OSCAL, such as FedRAMP, can clearly understand how to extend OSCAL to meet unique information needs while ensuring the integrity of the core OSCAL specification within files that use the extensions.

[wendellpiez]

Actually there are two issues here:

• Developers need to know how to produce a new schema with new elements for their applications - as @brianrufgsa has done for FedRAMP's SSP - by using the metaschema infrastructure. Among other things this means documenting and perhaps refining current support for modularity.

• Developers also need to know how to introduce and manage new semantics without introducing new elements. So you can use one of our metaschemas (for the appropriate layer) but with new flag values and/or other (schema-valid) indicators to distinguish your data points (values or chunks).

The second approach is called "extension by restriction" and permits developers to develop without having to retool at lower levels. It is consistent with what we are building with Schematron and the declarations model, to support consistency-checking (for semantic integrity) without having to add new elements or structures to the schema. This is a clean and lightweight approach that has worked well in the context of standard tag sets such as DITA or JATS, which span broad application domains.

Both these approaches probably need to be documented and demonstrated.

[wendellpiez]

I have ideas about how to extend the functionality. It should be possible to make the design for including a Metadata module (or an Attachments or Back Matter module) more like the design for prose. In other words you have a way to say "root of assembly defined in imported module goes here" and forget about the rest.

A question however is whether to keep the present capability to include and hence override definitions one at a time. Having both features may be unwanted complexity.

I need a sounding board, at least @david-waltermire-nist or @brianrufgsa or both.

[wendellpiez]

Sprint 18 Progress Feb 28 2019

@brianrufgsa and I can make some more progress on this once his model is stable and we have shareable data. (I think I have a better idea of requirements and a design to address them.)

[iMichaela]

02/28/2018

@brianrufgsa and @wendellpiez will meet next week to work on this issue.

[wendellpiez]

WIP is here: https://github.com/wendellpiez/OSCAL/tree/feature-metaschema-d3

Changes include a couple of improvements described in #267.

[wendellpiez]

Sprint 18 progress March 7 2019

Work on retooling the modularity functionality of the metaschema is proceeding in my repo (linked above). The new model is much simpler to use and explain.

Next up is to make the modifications to the catalog and profile metaschemas, produce new artifacts for those models (schemas, converter XSLTs, docs) using the tools, and examine and test these outputs.

There is a cross-dependency here on Issue #286, since one of the processes to be updated is the documentation pipeline.

% complete: 60% - design and testing of new version is mostly done; testing, review and acceptance remains. Will need @david-waltermire-nist and @brianrufgsa at least for that final phase.

[brian-ruf]

3/7/2019 - We have exceeded the acceptance criteria with actual implementation.
@brianrufgsa and @david-waltermire-nist need to review, test, and validate the changes.
Setting up meeting for 3/13.

[wendellpiez]

Basically the story here is, I cannot make a design without testing it, and I can't test it without building it, so I apologize :-), but since the solution I want to offer does not add complexity but removes it, I thought it would be better and faster to make my proposal in the form of a PoC.

However, nothing is rolled out into catalog or profile schemas yet -- no changes to metaschemas except the testing metaschema. XSLTs have been updated for production of XML and JSON schemas and conversion utilities, but not tested. The Schema Docs production pipeline has not yet been updated (but the code is modular all I have to do is drop it in).

For purposes of review, we could examine and discuss the test metaschema (here: https://github.com/wendellpiez/OSCAL/blob/feature-metaschema-d3/schema/demo/oscal-test-metaschema.xml). Or (since there's not much to see) I could simply sketch the changes, which are pretty easy to summarize:

• No more @show-as='xml json' since its semantics haven't been worked out for the current docs design. We can put it back when we need it.
• No more need for stating @group-as on fields and assemblies references where it can be determined statically by looking at the definition.
• Much simpler definitions importing. Having imported a metaschema module (no further configuration), you get any/all of the definitions in the imported module. Any 'orphan' definitions (never picked for an assembly by some definition descending from the root definition) are left out of the schema. Definitions take precedence in ascending order: all imports appear first, then all local definitions, any of which can override earlier definitions given earlier in any module. (And same thing All The Way Down.)

@david-waltermire-nist @brianrufgsa @anweiss feel free to examine or ask questions in advance of next discussion of this issue.

[brian-ruf]

These are the notes from today's meeting between @david-waltermire-nist, @brianrufgsa , and @wendellpiez.

Metaschemas:
We see the following core and supporting Metaschemas:
Core:

• Catalog (to be included in the milestone release)
• Profile (to be included in the milestone release)
• Implementation (to be released later)
• Assessment (to be drafted later)
• Assessment Results (to be drafted later)

Supporting (Modular across multiple Core metaschemas. All included in the milestone release) :

• Metadata (document properties, roles, etc.)
• Resources (references, attachments)
• Prose (more of a data type, but treated as modular)

Prose Includes

• Markup/down
• Paragraphs, Line-breaks
• Tables
• Bold, Italics
• Lists
• Headers
• Code Blocks
• Links
• Images

The following activities are needed

1. Create/revise individual metaschemas
2. Integrate metaschemas
3. Update examples and content
4. Document models and integrations

Resource Details:

• ID
• href (pointer) OR content (structured or Base64)
• type (normal ref, informational ref, artifact, etc.)
• mime-type
• bibliographic information
  • name
  • date
  • version
  • description

General Structure of an OSCAL File

• Metadata at top
• Layer-specific content in the middle
• Resources at bottom (References and attachments)

[wendellpiez]

In addition, Wednesday's meeting revealed a couple of requirements for schema documentation, namely that when a metaschema imports definitions from another metaschema, it also needs to be able to qualify definitions with (a) language regarding usage (in the context of the new schema) and (b) examples in new contexts.

Additionally, we identified a couple of other nice-to-haves for schema documentation: a definition for an element or construct should say which elements (assemblies) use it; and imported definitions should identify the metaschema in which they are defined.

[david-waltermire]

3/18/2019

This issue has been completed.