Move web content from OSCAL, and other streamlining. #1824

Compton-US · 2023-06-28T17:33:59Z

Committer Notes

In support of Issue #1802

Currently in rapid prototype mode with the team.

Website content moved to OSCAL-Pages project. This project will build and publish the public website.
Reference content moved to OSCAL-Reference project. This project will build and publish the developer documentation. AKA OSCAL Reference.
OSCAL repo in this branch has been reduced to only the model content.

Remaining to determine:

Linking up the websites so that it is least disruptive to the public.
The generated xml and json content has been removed from this branch. We need to talk about this, and what is the "right" way moving forward. I think there are a number of options and opinions to consider.
Figuring out the entry point into the public website. (linking, landing page, etc)

Benefits:

Less activity on the OSCAL project unrelated to models.
Will make it easier for the community to monitor.
More focused workstreams between model development, website content/education and experiments with models.
Depending on how we publish the schemas-and-converters artifact, may make it easier for the public to access and use releases of OSCAL with less tooling.
Reference will be in position to be updated and enhanced over time for development use.
Public website can update at-will without impacting the OSCAL model project.

All Submissions:

Have you selected the correct base branch per Contributing guidance?
Have you set "Allow edits and access to secrets by maintainers"?
Have you checked to ensure there aren't other open Pull Requests for the same update/change?
Have you squashed any non-relevant commits and commit messages? [instructions]
Do all automated CI/CD checks pass?

By submitting a pull request, you are agreeing to provide this contribution under the CC0 1.0 Universal public domain dedication.

(For reviewers: The wiki has guidance on code review and overall issue review for completeness.)

Changes to Core Features:

Have you added an explanation of what your changes do and why you'd like us to include them?
Have you written new tests for your core changes, as applicable?
Have you included examples of how to use your new feature(s)?
[x ] Have you updated all OSCAL website and readme documentation affected by the changes you made? Changes to the OSCAL website can be made in the docs/content directory of your branch.

nikitawootten-nist · 2023-07-06T16:39:52Z

Hi Chris, as part of this PR I'd like to propose some additional removals/changes:

src/assets: Layer diagrams can be retired or moved to the OSCAL Pages site?
src/content: Already marked as moved to oscal-content repo'
src/metaschema/examples: A better place for this is in the metaschema repo
src/utils: Should be reorganized:
- util: Contents should be absorbed into src/utils (no need for src/utils/util)
git-cheat-sheet.md: Can be moved safely to the wiki
slides-2023-02-09.pdf: Should be removed

aj-stein-nist

This is looking good so far. I have added a few minor comments. In addition, I am not sure deleting xml/README.md and json/README.md is aggressive enough for a recursive delete if this makes sense, let me know.

.github/workflows/link-check.yml

aj-stein-nist · 2023-07-06T16:42:25Z

.github/workflows/status-ci-cd.yml

@@ -28,29 +26,3 @@ jobs:
      commit_resources: true
    secrets:
      access_token: ${{ secrets.COMMIT_TOKEN }}
-  validate-website-reference:


Question, unclear if it is blocking: we need to determine how we will generate schemas in these pipelines even if they are not committed inline anymore, and that means adapting metaschema-xslt, in this PR branch or another.

I'm not sure if this has been addressed or not, or if it is still a potential issue -- @nikitawootten-nist ?

If we are planning on removing auto-comitting actions as stated in the ADR, we should probably do the following:

Streamline the schema and converter generation to be as simple to invoke as possible (another makefile?)

Modify the existing action to build the schemas and converters without committing for PRs

Also make the action upload schema artifacts on release

For existing releases we'll have to manually upload the correct artifacts (which is simple but a bit tedious, thankfully we only have a few releases so it shouldn't be too painful)

Should we align the ADR draft to support this?

This work has been summarized in a follow-on issue: #1847

.github/workflows/workflow-validate-repo-markdown.yml

iMichaela · 2023-07-08T00:06:53Z

@aj-stein-nist and @Compton-NIST:
The file ./versioning-and-branching.md in the https://github.com/usnistgov/OSCAL/tree/feature-1802-oscal-pages branch reads:

All individual work will be done in branches in a personal fork of this repository.
Personal branches should be named using the convention <issue #>-brief-dashed-name.
Once work is complete on a personal branch, the branch should be interactively rebased to tidy any commits. Then a PR should be opened against the target feature-* branch or the develop branch if the changes are to be included in the next release.

During my chat with AJ today around #1798, he indicated that working on a personal fork is no longer mandated. Should this file be updated?

Similar guidance exists in the CONTRIBUTING.md file, the block starting at line 72.

wendellpiez · 2023-07-10T13:10:59Z

If given the option I intend to continue to work in a personal fork.

However, I did not interpret the new guideline as discouraging this, only as not requiring it. However I note that work when not individual does not have to go in a personal fork, the alternative presumably being a feature branch in the main repo.

Maybe we only need guidance on when to create a feature branch vs work in a fork. 🌻

nikitawootten-nist

I've noticed a significant regression and the work I had done in #1666 and #1754 is not present:

src/utils/resolver-2018 which I had previously deleted has reappeared.
The profile resolution stylesheets no longer have the helper script and readme updates
The src/utils/testing scripts that I included in XSpec test suites script & CI #1754 is not present

iMichaela

Thank you for the fantastic work done to split the OSCAL repo into multiple content-specific repos. Kudos! I approve the PR moving the website. I have some small comments, that, if not addressed now, we can address them later when doing a fine tunning.

iMichaela · 2023-07-07T15:31:22Z

README.md

+| [OSCAL](https://github.com/usnistgov/OSCAL/)                     | The main OSCAL project that contains the source code for the OSCAL models.         |
+| [OSCAL-Pages](https://github.com/usnistgov/OSCAL-Pages/)         | The project that contains the public OSCAL website content.                        |
+| [OSCAL-Reference](https://github.com/usnistgov/OSCAL-Reference/) | The project that contains the model documentation and developer reference content. |
+| [OSCAL-Content](https://github.com/usnistgov/oscal-content/)     | The project that contains examples of OSCAL model content.                         |


For consistency reasons, do we want to rename oscal-content to become OSCAL-Content? Or maybe OSCAL-content and make all other projects follow the same convention? Consistency is what I am looking for, so we need to remember the convention and not specifics.

iMichaela · 2023-07-07T15:48:21Z

decisions/0005-repository-reorganization.md

+
+#### Content in OSCAL-Pages repository:
+
+A `published-pages` branch will also exist in OSCAL-Pages that contains the latest deployed content that is sent to the OSCAL repository.  Additionally, there is a `nist-pages` branch in the OSCAL-Pages project that contains a blank page.  This branch could be used to test a deployment of the website at `pages.nist.gov/OSCAL-Pages`, but it is important to be aware that this site could be indexed by search engines and cause confusion, as well as broken links.


What is the main branch used for? Should a description be documented?

iMichaela · 2023-07-07T16:00:39Z

decisions/0005-repository-reorganization.md

+
+#### Streamlined Build Process
+
+The new reference repository operates off of one GitHub Actions Workflow that produces the documentation from `metaschema` and uses `Hugo` to publish the website. All of the scripted aspects of the model interpretation are executed using `make`, and the `Makefile` is located in the root of the repository.


I am assuming the metaschema listed is the previous src/metaschema which appears to remain in the OSCAL repository. Is this the proposed approach in this ADR?

iMichaela · 2023-07-07T16:04:10Z

decisions/0005-repository-reorganization.md

+1. We will separate one large repository into three distinct repositories.
+   - Links to any content that is moved will break.
+
+2. We remove the schemas and converters from the repository.  


The ADR does not document where the removed schemas and converters are relocated after removing them from the OSCAL repo.

iMichaela · 2023-07-07T16:16:34Z

decisions/0005-repository-reorganization.md

+- OSCAL
+    - We should notify the public of the pending release, and the changes noted in this ADR.
+        - For developers (and products) that link directly to content in the OSCAL repository, they may update their links to point to tagged releases, rather than using source files in the main branch.  Future dependencies linking directly to source code in the OSCAL repository, particularly `main`, should be discouraged since this could impact the ability of the development team to improve source code as necessary to build OSCAL.
+        - Future links to schemas and converters should use release assets (see [v1.0.6](https://github.com/usnistgov/OSCAL/releases/tag/v1.0.6) for an example of assets). We currently produce zip archives as a part of releases.  We intend to also include the schemas and converters directly in the release assets without having to extract from the zip file.  **This will be implemented in a future sprint.**


Guidance for submitting issues against different schemas is not captured in this ADR.

iMichaela · 2023-07-07T23:36:29Z

CONTRIBUTING.md

@@ -83,10 +83,6 @@ The OSCAL project uses a typical GitHub fork and pull request [workflow](https:/
 This repository consists of the following directories and files pertaining to the OSCAL project:



Missing decisions subdirectory from the list of subdirectories.

iMichaela · 2023-07-07T23:43:34Z

README.md

The file indicates that the NIST team is maintaining FedRAMP baselines... I think we should update the information:

The NIST team is also maintaining OSCAL content that is updated to the latest OSCAL revision. The [OSCAL content repository](https://github.com/usnistgov/oscal-content/) provides OSCAL examples, in addition to: The [NIST SP 800-53 revision 5 catalog](https://github.com/usnistgov/oscal-content/tree/main/nist.gov/SP800-53/rev5) and the security and privacy [NIST SP 800-53B baselines](https://github.com/usnistgov/oscal-content/tree/main/nist.gov/SP800-53/rev5). The [NIST SP 800-53 revision 4 catalog](https://github.com/usnistgov/oscal-content/tree/main/nist.gov/SP800-53/rev4) and the [three NIST SP 800-53 revision 4 baselines](https://github.com/usnistgov/oscal-content/tree/main/nist.gov/SP800-53/rev4). The [FedRAMP SP 800-53 revision 4 baselines](https://github.com/GSA/fedramp-automation/tree/master/dist/content/rev4/baselines).

The base branch was changed.

- Content was deleted and distributed between OSCAL-Pages and OSCAL-Reference. - Unnecessary scripts and code were pruned from the repo. - Workflows were reduced as necessary.

Compton-US · 2023-07-13T16:54:12Z

I'm going to close this and reopen with the feature branch that was not squashed so that we have a more clean rebase to the updated develop branch.

Compton-US requested review from aj-stein-nist and iMichaela as code owners June 28, 2023 17:33

aj-stein-nist reviewed Jul 6, 2023

View reviewed changes

Compton-US requested a review from wendellpiez as a code owner July 7, 2023 12:49

Compton-US mentioned this pull request Jul 10, 2023

Revise branching and forking in contributions and versioning markdown. #1846

Closed

Compton-US force-pushed the feature-1802-oscal-pages branch from 2d8b7f9 to bf9b66c Compare July 10, 2023 14:07

Compton-US changed the title ~~[DRAFT] Move web content from OSCAL, and other streamlining.~~ Move web content from OSCAL, and other streamlining. Jul 10, 2023

nikitawootten-nist requested changes Jul 10, 2023

View reviewed changes

iMichaela previously approved these changes Jul 10, 2023

View reviewed changes

nikitawootten-nist mentioned this pull request Jul 10, 2023

Revamp CI/CD to use the new metaschema-xslt, generate artifacts on release #1847

Closed

8 tasks

nikitawootten-nist changed the base branch from main to develop July 11, 2023 17:25

Compton-US added 2 commits July 12, 2023 14:03

Adjustment of OSCAL project to support multi-repository approach

fc22bd7

- Content was deleted and distributed between OSCAL-Pages and OSCAL-Reference. - Unnecessary scripts and code were pruned from the repo. - Workflows were reduced as necessary.

Adjustment of OSCAL project to support multi-repository approach

2e3e889

- Content was deleted and distributed between OSCAL-Pages and OSCAL-Reference. - Unnecessary scripts and code were pruned from the repo. - Workflows were reduced as necessary.

Compton-US force-pushed the feature-1802-oscal-pages branch from 74e4436 to 2e3e889 Compare July 12, 2023 19:16

Chris Compton and others added 7 commits July 12, 2023 15:13

Revising commits based on rebase to develop.

82181bd

Remove example file.

a224501

Cleanup based on pipeline updates.

339f1fc

Publishing generated metaschema resources [ci skip]

f4aa6ae

Update metaschema.

960f972

Publishing generated metaschema resources [ci skip]

318bd17

Stop auto-commit in main CI/CD

bfdc2bf

Compton-US closed this Jul 13, 2023

Compton-US mentioned this pull request Jul 13, 2023

Move web content from OSCAL, and other streamlining. (Revised for Release) #1856

Merged

7 tasks

Compton-US deleted the feature-1802-oscal-pages branch July 13, 2023 17:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move web content from OSCAL, and other streamlining. #1824

Move web content from OSCAL, and other streamlining. #1824

Compton-US commented Jun 28, 2023 •

edited

Loading

nikitawootten-nist commented Jul 6, 2023

aj-stein-nist left a comment

aj-stein-nist Jul 6, 2023

Compton-US Jul 10, 2023

nikitawootten-nist Jul 10, 2023 •

edited

Loading

nikitawootten-nist Jul 10, 2023

iMichaela commented Jul 8, 2023 •

edited

Loading

wendellpiez commented Jul 10, 2023

nikitawootten-nist left a comment •

edited

Loading

iMichaela left a comment

iMichaela Jul 7, 2023

iMichaela Jul 7, 2023

iMichaela Jul 7, 2023

iMichaela Jul 7, 2023

iMichaela Jul 7, 2023

iMichaela Jul 7, 2023

iMichaela Jul 7, 2023

Compton-US commented Jul 13, 2023


		#### Content in OSCAL-Pages repository:

		A `published-pages` branch will also exist in OSCAL-Pages that contains the latest deployed content that is sent to the OSCAL repository. Additionally, there is a `nist-pages` branch in the OSCAL-Pages project that contains a blank page. This branch could be used to test a deployment of the website at `pages.nist.gov/OSCAL-Pages`, but it is important to be aware that this site could be indexed by search engines and cause confusion, as well as broken links.


		#### Streamlined Build Process

		The new reference repository operates off of one GitHub Actions Workflow that produces the documentation from `metaschema` and uses `Hugo` to publish the website. All of the scripted aspects of the model interpretation are executed using `make`, and the `Makefile` is located in the root of the repository.

		@@ -83,10 +83,6 @@ The OSCAL project uses a typical GitHub fork and pull request [workflow](https:/
		This repository consists of the following directories and files pertaining to the OSCAL project:

Move web content from OSCAL, and other streamlining. #1824

Move web content from OSCAL, and other streamlining. #1824

Conversation

Compton-US commented Jun 28, 2023 • edited Loading

Committer Notes

All Submissions:

Changes to Core Features:

nikitawootten-nist commented Jul 6, 2023

aj-stein-nist left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nikitawootten-nist Jul 10, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

iMichaela commented Jul 8, 2023 • edited Loading

wendellpiez commented Jul 10, 2023

nikitawootten-nist left a comment • edited Loading

Choose a reason for hiding this comment

iMichaela left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Compton-US commented Jul 13, 2023

Compton-US commented Jun 28, 2023 •

edited

Loading

nikitawootten-nist Jul 10, 2023 •

edited

Loading

iMichaela commented Jul 8, 2023 •

edited

Loading

nikitawootten-nist left a comment •

edited

Loading