Document and Diagram CI/CD Workflows

[aj-stein-nist]

User Story:

As a NIST OSCAL developer, in order to best understand the different logical steps of the development pipeline and their implementation in the OSCAL CI/CD system, I would the logical steps (and their implementation) for different forms of project automation (release management, website updates, and GitHub issue management) diagrammed to understand current state and plan future improvements.

Goals:

Clears docs and diagrams will let us understand what works well, what doesn't, and what gaps we have now, so we can consider updates in the future. This also will help outside contributors understand the workflow from recommendation to getting a change into the next production release of OSCAL.

Dependencies:

N/A

Acceptance Criteria

• All OSCAL website and readme documentation affected by the changes in this issue have been updated. Changes to the OSCAL website can be made in the docs/content directory of your branch.
• A Pull Request (PR) is submitted that fully addresses the goals of this User Story. This issue is referenced in the PR.
• The CI-CD build process runs without any reported errors on the PR. This can be confirmed by reviewing that all checks have passed in the PR.

{The items above are general acceptance criteria for all User Stories. Please describe anything else that must be completed for this issue to be considered resolved.}

[aj-stein-nist]

@david-waltermire-nist, per conversation today, I thought we would add this here and then work it when we are ready/able. We can definitely talk about this during the week as you see fit, and then bring it into sprint accordingly.

[aj-stein-nist]

From discussion with Dave via Gitter, the high-level current overview of our CI/CD processes.

OSCAL repository Quality Assurance Checks

Provided by metaschema-artifacts.yml:

1. Validate all metaschemas using the metaschema.xsd and schematron (build/cicd/validate-metaschema.sh)
2. Check that schemas properly generate (build/cicd/generate-schema.sh)
3. Check that converters properly generate (build/cicd/generate-content-converters.sh)
4. Run any content unit tests against the schemas (build/cicd/run-unittests.sh)

Provided by validate-repo-markdown.yml:

1. Use markdown-link-check to check for broken links in all repository markdown files (excluding docs/content)

Provided by website-artifacts.yml:

1. Check that the profile resolution specification documentation is properly generated
2. Generate the Hugo-based, model documentation for (main, develop, release tags)
3. Check that the site builds using Hugo.
4. Check for broken links on the website (currently only internal links)

Performing a release

1. Increment OSCAL metaschema versions

[flickerfly]

Is this about the OSCAL project specifically or about using OSCAL in another CI/CD process?

[aj-stein-nist]

Is this about the OSCAL project specifically or about using OSCAL in another CI/CD process?

The former, the OSCAL project specifically.

[aj-stein-nist]

@flickerfly, if you are interested in how to use OSCAL in a different CI/CD process, definitely open up a discussion to brainstorm ideas or an issue if you think certain enhancements are in order and/or you found some bugs that need improving!

[flickerfly]

I'll still be watching here because I'm curious to see how you folks are thinking about CI/CD.

[aj-stein-nist]

NOTE: This is obsolete, see update spec diagram below in #1121 (comment).

So, @david-waltermire-nist, are we going to pull this into sprint?

I started working a small working example in Mermaid live editor and focused on one workflow so far. How does this look so far, continue?

I am trying to write out the conceptual goals of a step (actions with arrow), and I am slowly replacing the subject of that action (usually a shell script) as a box as I figure it out and get closer to ready for one flow. Thoughts?

[aj-stein-nist]

Per conversation with Dave have some tweaks and good feedback on next steps. Will bring this into sprint to work and collab with him and work alongside other assigned tasks.

[guyzyl]

@aj-stein-nist how about using the Python library diagrams for creating the diagram?
Creating diagrams via code has a lot of benefits such as making it versioned via git, easier to modify, and can also be automatically built via the CI/CD on changes.

[aj-stein-nist]

@aj-stein-nist how about using the Python library diagrams for creating the diagram? Creating diagrams via code has a lot of benefits such as making it versioned via git, easier to modify, and can also be automatically built via the CI/CD on changes.

I will talk to the team about this. A few points.

1. We are using MermaidJS as a diagram markup language to express diagrams via code (I assume you mean a "full programming language" potentially, if so ignore this bit and you can assert this is what you really meant) and we can and will store it in version control.
2. We already have used some MermaidJS in other parts of the OSCAL projects, like this diagram in the Metaschema repo
3. I really like your idea about tracking CI/CD changes, but we have a flow of GitHub Actions jobs and tasks that orchestrate a sequence of shell commands, so connecting the diagram to the actual CI/CD code is interesting, but not easily feasible at this time (Ironically enough, I actually talked to Dave about this yesterday as an interesting idea prior to this comment, but we would have to do some very kludgey stuff with bash and python and Java invocation comments and build a whole tool ecosystem for that, it'd be a lift big for now. 😆)

[aj-stein-nist]

On that note, below is the updated the diagram from yesterday wrapping in Dave's feedback and recommendations. We will sync back up with these on Wednesday when we pair.

Live Editor view link

Rendered image:

[guyzyl]

@aj-stein-nist how about using the Python library diagrams for creating the diagram? Creating diagrams via code has a lot of benefits such as making it versioned via git, easier to modify, and can also be automatically built via the CI/CD on changes.

I will talk to the team about this. A few points.

1. We are using MermaidJS as a diagram markup language to express diagrams via code (I assume you mean a "full programming language" potentially, if so ignore this bit and you can assert this is what you really meant) and we can and will store it in version control.
2. We already have used some MermaidJS in other parts of the OSCAL projects, like this diagram in the Metaschema repo
3. I really like your idea about tracking CI/CD changes, but we have a flow of GitHub Actions jobs and tasks that orchestrate a sequence of shell commands, so connecting the diagram to the actual CI/CD code is interesting, but not easily feasible at this time (Ironically enough, I actually talked to Dave about this yesterday as an interesting idea prior to this comment, but we would have to do some very kludgey stuff with bash and python and Java invocation comments and build a whole tool ecosystem for that, it'd be a lift big for now. 😆)

1. Sounds just as good! As long as the diagrams are defined via code everything's great.
2. Same
3. This can defiantly be done via GitHub Actions. I previously did something similar with diagrams and can share the action if you'd like.

[aj-stein-nist]

3. This can defiantly be done via GitHub Actions. I previously did something similar with `diagrams` and can share the action if you'd like.

This is I want to see, can you point me to the example of how you can orchestrate GitHub Actions steps and trace it with diagrams on the fly? I am not sure we will commit to that, but I am definitely interest in seeing it!

[guyzyl]

You can see it here, it's part of the oscal-diagrams repo I created.
The purpose is so the "built" diagrams are automatically updated when a change is detected / diagram build is triggered (manually).
(and to correct my previous comment, this isn't done with diagrams but with erdantic in this one)

[aj-stein-nist]

You can see it here, it's part of the oscal-diagrams repo I created.

Ah I thought you meant something different with this script. I thought you could write code that already reads GitHub Actions job and tasks steps and diagrams what they call and how the inputs and outputs relate for the CI/CD itself, not as the core development artifact of the repo. That I had seen already. I was curious how it documented its own CI/CD steps. This is still super cool thought!

[aj-stein-nist]

Caught up with Dave today during a pairing session. He quickly reviewed the updates in the above comment and likes where they are at so far. He agreed I should continue and then work them into a PR against the build/ci-cd/README.md file when we have document the current existing state, and move forward with a plan to progressively add improvements around this work after this issue is complete.

[aj-stein-nist]

Also a note for future work and to jog my memory: Dave mentioned how we could compose future workflows that trigger each other, we can find a way to deprecate the stefanzweifel/git-auto-commit-action and improve security with token handling that is a necessary workaround.

[guyzyl]

@aj-stein-nist checkout this announcement from GitHub regarding Mermaid integration with markdown files.
Might not apply to this case, but FYI.

[aj-stein-nist]

@aj-stein-nist checkout this announcement from GitHub regarding Mermaid integration with markdown files. Might not apply to this case, but FYI.

Oh I saw! haha

[aj-stein-nist]

Per conversation with @david-waltermire-nist in weekly sync meeting and the earlier Lunch with Devs, I will touch this up and get it ready for public consumption and to drive discussion around the OSCAL team's developer tooling after Dave gauged a significant amount of interest in today's session.

[aj-stein-nist]

@david-waltermire-nist, thanks for merging the PR. In terms of next steps for the work, we close this out and say we are done with documenting the current state of affairs and open new issues around how we would want to make enhancements and improvements? Perhaps we meet and discuss low to high effort changes and what gets us the most rewards the fastest?