Updates to Testing

Remove some JSON testing custom code and pip installs for Python in favor of AJV-CLI. Added XPath to the XML comparison code. Testing for XMLDIFF. Remove Unnecessary Python Libraries No longer using several external libraries Minor Updates FIxed git ignores Changed from -v check to something more compatible Updates to Testing Remove some JSON testing custom code and pip installs for Python in favor of AJV-CLI. Added XPath to the XML comparison code. Testing for XMLDIFF. Minor Updates FIxed git ignores Changed from -v check to something more compatible Bash Generalization Fixed the -v statements to use a more portable -z statement. Added Exit Code Logic to XML Round Trip Comparison Refactor XML Comparison Removes XMLDiff Adds exit code to the Python code, does error count tracking Test Content Converer Fix XML RoundTrip in Run All Tests Round Trip XML->JSON->XML Working for all NIST 800-53 checks (FedRAMP currently bombing) Color Output Printf messages are now color coded for improved readability in large log files Temp Directory Round trip files now written to the Temp directory, this directory added to build artifacts in CI/CD Rename Temporal Files Build files for roundtrip now carry their base naming conventions for troubleshooting Add Roundtrip Check to CI/CD Initial Attempt Fix CI/CD Bug Typo on base image Local JAR Trying to get it working end to end with local file Dynamic Paths Fix CI/CD bug finding Python/Saxon scripts PIP Installs Python Import Error Fix UTF Encoding Refactor Python Get etree.parse to work in CI/CD Shorten FileNames Build Artifacts Use the supplied build directory Troubleshoot Artifacts Dynamic JAR Dynamic JAR 2 Maven Install Adds Maven to the Circle CI image Maven Attempt #2 Class Path Refactor Refactor CI/CD Faster build time CI/CD Bug Fix Fix Path Error Copy Build Artifacts Copy Artifacts Attempt #2 Artifact Try #3 Artifact Attempt #4 Copy Step Artifacts #5 Fix Tab Artifact #6 Artifact #7 Artifact #8 Artifact #9 Initial JSON Roundtrip Remove Diff Updates to Testing Remove some JSON testing custom code and pip installs for Python in favor of AJV-CLI. Added XPath to the XML comparison code. Testing for XMLDIFF. Minor Updates FIxed git ignores Changed from -v check to something more compatible Updates to Testing Remove some JSON testing custom code and pip installs for Python in favor of AJV-CLI. Added XPath to the XML comparison code. Testing for XMLDIFF. Bash Generalization Fixed the -v statements to use a more portable -z statement. Added Exit Code Logic to XML Round Trip Comparison Refactor XML Comparison Removes XMLDiff Adds exit code to the Python code, does error count tracking Test Content Converer Fix XML RoundTrip in Run All Tests Round Trip XML->JSON->XML Working for all NIST 800-53 checks (FedRAMP currently bombing) Color Output Printf messages are now color coded for improved readability in large log files Temp Directory Round trip files now written to the Temp directory, this directory added to build artifacts in CI/CD Rename Temporal Files Build files for roundtrip now carry their base naming conventions for troubleshooting Add Roundtrip Check to CI/CD Initial Attempt Fix CI/CD Bug Typo on base image Local JAR Trying to get it working end to end with local file Dynamic Paths Fix CI/CD bug finding Python/Saxon scripts PIP Installs Python Import Error Fix UTF Encoding Refactor Python Get etree.parse to work in CI/CD Shorten FileNames Build Artifacts Use the supplied build directory Troubleshoot Artifacts Dynamic JAR Dynamic JAR 2 Maven Install Adds Maven to the Circle CI image Maven Attempt #2 Class Path Refactor CI/CD Bug Fix Fix Path Error Copy Build Artifacts Copy Artifacts Attempt #2 Artifact Try #3 Artifact Attempt #4 Copy Step Artifact #7 Artifact #8 Artifact #9 Initial JSON Roundtrip Remove Diff Merge with Master CI/CD Fix #2 CI/CD Fix #4 CI/CD Fix #5 CI/CD Fix #6
wendellpiez · Jun 6, 2019 · 11d13e9 · 11d13e9
1 parent 772df5b
commit 11d13e9
Show file tree

Hide file tree

Showing 36 changed files with 126,764 additions and 100 deletions.
diff --git a/.circleci/config.yml b/.circleci/config.yml
@@ -50,6 +50,20 @@ commands:
             git clone --depth 1 --no-checkout https://github.com/Schematron/schematron.git "$SCHEMATRON_HOME"
             cd "$SCHEMATRON_HOME"
             git checkout master -- trunk/schematron/code
+  install-maven:
+    description: "Installs Maven"
+    steps:
+      - run:
+          name: Get Maven
+          command: |
+            sudo apt-get install maven
+  install-lxml:
+    description: "Installs Python lxml package"
+    steps:
+      - run:
+          name: Get lxml
+          command: |
+            sudo pip install lxml
   install-xmllint:
     description: "Retrieve xmllint and setup the environment to run it"
     steps:
@@ -92,14 +106,6 @@ commands:
           name: Get hub
           command: |
             sudo brew install hub
-  install-python-libs:
-    description: "Retrieve python libs and setup the environment to run it"
-    steps:
-      - run:
-          name: Get Python Libs
-          command: |
-            sudo pip install 'jsonschema>=3.0.1'
-            sudo pip install simplejson
   install-bundler:
     description: "Install bundler and html-proofer"
     steps:
@@ -121,16 +127,16 @@ commands:
             fi
 executors:
   java-executor:
-    docker: # run the steps with Docker
+    docker: # run the java jobs with Docker
       - image: circleci/openjdk:8-jdk-node-browsers
     <<: *common_environment 
   ruby-node-executor:
-    docker: # run the steps with Docker
+    docker: # run the ruby jobs with Docker
       - image: circleci/ruby:2.4.6-node-browsers
-    <<: *common_environment 
+    <<: *common_environment
   python-executor:
-    docker:
-      - image: circleci/python:3.7.3-node-browser
+    docker: #run the python jobs with Docker
+      - image: circleci/python:3.7.3-node-browsers
     <<: *common_environment 
 jobs: # a collection of 
   job-ci-skip-check:
@@ -246,6 +252,20 @@ jobs: # a collection of
           name: Validate content instances
           command: |
             bash "$CICD_DIR/validate-content.sh" "$OSCAL_BUILD_DIR"
+  job-roundtrip-conversions:
+    executor: python-executor
+    steps: 
+      - *checkout
+      - *attach_build_workspace
+      - install-maven
+      - install-saxon
+      - install-ajv
+      - install-lxml
+      - run:
+          name: Generate round trip XML and JSON conversions
+          command: |
+            bash "${CICD_DIR}/roundTripXML.sh" "$OSCAL_BUILD_DIR"
+      *publish_artifacts
   job-generate-docs:
     executor: java-executor
     steps:
@@ -359,7 +379,9 @@ workflows:
   build:
     jobs:
       - job-ci-skip-check
+      - job-roundtrip-conversions
       - job-validate-markdown:
+
           requires:
             - job-ci-skip-check
           filters:
@@ -412,3 +434,191 @@ workflows:
           filters:
             branches:
               only: master
+      - install-xmllint
+      - install-ajv
+      - run:
+          name: Validate content instances
+          command: |
+            bash "$CICD_DIR/validate-content.sh" "$OSCAL_BUILD_DIR"
+  job-roundtrip-conversions:
+    executor: python-executor
+    steps: 
+      - *checkout
+      - *attach_build_workspace
+      - install-maven
+      - install-saxon
+      - install-ajv
+      - install-lxml
+      - run:
+          name: Generate round trip XML and JSON conversions
+          command: |
+            bash "${CICD_DIR}/roundTripXML.sh" "$OSCAL_BUILD_DIR"
+      *publish_artifacts
+  job-generate-docs:
+    executor: java-executor
+    steps:
+      - *checkout
+      - *attach_build_workspace
+      - check-skip-build
+      - install-saxon
+      - run:
+          name: Generate schema documentation
+          command: |
+            #cp -r "$OSCAL_BUILD_DIR"/* "$OSCAL_REPO_DIR"
+            bash -x "$CICD_DIR/generate-model-documentation.sh" "$OSCAL_BUILD_DIR"
+            # echo "-- fix the fact that we cannot generate directly to the build directory due to a bug"
+            # mv docs "$OSCAL_BUILD_DIR"/docs
+      - *publish_artifacts
+      - persist_to_workspace:
+          root: build_artifacts
+          paths:
+            - docs
+  job-deploy-artifacts:
+    executor: java-executor
+    steps: # a collection of executable commands
+      - *checkout
+      - *attach_build_workspace
+      - check-skip-build
+      - deploy: # delete and copy artifacts before deploy
+          name: Commit Artifacts
+          command: |
+            cd "$OSCAL_REPO_DIR"
+            # Remove existing generated files
+            git rm -r --ignore-unmatch xml/convert/*.xsl
+            git rm -r --ignore-unmatch xml/schema/*.xsd
+            git rm -r --ignore-unmatch json/convert/*.xsl
+            git rm -r --ignore-unmatch json/schema/*.json
+            git rm -r --ignore-unmatch content/**/*.xml
+            git rm -r --ignore-unmatch content/**/*.json
+            git rm -r --ignore-unmatch content/**/*.yaml
+            # Copy new built files to repo
+            cd -
+            cp -r "$OSCAL_BUILD_DIR"/* "$OSCAL_REPO_DIR"
+            cd "$OSCAL_REPO_DIR"
+            # add the new files
+            git add -f --all xml
+            git add -f --all json
+            git add -f --all content
+            # check for changes
+            echo "Changed files:"
+            
+            if ! $(git diff --exit-code --name-only HEAD~1 xml json content); then
+              # Only deploy if something relevant has changed
+              # Setup deployment
+              git config user.name "Deployment Bot"
+              git commit --allow-empty -m "Deploying content [ci deploy skip]"
+              # Ensure we are deploying against the latest
+              git fetch
+              git rebase origin/master
+              # deploy
+              git push
+            else
+              echo "  No files changed"
+            fi
+      - *publish_artifacts
+  job-validate-site-content:
+    executor: ruby-node-executor
+    steps:
+      - *checkout
+      - *attach_build_workspace
+      - check-skip-build
+      - *restore_gem_cache
+      - install-bundler
+      - run:
+          name: Build jekyll site
+          command: |
+            cp -vr "$OSCAL_BUILD_DIR"/* "$OSCAL_REPO_DIR"
+            cd "$OSCAL_REPO_DIR/docs"
+            bundle exec jekyll build -d _site/OSCAL
+            htmlproofer --assume-extension ./_site
+          environment:
+            JEKYLL_ENV: production
+      - *save_gem_cache
+      - *publish_artifacts
+  job-build-and-deploy-website:
+    executor: ruby-node-executor
+    steps:
+      - *checkout
+      - *attach_build_workspace
+      - check-skip-build
+      - *restore_gem_cache
+      - install-bundler
+      - run:
+          name: Build jekyll site
+          command: |
+            cp -vr "$OSCAL_BUILD_DIR"/* "$OSCAL_REPO_DIR"
+            cd "$OSCAL_REPO_DIR/docs"
+            bundle exec jekyll build
+            mkdir -p _site/.circleci
+            # copy the circleci config
+            cp ../.circleci/config.yml _site/.circleci/config.yml
+          environment:
+            JEKYLL_ENV: production
+      - *save_gem_cache
+      - run:
+          name: Run deploy script
+          command: |
+            cd "$OSCAL_REPO_DIR/docs"
+            git config user.name "Deployment Bot"
+            bash ./deploy.sh --push-only -v -m "Deploying website [ci deploy skip]"
+      - *publish_artifacts
+workflows:
+  version: 2
+  build:
+    jobs:
+      - job-ci-skip-check
+      - job-roundtrip-conversions
+      - job-validate-markdown:
+
+          requires:
+            - job-ci-skip-check
+          filters:
+            branches:
+              ignore:
+                - master
+                - nist-pages
+      - job-validate-metaschema:
+          requires:
+            - job-ci-skip-check
+          filters:
+            branches:
+              ignore: nist-pages
+      - job-generate-schema:
+          requires:
+            - job-validate-metaschema
+      - job-generate-converters:
+          requires:
+            - job-validate-metaschema
+      - job-validate-content:
+          requires:
+            - job-generate-schema
+      - job-copy-and-convert-content:
+           requires:
+            - job-generate-converters
+            - job-validate-content
+      - job-generate-docs:
+          requires:
+            - job-validate-metaschema
+            - job-generate-converters
+      - job-validate-site-content:
+          requires:
+            - job-generate-docs
+          filters:
+            branches:
+              # only from PRs
+              ignore:
+                - master
+                - nist-pages
+      - job-deploy-artifacts:
+          requires:
+            - job-copy-and-convert-content
+          filters:
+            branches:
+              only: master
+      - job-build-and-deploy-website:
+          requires:
+            - job-generate-docs
+            - job-deploy-artifacts
+          filters:
+            branches:
+              only: master
diff --git a/.vscode/settings.json b/.vscode/settings.json
@@ -0,0 +1,3 @@
+{
+    "python.pythonPath": "/Library/Frameworks/Python.framework/Versions/3.7/bin/python3"
+}
diff --git a/build/ci-cd/.gitignore b/build/ci-cd/.gitignore
@@ -0,0 +1,3 @@
+/python/*.pyc
+/temp/*
+/python/*.jar
diff --git a/build/ci-cd/README.md b/build/ci-cd/README.md
@@ -17,6 +17,7 @@ The primary CircleCI supported CI/CD workflow, ```build```, is uses a series of
 1. [validate-content.sh](validate-content.sh): Validates OSCAL content in the repository's [src](https://github.com/usnistgov/OSCAL/tree/master/src) against the content's respective OSCAL model and format.
 1. [generate-content-converters.sh](generate-content-converters.sh): Generates [JSON to XML](https://github.com/usnistgov/OSCAL/tree/master/xml/convert) and [XML to JSON](https://github.com/usnistgov/OSCAL/tree/master/json/convert) content conversion scripts for each OSCAL model based on their [respective metaschemas](https://github.com/usnistgov/OSCAL/tree/master/src/metaschema).
 1. [copy-and-convert-content.sh](copy-and-convert-content.sh): Copies selective content from the [src/content](https://github.com/usnistgov/OSCAL/tree/master/src/content) directory, and generates instance of this content in alternate OSCAL formats using the appropriate content converters. Note __This script requires that the process have write access to the Git repository branch that the CI/CD is operating on to work completely.__
+1. [roundTripXML.sh](roundTripXML.sh): Performs round trip validations of the content from XML->JSON->XML to confirm that the conversions are lossless and that converted files comply with their schemas.  The same conversions are done in the inverse: JSON->XML->JSON.
 
 Additional scripts are provided that generate content for the OSCAL project website, supported by the CircleCI ```build-and-deploy-site``` workflow.
 1. [generate-schema-documentation.sh](generate-schema-documentation.sh): This is used as part of the site generation pipeline to build HTML documentation for the OSCAL models.
@@ -38,13 +39,10 @@ NodeJS Packages:
 
 ```
 npm install -g ajv-cli
-```
 
-Python packages:
+Python Packages:
 
-```
-pip install 'jsonschema>=3.0.1'
-pip install simplejson
+pip install lxml
 ```
 
 The Saxon jar file needs to be retrieved. This can be two ways:

diff --git a/build/ci-cd/config/content b/build/ci-cd/config/content
@@ -1,6 +1,11 @@
 # path to source|format of source|model of source|format(s) to convert to
-src/content/fedramp.gov/xml/*profile.xml|xml|profile|json
-src/content/nist.gov/**/xml/*catalog.xml|xml|catalog|json
-src/content/nist.gov/**/xml/*profile.xml|xml|profile|json
-#content/**/*catalog.json|json|catalog|xml
+#### FedRAMP Round Trips (throwing errors)
+# src/content/fedramp.gov/xml/*catalog.xml|xml|catalog|json
+# src/content/fedramp.gov/xml/*profile.xml|xml|profile|json
+#### NIST 800-53 round trips
+src/content/nist.gov/SP800-53/rev4/xml/*catalog.xml|xml|catalog|json
+src/content/nist.gov/SP800-53/rev4/xml/*profile.xml|xml|profile|json
+#### Not yet ready for JSON round trips
+content/nist.gov/SP800-53/rev4/json/*catalog.json|json|catalog|xml
+content/nist.gov/SP800-53/rev4/json/*profile.json|json|profile|xml
 
diff --git a/build/ci-cd/generate-content-converters.sh b/build/ci-cd/generate-content-converters.sh
@@ -76,4 +76,4 @@ done < $OSCALDIR/build/ci-cd/config/metaschema
 shopt -u nullglob
 shopt -u globstar
 
-exit $exitcode
+exit $exitcode
diff --git a/build/ci-cd/python/.gitignore b/build/ci-cd/python/.gitignore
diff --git a/build/ci-cd/python/jsonSchemaValidation.py b/build/ci-cd/python/jsonSchemaValidation.py
diff --git a/build/ci-cd/python/saxon9he.jar b/build/ci-cd/python/saxon9he.jar