Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implementation Agnostic Testing #1946

Conversation

nikitawootten-nist
Copy link
Contributor

@nikitawootten-nist nikitawootten-nist commented Oct 6, 2023

Committer Notes

Fixes #1771

Side-effects:

  • Fixed typo in spec, added missing @level to a few spec requirements
  • Removed extra copy of metaschema datatypes xsd

All Submissions:

By submitting a pull request, you are agreeing to provide this contribution under the CC0 1.0 Universal public domain dedication.

(For reviewers: The wiki has guidance on code review and overall issue review for completeness.)

Changes to Core Features:

  • Have you added an explanation of what your changes do and why you'd like us to include them?
  • Have you written new tests for your core changes, as applicable?
  • Have you included examples of how to use your new feature(s)?
  • Have you updated all OSCAL website and readme documentation affected by the changes you made? Changes to the OSCAL website can be made in the docs/content directory of your branch.

@nikitawootten-nist
Copy link
Contributor Author

nikitawootten-nist commented Oct 6, 2023

It seems this PR's develop has diverged from usnistgov's develop. I will cherry pick the relevant commits out of this branch and continue work on it in the upcoming week.

Edit: it seems fixing the history was simple.

* CLI application args
* Outline of test execution (test file parsing, testing driver, etc)
* Test results/reporting not implemented yet
* Test file coverage not implemented yet
@nikitawootten-nist
Copy link
Contributor Author

Progress update: I have implemented a "coverage" subcommand to show the spec test coverage against the spec document.
Screenshot 2023-10-11 at 2 12 48 PM

@nikitawootten-nist
Copy link
Contributor Author

Progress update: The tester is now runnable, however I do not have a working test file yet.

Below is a screenshot of the report generated by the script:
Screenshot 2023-10-12 at 5 13 03 PM

The script returns a list of XPath-like selections that yielded incorrect results.

@nikitawootten-nist nikitawootten-nist marked this pull request as ready for review October 13, 2023 15:18
Copy link
Contributor

@wendellpiez wendellpiez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Eager to see this work! very cool.

@nikitawootten-nist nikitawootten-nist linked an issue Oct 13, 2023 that may be closed by this pull request
4 tasks
@aj-stein-nist aj-stein-nist requested a review from a team October 17, 2023 14:45
The specification format will remain unchanged for now.
There is an argument for the format to be replaced or simplified in the future, but the use of `@id` attributes for sections and requirements make linking a test to a example simple.

### Test Suite Data Format
Copy link
Contributor

@aj-stein-nist aj-stein-nist Oct 17, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: as an aside, did you consider/review the Test Anything Protocol (TAP) or any similar efforts while designing the MVP JSON test data format you are proposing?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have not heard of this protocol, though it looks like compelling follow-up work. The format seems simple enough to adapt the current output to (perhaps behind a CLI flag). The current output format is arbitrarily chosen as it's "easy enough" to read both as a human and to grep for basic automation.

Copy link
Contributor

@Compton-US Compton-US left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is worth moving forward and trying, noting the comment that the JSON schema for the tests would be a future specification.

@wendellpiez
Copy link
Contributor

wendellpiez commented Oct 18, 2023

@Compton-NIST how about a little metaschema for the test harness format? We don't really need an XML capability (since XSLT can read JSON if need be) but it might be a fun little exercise.

@nikitawootten-nist nikitawootten-nist merged commit dde71c3 into usnistgov:develop Oct 25, 2023
1 check passed
@nikitawootten-nist nikitawootten-nist deleted the nikitawootten-nist/issue1771 branch October 25, 2023 16:57
Compton-US pushed a commit that referenced this pull request Dec 6, 2023
* Deleted duplicate `metaschema_datatypes` file
* Added spec test adr and prototype spec test file
* Spec test harness and minimal example
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

SPIKE: Implementation-agnostic testing
4 participants