-
Notifications
You must be signed in to change notification settings - Fork 184
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implementation Agnostic Testing #1946
Implementation Agnostic Testing #1946
Conversation
It seems this PR's develop has diverged from Edit: it seems fixing the history was simple. |
18c5ff4
to
aaec9f9
Compare
* CLI application args * Outline of test execution (test file parsing, testing driver, etc) * Test results/reporting not implemented yet * Test file coverage not implemented yet
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Eager to see this work! very cool.
The specification format will remain unchanged for now. | ||
There is an argument for the format to be replaced or simplified in the future, but the use of `@id` attributes for sections and requirements make linking a test to a example simple. | ||
|
||
### Test Suite Data Format |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Non-blocking: as an aside, did you consider/review the Test Anything Protocol (TAP) or any similar efforts while designing the MVP JSON test data format you are proposing?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have not heard of this protocol, though it looks like compelling follow-up work. The format seems simple enough to adapt the current output to (perhaps behind a CLI flag). The current output format is arbitrarily chosen as it's "easy enough" to read both as a human and to grep for basic automation.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this is worth moving forward and trying, noting the comment that the JSON schema for the tests would be a future specification.
@Compton-NIST how about a little metaschema for the test harness format? We don't really need an XML capability (since XSLT can read JSON if need be) but it might be a fun little exercise. |
* Deleted duplicate `metaschema_datatypes` file * Added spec test adr and prototype spec test file * Spec test harness and minimal example
Committer Notes
Fixes #1771
Side-effects:
@level
to a few spec requirementsAll Submissions:
By submitting a pull request, you are agreeing to provide this contribution under the CC0 1.0 Universal public domain dedication.
(For reviewers: The wiki has guidance on code review and overall issue review for completeness.)
Changes to Core Features: