-
Notifications
You must be signed in to change notification settings - Fork 4
TestScript error handling
This is a summary of error handling in TestScript. Content is taken from http://hl7.org/fhir/testing.html. To understand the extensions we added to TestScript it first necessary to understand the error handling capabilities native to TestScript. At issue is how errors and failures are reported in a TestReport.
In this discussion, TestScript is referred to as script, TestReport as report, and a test section inside a script as a test. The term "script failure" indicates a TestReport with result=fail.
TestScript is divided into a collection of phases. Each phase is optional:
fixtures - messages/resources being managed. Can be static (defined with the test) or dynamic (created during the execution of the test). These can be specified as auto-create and/or auto-delete.
setup - pre-test interactions with the SUT. Consists of one or more actions. There is at most one setup section.
tests - specification of tests. Each test consists of one or more actions. There can be any number of tests in a script.
teardown - operations to clean up the SUT after testing
An action is either an operation or an assertion. An operation is a HTTP interaction with the SUT. An assertion is a pass/fail evaluation of the current state.
Overall, a report returns pass or fail as a top level result. Fail is indicated when one or more actions (operation or assertion) generates a fail or error result. Actions generate fail when an assertion is not satisfied. There is a special case where an operation is not followed immediately by an assertion. If the operation returns a 4XX or 5XX HTTP response code this is treated as a fail. An operation immediately followed by an assertion allows for negative testing. The assertion can determine whether the HTTP response is acceptable. Assertions can be coded with warningOnly=true so that the assertion failure does not result in a script failure but still allows for reporting to a user. Errors are generated for internal test engine problems.
An error, generated in any phase of testing, always generates a fail top-level result in the report. The handling of assertion failures depends on the phase of testing.
When an action in setup fails, further action execution stops and the script is terminated, reporting a top-level result of fail. Subsequent actions in setup are not run. Tests are not run. Teardown is run and fixture auto-deletes are run.
When an action in a test fails, the actions following the failed action, in the same test, are not run. But, tests following the failed test are run. Teardown is run and fixture auto-deletes are run.