-
Notifications
You must be signed in to change notification settings - Fork 4
TestScript error handling
This is a summary of error handling in TestScript. Content is taken from http://hl7.org/fhir/testing.html. To understand the extensions we added to TestScript it first necessary to understand the error handling capabilities native to TestScript. At issue is how errors and failures are reported in a TestReport.
In this discussion, TestScript is referred to as script, TestReport as report, and a test section inside a script as a test. The term "script failure" indicates a TestReport with result=fail.
TestScript is divided into a collection of phases. Each phase is optional:
fixtures - these can be specified as auto-create and/or auto-delete.
setup - pre-test interactions with the SUT. Consists of one or more actions.
tests - multiple sections for specifying tests. Each test consists of one or more actions. teardown - operations to clean up the SUT after testing
An action is either an operation or an assertion. An operation is a HTTP interaction with the SUT. An assertion is a pass/fail test based on current state.
Overall, a report returns pass or fail as a top level result. Fail is indicated when one or more actions (operation or assertion) generates a fail or error result. Actions generate fail when an assertion is not satisfied. There is a special case where an operation is not followed immediately by an assertion. If the operation returns a 4XX or 5XX HTTP response code this is treated as a fail. An operation immediately followed by an assertion allows for negative testing. Assertions can be coded with warningOnly=true so that the assertion failure does not result in a script failure but still allows for reporting to a user. Errors are generated for internal test engine problems.
An error, generated in any phase of testing, always generates a fail top-level result in the report. The handling of assertion failures depends on the phase of testing.
When an action in setup fails, further action execution stops and the script is terminated, reporting a top-level result of fail. Subsequent actions in setup are not run. Tests are not run. Teardown is run and fixture auto-deletes are run.
When an action in a test fails, the actions following the failed action are not run. But, tests following the failed test are run. Teardown is run and fixture auto-deletes are run.