-
Notifications
You must be signed in to change notification settings - Fork 4
TestScript Extensions
This is a catalog of extensions added by the project to standard TestScript. These extensions use Extension and Contained facets of the FHIR Standard. This is organized by feature as some features rely on multiple extensions used in concert.
Many of the extensions affect error handling. Before reading this it is important to understand error handling in standard TestScript. An overview is available here.
Standard TestScript offers a form of conditional logic. A failed action in setup terminates the script with fail status and does not execute the actions following the failed action. All test sections are skipped as well. A failed assertion in a test terminates only the test, not executing the following actions in the test. Following tests are still run. These terminations still allow teardown and auto-delete to occur. In each case the script returns failure in the report.
The conditional extension offers similar if-then semantics but without indicating a script failure. An action labeled with this extension, if it fails, terminates the section (setup or test) according to the standard but does not indicate failure. It is interpreted as
if assertion fails
then skip rest of section but do not trigger script failure
The conditional extension allows for an assertion to fail and terminate the execution of the surrounding setup
or test section without resulting in script failure. The assertion warningOnly attribute must be false in order to terminate rest of test section. A warning in the log file is generate if conditional extension is used with warningOnly set to true. In this case execution still proceeds even if conditional assertion failed. If used in setup, the test sections still run.
So instead of terminate on assertion
failure it offers skip rest of section on assertion failure.
A section (setup or test) can have multiple conditional extensions. Each one interprets the rest of the section as its then clause.
<test>
<action>
<!--
This extension labels this action as conditional.
If this assert fails then the remaining actions in this test do not run - per the standard.
But, this assert failing does not cause the overall script to fail.
This action could have been an operation.
-->
<modifierExtension url="https://github.com/usnistgov/asbestos/wiki/TestScript-Conditional"/>
<assert>
<description value="needs loading."/>
<expression value="Bundle.total"/>
<operator value="equals"/>
<value value="0"/>
<warningOnly value="false"/>
</assert>
</action>
<!--
This action does not execute if the above assert fails but such a failure does not trigger
the script to fail.
-->
<action>
<operation>
<description value="submit patient."/>
<type>
<code value="create"/>
</type>
<sourceId value="thePatient"/>
</operation>
</action>
<!-- Since test is not followed by an assertion - if this operation fails the rest of the test
will not run and the script will fail
-->
</test>
Standard TestScript offers no facility for modularity and reuse of components. This extension, actually a collection of extensions and a guide for their collective use, offers a way to define reusable components and to reference them from a TestScript. The details are documented here.
TestReport.result can be returned as pass or fail to indicate the overall status of the execution. A fail result indicates that somewhere in the details is an operation or assertion with fail or error status. This leaves out two use case.
When an error happens executing an auto-create, auto-delete, or loading a static fixture there is no place to indicate a failure in the TestReport. TestReport.result indicating fail without underlying details is difficult to diagnose. Handling this failure out-of-band, not coded in the TestReport, does not allow the TestReport to be used as the total result of running the test. A failure extension has been added to allow this reporting in a TestReport. A TestReport can contain multiple failure extensions.
TestScript/TestReport have been adapted to code client tests where the SUT sends a message to the test engine. The test engine evaluates the message by executing a TestScript resulting in a TestReport. A client test TestScript contains only assertions (no operations).
It is frequently desirable to evaluate an input message against all assertions returning to the user a consolidated result. The standard TestScript interpretation of handling assertions, terminate the test when the first assertion fails, is contrary to this goal. Assertions could be coded each in a separate test to get this outcome but this is inconvenient when the test writer wants to use tests to organize collections of related assertions in a script.
The multiple errors extension changes the interpretation of assertions. A test labeled with this extension is not terminated when the first assertion fails. All assertions are executed always. The exception is if an assertion generates an error. This still causes early termination.
<TestScript>
<test>
<modifierExtension url="urn:multipleErrors"/>
<!-- this will succeed -->
<action>
<operation>
<description value="search for patient."/>
<type>
<code value="search"/>
</type>
<resource value="Patient"> </resource>
<contentType value="json"> </contentType>
<params value="?given=${FirstName}&family=${LastName}&birthdate=${birthDate}"> </params>
<responseId value="patient1-search"/>
</operation>
</action>
<!-- this will fail -->
<action>
<assert>
<sourceId value="patient1-search"/>
<response value="bad"/>
<warningOnly value="false"/>
</assert>
</action>
<!-- this will run (and fail) because of urn:multipleErrors -->
<action>
<assert>
<sourceId value="patient1-search"/>
<response value="bad"/>
<warningOnly value="false"/>
</assert>
</action>
</test>
</TestScript>
TestScript specifies that when an action fails, the rest of the test is skipped and execution moves on to the next test. This extension offers an alternative. By specifying urn:noErrors extension in a test, that test is skipped if any prior test reported a failure.
<TestScript>
<test>
<!-- this will succeed -->
<action>
<operation>
<description value="search for patient."/>
<type>
<code value="search"/>
</type>
<resource value="Patient"> </resource>
<contentType value="json"> </contentType>
<params value="?given=${FirstName}&family=${LastName}&birthdate=${birthDate}"> </params>
<responseId value="patient1-search"/>
</operation>
</action>
<!-- this will fail -->
<action>
<assert>
<sourceId value="patient1-search"/>
<response value="bad"/>
<warningOnly value="false"/>
</assert>
</action>
</test>
<test>
<!-- this would normally run because it is in a different test
The extension changes that - a previous failure causes this not to run
-->
<modifierExtension url="urn:noErrors"/>
<action>
<assert>
<description value="Multiple instances?"/>
<expression value="Bundle.total"/>
<operator value="greaterThan"/>
<value value="-1"/>
<warningOnly value="false"/>
</assert>
</action>
</test>
</TestScript>
Negative tests send invalid data and the Test Action Operation is expected to fail. In this case, if there a Test with a single Action and an Operation with an imported component, the following extension can be used to treat the Test as passed. This is a blanket approach on the entire imported series of actions for now in that, the user must check to see if the actual error is expected on a particular action and not some other unexpected error has happened. In the future, a more precise way to target a certain failure should be considered. This extension is usually used for passive assertions or assertions against a static fixture, any unexpected errors should be caught by the persistence search operation on the document responder.
<test>
<description value="PDB Validations."/>
<action>
<modifierExtension url="urn:asbestos:test:action:expectFailure"/>
<operation>
...
Fixtures are a useful tool for identifying the result resource from one operation and feeding it to another operation or assertion. When the result is a Bundle, assertions can still be run because FHIRPath can navigate the structure of a Bundle. Operations rely on fixtures for their inputs. In TestScript/FHIRPath there is no way to assign one resource out of a bundle to a fixture so that it can be used in a subsequent write operation. This occurs frequently when testing transactions which rely on the Bundle resource.
This extension allows the definition of a new type of fixture that takes functionality from fixture, Bundle, and variable.
<TestScript>
<fixture id="submittedDocRef">
<autocreate value="false"/>
<autodelete value="false"/>
<extension url="urn:subFixture">
<extension url="urn:fhirPath">
<valueString value="Bundle.entry[0]"/>
</extension>
<extension url="urn:sourceId">
<valueString value="pdb-bundle"/>
</extension>
</extension>
</fixture>
</TestScript>
A sub-fixture is identified by the use of the urn:subFixture extension inside. Autocreate and autodelete must be false since they have no meaning in this context. They are included because of their cardinality [1..1]. The subFixture extension consists of two required sub-extensions, urn:fhirPath and urn:sourceId.
The urn:sourceId extension must identify an existing fixture, static or not. This fixture must contain a Bundle. The urn:fhirPath extension specifies a FHIRPath to be executed against the Bundle. It must result in a single resource. This resource is the value of the fixture.
Valueset extensions needed to support MHD are found here.