Differentiate between errors and failures

[marcphilipp]

Motivation

Some IDEs, build tools, and testing frameworks differentiate between "assertion failures" and other errors. In Java both are represented as exceptions where the former extend AssertionError and the latter use any other exception type.

In the Java extension schema, this can be represented for an exception ("throwable") using an attribute:

open-test-reporting/schema/src/main/resources/org/opentest4j/reporting/schema/java-0.2.0.xsd

Line 65 in 9b7e27d

<xs:attribute name="assertionError" type="xs:boolean" use="required"/>

However, making this a core concept would allow downstream tools to handle this consistently across frameworks. This could be achieved by introducing an additional ERRORED constant in the core:Status enum.

open-test-reporting/schema/src/main/resources/org/opentest4j/reporting/schema/core-0.2.0.xsd

Lines 119 to 126 in 9b7e27d

	<xs:simpleType name="Status">
	<xs:restriction base="xs:string">
	<xs:enumeration value="SUCCESSFUL"/>
	<xs:enumeration value="SKIPPED"/>
	<xs:enumeration value="ABORTED"/>
	<xs:enumeration value="FAILED"/>
	</xs:restriction>
	</xs:simpleType>

[marcphilipp]

@sebastianbergmann Would that suit your needs in PHPUnit?

[sebastianbergmann]

Yes, an ERRORED value for the Status enumeration would help us. Thank you!