Add flaky test support for JUnit format

[MikeSchulze]

Describe

I actually work on implementing flaky test handling with a configured retry count.
Means a test could be executed multiple times and finally ends with success or failure.

My actual created JUnit report looks like:

<?xml version="1.0" encoding="UTF-8" ?>
<testsuites id="2024-09-03" name="report_4" tests="5" failures="4" skipped="1" flaky="2" time="0.347">
	<testsuite id="0" name="GdUnitTestSuiteTest" package="addons/gdUnit4/test" timestamp="2024-09-03T15:16:43" hostname="localhost" tests="5" failures="4" errors="0" skipped="1" flaky="1" time="0.347">
		<testcase name="test_assert_that_types" classname="GdUnitTestSuiteTest" time="0.013">
		</testcase>
		<testcase name="test_unknown_argument_in_test_case" classname="GdUnitTestSuiteTest" time="0.000">
			<skipped message="SKIPPED: res://addons/gdUnit4/test/GdUnitTestSuiteTest.gd:50">
			</skipped>
		</testcase>
		<testcase name="test_find_child" classname="GdUnitTestSuiteTest" time="0.049">
		</testcase>
		<testcase name="test_find_by_path" classname="GdUnitTestSuiteTest" time="0.050">
		</testcase>
		<testcase name="test_flaky_success" classname="GdUnitTestSuiteTest" time="0.033">
			<failure message="FAILED: res://addons/gdUnit4/test/GdUnitTestSuiteTest.gd:90" type="FAILURE">
<![CDATA[
failure 1: at retry 1
]]>
			</failure>
			<failure message="FAILED: res://addons/gdUnit4/test/GdUnitTestSuiteTest.gd:91" type="FAILURE">
<![CDATA[
failure 2: at retry 1
]]>
			</failure>
		</testcase>
		<testcase name="test_flaky_success" classname="GdUnitTestSuiteTest" time="0.033">
			<failure message="FAILED: res://addons/gdUnit4/test/GdUnitTestSuiteTest.gd:90" type="FAILURE">
<![CDATA[
failure 1: at retry 2
]]>
			</failure>
			<failure message="FAILED: res://addons/gdUnit4/test/GdUnitTestSuiteTest.gd:91" type="FAILURE">
<![CDATA[
failure 2: at retry 2
]]>
			</failure>
		</testcase>
		<testcase name="test_flaky_success" classname="GdUnitTestSuiteTest" time="0.033">
		</testcase>
	</testsuite>
</testsuites>

The test test_flaky_success is executed three times and ends finally with success.

The report is not aware of multiple test executions and reports as failure.

Test suite	Passed	Failed	Skipped	Time
GdUnitTestSuiteTest	4✅	2❌	1⚪	196ms

✅ test_assert_that_types
⚪ test_unknown_argument_in_test_case
✅ test_find_child
✅ test_find_by_path
❌ test_flaky_success
failure 1: at retry 1
❌ test_flaky_success
failure 1: at retry 2
✅ test_flaky_success

Proposed solution

It would be great to extend the report by flaky column.

Test suite	Passed	Failed	Flaky	Skipped	Time
GdUnitTestSuiteTest	4✅		1⚠️	1⚪	196ms

The report generation should be aware of multiple test with same name in test suite and collect them to one state.
If a test found be multiple times (equal test case and test suite), it should be handled as:

• all failed -> failed
• failed and the last is success -> flaky
• all success -> success

Alternatives considered

If there is already a way or a workaround to mark a test as flaky, please let me know how to write the JUnit report.

Thanks
Regards Mike Schulze