Skipping a test, skips all subsequent tests in file

[havardAasen]

Description of Problem:

If a file has multiple tests and one of them is skipped, all subsequent test in the file is skipped.

OpenSCAP Version:

Building from main with 74043f7.

Operating System & Version:

Debian Bookworm

Steps to Reproduce:

After a normal build, run ctest -R test_ds_misc --verbose

Log output

Test project /openscap/build
Constructing a list of tests
Done constructing a list of tests
Updating test list for fixtures
Added 0 tests to meet fixture requirements
Checking test dependency graph...
Checking test dependency graph end
test 176
    Start 176: DS/test_ds_misc.sh

176: Test command: /openscap/tests/DS/test_ds_misc.sh
176: Working Directory: /openscap/build/tests/DS
176: Environment variables: 
176:  srcdir=/openscap/tests/DS
176:  top_srcdir=/openscap
176:  builddir=/openscap/build
176: Test timeout computed to be: 10000000
176: + sds_external_xccdf_in_ds                                    
176: TEST: sds_external_xccdf_in_ds
176: --- Starting Evaluation ---
176: 
176: Title
176: Rulxccdf_moc.elpmaxe.www_rule_1
176: Respass
176: 
176: OVAL Results are exported correctly.
176: XCCDF Results are exported correctly.
176: Result data stream exported correctly.
176: RESULT: PASSED
176: 
176: + sds_external_xccdf                                          
176: TEST: sds_external_xccdf
176: --- Starting Evaluation ---
176: 
176: Title
176: Rulxccdf_moc.elpmaxe.www_rule_1
176: Respass
176: 
176: OVAL Results are exported correctly.
176: XCCDF Results are exported correctly.
176: Result data stream exported correctly.
176: RESULT: PASSED
176: 
176: + sds_tailoring                                               
176: TEST: sds_tailoring
176: Document type: Source Data Stream
176: Imported: 2024-05-24T06:18:36
176: 
176: Stream: scap_com.example_datastream_with_checklist
176: Generated: 2016-09-26T15:00:00
176: Version: 1.2
176: Checklists:
176: 	Ref-Id: scap_com.example_cref_xccdf_01
176: 		Status: incomplete
176: 		Resolved: false
176: 		Profiles:
176: 		Referenced check files:
176: 			oval
176: 				system: http://oval.mitre.org/XMLSchema/oval-definitions-5
176: Checks:
176: 	Ref-Id: scap_com.example_cref_oval_01
176: No dictionaries.
176: 
176: Stream: scap_com.example_datastream_with_tailoring
176: Generated: 2016-05-09T15:00:00
176: Version: 1.2
176: Checklists:
176: 	Ref-Id: scap_com.example_cref_tailoring_01
176: 		Benchmark Hint: #scap_com.example_cref_xccdf_01
176: 		Profiles:
176: 			Title: Tailoring profile
176: 				Id: xccdf_com.example_profile_tailoring
176: 	Ref-Id: scap_com.example_cref_xccdf_01-source
176: 		Status: incomplete
176: 		Resolved: false
176: 		Profiles:
176: 		Referenced check files:
176: 			oval
176: 				system: http://oval.mitre.org/XMLSchema/oval-definitions-5
176: Checks:
176: 	Ref-Id: scap_com.example_cref_oval_01-source
176: No dictionaries.
176: --- Starting Evaluation ---
176: 
176: TitRule 2
176: Rulxccdf_com.example_rule_2
176: Respass
176: 
176: XCCDF Results are exported correctly.
176: RESULT: PASSED
176: 
176: + eval_simple                                                 
176: TEST: eval_simple
1/1 Test #176: DS/test_ds_misc.sh ...............***Skipped   1.45 sec

The following tests passed:
	DS/test_ds_misc.sh

100% tests passed, 0 tests failed out of 1

Total Test time (real) =   1.46 sec

The following tests did not run:
	176 - DS/test_ds_misc.sh (Skipped)

From the output, you can see that the last test executed is

openscap/tests/DS/test_ds_misc.sh

Line 293 in 74043f7

test_run "eval_simple" test_eval eval_simple/sds.xml

This is the fourth, out of 23 test in tests/DS/test_ds_misc.sh. This behavior can also be seen from GitHub Actions if we look at how long time it takes to test this file. Fedora runs all of the tests and uses 7.93 sec, while Ubuntu, which skips a test, only uses 1.29 sec. The timing itself is not enough, but combined with the attached log, I believe it is obvious that all subsequent tests is also skipped during CI pipeline.

Actual Results:

The test-suite doesn't run all the tests, if one or more test is being skipped.

Expected Results:

All tests except the ones to be skipped should run, regardless if a previous test has been skipped.