how to disable test suites that are known to not work

[markus2330]

Continuing the discussion from #1895, we need a generic solution to disable test suites if they fail and no reasonable way to fix it is available (like missing suppression file for asan-gcc).

We have following options:

• completely exclude plugins if they are known to cause problems on a platform.
• add more flags similar to "memleak" and "kdbtests" that say that tests do not run in certain situations. The agents then need to take care to exclude tests with certain flags.
• simply remove the test within the build system if such situations are detected (currently partly already done, e.g., grep for ENABLE_ASAN to find some), and document what is removed in TESTING.md
• use CTEST_CUSTOM_TESTS_IGNORE and CTEST_CUSTOM_MEMCHECK_IGNORE
• use WILL_FAIL CMake property (only added for tests in such situations)

Please tell your opinion.

[markus2330]

@ingwinlu said:

For me a test run that hides failing tests because they are disabled is not successful.

It depends if the test failed because of external software or because of our code.

At the very least the user (in that case the programmer who made the pr) should be very well made aware of that the tests have only run in some configurations (or not run at all).

I fully agree, I added some options of what we could do above.

It probably would be a discussion for another issue to change the existing build setup so that tests are skipped instead of not build to track skipped tests due to misconfiguration in the build system...

Yes, see above.

Compromising the trustworthiness of a test run just to make it not fail is the Volkswagen approach. Then we don't need suppression/blacklist files and I just edit the job to run sh 'true'. I would prefer to not run a job instead of having one that gives a false positive result.

I fully agree that tests that are believed that they are running, but did not run, are dangerous.

The more visible we the situation make, the better. But failing them does not work, then you would not see if you broke something.

[markus2330]

In #1888 I was playing a bit around with CMake. The skipped feature does not work, I did not try the other ways.

[markus2330]

Procedure is to simply uncomment/remove test cases and write a note about it in doc/todo/TESTING