debuginfo test suite shows zero tests being executed if pre-requisites are missing

[pietroalbini]

Most tests in compiletest use annotations to denote whether they can be run or should be ignored, for example if some pre-requisites are missing. The debuginfo test suite is different though: since all tests have some common pre-requisites (like requiring a debugger to be installed or not being executed with remote-test), the some checks are done by compiletest on all of them.

The problem is, compiletest then doesn't mark the tests as ignored, it just doesn't load the tests at all. For example, this is the output when trying to run the test suite with remote-test enabled:

Testing stage2 compiletest suite=debuginfo mode=debuginfo (x86_64-unknown-linux-gnu -> aarch64-unknown-none)
WARNING: debuginfo tests are not available when testing with remote

running 0 tests


test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 15.19µs

This hides all information about which tests were ignored, the warning is not recorded in the build metrics or anywhere else, it's just printed to stdout by compiletest. So, unless someone reads the logs and spots that sentence, there is no way to know the tests were actually ignored.

While in the remote-test case the situation is kinda acceptable right now (at least there is a warning explaining why tests are ignored), the situation is even worse when you don't have any debugger installed on your system. In that case, this is the whole output:

Testing stage2 compiletest suite=debuginfo mode=debuginfo (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)

running 0 tests


test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 15.19µs

There is no indication whatsoever of what caused the tests to be ignored, and the only way to figure out what's wrong is to read the compiletest source code and try different things until one sticks.

[jieyouxu]

compiletest triage: debuginfo suites should probably act like each test is revisioned, one for each specific debugger version (cdb, lldb, gdb) you name it, then for each revision implicitly inject a //@ needs-debugger: cdb-equivalent logic so libtest marks every test in the suite as ignored.