Test & Build: Distinguish between "unit", "system" and "integration" tests

[buhtz]

Blocked by

• Migrate to Python build-system (PEP 517 & PEP 621) #1575

Problem

BITs test suite do have three kinds of tests in it: unit, integration and system tests. Integration and system tests are valuable but they do not run under every circumstances (e.g. chroot build environments or other sandboxes). Debian do has the problem (Debian#940319) for example.

Proposal

Separate between unittests and all other tests to account for the different characteristics of these test categories.

Solution

Currently BIT do use make as build system also to invoke tests via pytest or python -m unittest. Create folder common/tests/unittests containing unittests only. With this solution a make test will invoke all tests (including the ones in the sub-folder tests/unittests) and make unittest will invoked only the tests in the sub-folder. This should be easy to realize via our configure script.
More accurate we also could create tests/system and tests/integration which would force us to think about what test kind we do have. Another side effect would be a more realistic coverage number. IMHO "code coverage" should count only for unittests.

The usual make unittest then could be used by Debians build system (see below).

More background

Related to the Debian bug ticket #940319 I did some more research and also got some valuable advice of people around Debians Python Packaging Team. The Debian ticket is about the problem that our tests want to write to HOME folder which is not possible in Debians build system because they do chroot as a sandbox solution.

But fixing the HOME problem with using a temp folder or a fake in-memory-filesystem won't solve the problem described in this Issue here. Our tests do create ssh keys, installing them, contacting an ssh server, etc. There is no way to mock that and there is no need for it. Integration and system tests are also valuable and they still prevented us from some mistakes.

Which tests to separate?

Looking through the code in common/tests I only can find diagnostics.py to be a real unittest without dependencies to other BIT components. Also argparse.py is a candidate for a unittest but need some more research and modificaton. All other tests seem to be real integration- or system-tests to me. No other unittests do exist.

Sorry, this issue is not fun. 😿

[buhtz]

Related to #1463

[buhtz]

I tried to fix this with buhtz:1489testcategories but messed up with TravisCI, Coveralls and configure-make script I'm not familiar with.

My original proposal still make sense to me and should be the goal. But I will make a pause from here on that Issue. It should wait until we migrated to modern Python packaging standards (slightly discussed in #1513). After that the build environment and imports are way easier to handle.

[buhtz]

Depends on #1575