-
Notifications
You must be signed in to change notification settings - Fork 199
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Test & Build: Add target "unittest" #1489
Comments
Related to #1463 |
I tried to fix this with My original proposal still make sense to me and should be the goal. But I will make a pause from here on that Issue. It should wait until we migrated to modern Python packaging standards (slightly discussed in #1513). After that the build environment and imports are way easier to handle. |
Depends on #1575 |
Depends on #1575 which need to be fixed first.
Problem
BITs test suite do have three kinds of tests in it: unit, integration and system tests. Integration and system tests are valuable but they do not run under every circumstances (e.g. chroot build environments or other sandboxes). Debian do has the problem (Debian#940319) for example.
Proposal
Separate between unittests and all other tests to account for the different characteristics of these test categories.
Solution
Currently BIT do use
make
as build system also to invoke tests viapytest
orpython -m unittest
. Create foldercommon/tests/unittests
containing unittests only. With this solution amake test
will invoke all tests (including the ones in the sub-foldertests/unittests
) andmake unittest
will invoked only the tests in the sub-folder. This should be easy to realize via ourconfigure
script.More accurate we also could create
tests/system
andtests/integration
which would force us to think about what test kind we do have. Another side effect would be a more realistic coverage number. IMHO "code coverage" should count only for unittests.The usual
make unittest
then could be used by Debians build system (see below).More background
Related to the Debian bug ticket #940319 I did some more research and also got some valuable advice of people around Debians Python Packaging Team. The Debian ticket is about the problem that our tests want to write to HOME folder which is not possible in Debians build system because they do
chroot
as a sandbox solution.But fixing the HOME problem with using a temp folder or a fake in-memory-filesystem won't solve the problem described in this Issue here. Our tests do create ssh keys, installing them, contacting an ssh server, etc. There is no way to mock that and there is no need for it. Integration and system tests are also valuable and they still prevented us from some mistakes.
Which tests to separate?
Looking through the code in
common/tests
I only can finddiagnostics.py
to be a real unittest without dependencies to other BIT components. Alsoargparse.py
is a candidate for a unittest but need some more research and modificaton. All other tests seem to be real integration- or system-tests to me. No other unittests do exist.Sorry, this issue is not fun. 😿
The text was updated successfully, but these errors were encountered: