Development: Add code coverage of unit tests as MAKE target

[aryoda]

Currently there are two make targets generated by common/configure that executes all unit tests with different verbosity levels:

make unittest    # same as make test
make unittest-v # same as make test-v

It would be great to know how good the existing python code is covered by the unit test ("code coverage").

I propose to extend common/configure to generate a new make target for this:

make coverage

I am working on this... (hope it works in the end ;-)

[aryoda]

First manual coverage result: 33 % (including the qt GUI source code):

Name                                                      Stmts   Miss  Cover
-----------------------------------------------------------------------------
applicationinstance.py                                       79      4    95%
askpass.py                                                   10     10     0%
backintime.py                                               528    278    47%
bcolors.py                                                   18      8    56%
cli.py                                                      171    153    11%
config.py                                                   934    448    52%
configfile.py                                               333     18    95%
create-manpage-backintime-config.py                         125    125     0%
driveinfo.py                                                119    119     0%
dummytools.py                                                28     28     0%
encfstools.py                                               418    353    16%
exceptions.py                                                38      8    79%
guiapplicationinstance.py                                    33     33     0%
logger.py                                                    68      6    91%
mount.py                                                    349    265    24%
password.py                                                 199    155    22%
password_ipc.py                                              82     58    29%
pluginmanager.py                                            134     32    76%
plugins/usercallbackplugin.py                                56     11    80%
progress.py                                                  16      2    88%
snapshotlog.py                                               89     11    88%
snapshots.py                                               1307    308    76%
sshMaxArg.py                                                 31     31     0%
sshtools.py                                                 412    351    15%
tools.py                                                   1046    362    65%
/dev/backintime/qt/app.py                                  1103   1103     0%
/dev/backintime/qt/icon.py                                   44     44     0%
/dev/backintime/qt/logviewdialog.py                         185    185     0%
/dev/backintime/qt/messagebox.py                             63     63     0%
/dev/backintime/qt/plugins/notifyplugin.py                   35     22    37%
/dev/backintime/qt/plugins/qt4plugin.py                      35      6    83%
/dev/backintime/qt/qtsystrayicon.py                         148    148     0%
/dev/backintime/qt/qttools.py                               303    303     0%
/dev/backintime/qt/restoredialog.py                          79     79     0%
/dev/backintime/qt/serviceHelper.py                         148    148     0%
/dev/backintime/qt/settingsdialog.py                       1386   1386     0%
/dev/backintime/qt/snapshotsdialog.py                       285    285     0%
-----------------------------------------------------------------------------
TOTAL                                                     10437   6949    33%

Steps to create the code coverage:

1. Install the coverage python package.
   On Ubuntu 20.04 eg. use sudo apt install python3-coverage, otherwise pip install coverage
2. Open a terminal in the "common" folder
3. python -m coverage run --source=.,../qt --omit */test/* -m unittest -b # exclude unit test and standard module files...
4. Create a report based on the coverage results stored in the file .coverage:
   python3 -m coverage report

To create an HTML report where you can even click on the py file names to see the code lines that are NOT covered so far use:

python3 -m coverage html
firefox htmlcov/index.html

Note: The real coverage is higher since I have skipped unit tests for unknown reasons

[aryoda]

If I ignore the qt GUI source code files the coverage is 52 % (or higher due to some skipped unit tests in my setup):

python3 -m coverage run --source=. --omit */test/* -m unittest -b
python3 -m coverage report

Name                                  Stmts   Miss  Cover
---------------------------------------------------------
applicationinstance.py                   79      4    95%
askpass.py                               10     10     0%
backintime.py                           528    278    47%
bcolors.py                               18      8    56%
cli.py                                  171    153    11%
config.py                               934    448    52%
configfile.py                           333     18    95%
create-manpage-backintime-config.py     125    125     0%
driveinfo.py                            119    119     0%
dummytools.py                            28     28     0%
encfstools.py                           418    353    16%
exceptions.py                            38      8    79%
guiapplicationinstance.py                33     33     0%
logger.py                                68      6    91%
mount.py                                349    265    24%
password.py                             199    155    22%
password_ipc.py                          82     58    29%
pluginmanager.py                        134     32    76%
plugins/usercallbackplugin.py            56     11    80%
progress.py                              16      2    88%
snapshotlog.py                           89     11    88%
snapshots.py                           1307    308    76%
sshMaxArg.py                             31     31     0%
sshtools.py                             412    351    15%
tools.py                               1046    362    65%
---------------------------------------------------------
TOTAL                                  6623   3177    52%

[buhtz]

Thanks a lot in your work for this.

It seems to me that the Qt part is totally untested. But I still doing research about how to test Qt code. It is possible and I put it on my personal todo list as a far-away-job.

[aryoda]

It seems to me that the Qt part is totally untested.

Exactly. The way to go for "qt testing" could be

• add unit tests for all action handlers (which call the backend backintime finally)
• add unit tests to construct the qt GUI
• define the "click" steps for a manual UAT (user acceptance test) -> just to test if the GUI works as expected