Out-of-the-box behavior unexpected by new users / Modify FAQ/QuickStart/Doc

[buhtz]

I do run coverage via $ coverage -m pytest following the QuickStart section in your docs.

The problem is that the resulting report do contain two kinds of files that IMHO do not belong into a coverage report: test-files and python-standard-lib files. Here is a snipped from the report:

Name                                                                                  Stmts   Miss  Cover
---------------------------------------------------------------------------------------------------------
/home/user/mluCloud/my.work/bit/backintime/qt/plugins/notifyplugin.py                    35     22    37%
/home/user/mluCloud/my.work/bit/backintime/qt/plugins/systrayiconplugin.py               44     19    57%
/usr/lib/python3/dist-packages/PyQt5/__init__.py                                          1      0   100%
/usr/lib/python3/dist-packages/_distutils_hack/__init__.py                               98     93     5%
/usr/lib/python3/dist-packages/cairo/__init__.py                                         13     11    15%
/usr/lib/python3/dist-packages/cryptography/__about__.py                                  4      0   100%
/usr/lib/python3/dist-packages/cryptography/__init__.py                                   2      0   100%
...
/usr/lib/python3/dist-packages/six.py                                                   504    250    50%
applicationinstance.py                                                                   79      4    95%
backintime.py                                                                           534    258    52%
bcolors.py                                                                               18      8    56%
cli.py                                                                                  171    153    11%
config.py                                                                               952    428    55%
...
test/__init__.py                                                                          0      0   100%
test/generic.py                                                                         162      9    94%
test/test_applicationinstance.py                                                        162      0   100%
test/test_argparser.py                                                                  174      0   100%
test/test_backintime.py                                                                  55      2    96%
---------------------------------------------------------------------------------------------------------
TOTAL                                                                                 39172  22261    43%

Technically I do understand why this happens. But the users perspective and needs are what counts.
It is IMHO clear what a users expects from a coverage run. A user do not want to calculate the code coverage of test-files or python-standard-lib files. A user only want the code coverage of the productive code and nothing else.

This should be the default behavior.
Technically it is IMHO easy to know the test files using the same discover patters as pytest/unittest. It is similar for python-standard-lib files to always using the "dist-packages" folder.

If you disagree and do not want to implement that behavior that minimally mention this "problem" in the QuickStart and the FAQ or elsewhere in the docu. Also please mention a solution for that problem.

[nedbat]

I see your point about the third-party code (it's not standard-lib).

Although many people feel as you do about test files, I absolutely disagree: You should include your tests in coverage.

Try using --source=. on your command line. Does that make it do what you want? I can add something about these issues to the quick start.

[buhtz]

Thanks for the link. I read it and understand your arguments. I disagree and could give opposite arguments. But I also assume you have more experience then me. So there is enough room in the future where I might change my opinion. :)

But I wouldn't make this default behavior. The concept of test coverage is quit clear and in most cases this what users of tools like yours do expect. The question most users try to answer with your tool: "How much of my productive code is covered/executed by my test code?". Having test files and other things in it would violate that concept.

You should be very clear and direct about that in your QuickStart guide that your tool do measure a lot more then just the test coverage. Not only mention the --source option but show the two variants: My "strict" one and your "extended" one. Link to your blog entry but replace "should" with "could".

[nedbat]

I can certainly try to clarify these factors in the quick start, though perhaps this is too much detail for there. The quick start already says:

To limit coverage measurement to code in the current directory, and also find files that weren’t executed at all, add the --source=. argument to your coverage command line.

The fact is, coverage.py knows almost nothing about tests. It monitors execution, and tells you what code ran and what code didn't. It doesn't try to distinguish between "test" code and "product" code. You can control that with a number of settings.

[buhtz]

The fact is, coverage.py knows almost nothing about tests. It monitors execution, and tells you what code ran and what code didn't. It doesn't try to distinguish between "test" code and "product" code.

But that behavior is not what most of users expect or what the tools name suggest.
Just copy and paste this sentences "The fact is..." and the QuickStart would be more clear and save a lot of users a lot of time and frustration.

[nedbat]

What do you think? 1adda03

[nedbat]

Docs updated in that commit.