-
Notifications
You must be signed in to change notification settings - Fork 211
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Proposal: pytest-cov should do less #337
Comments
/cc @ionelmc @graingert |
@jezdez Is there something i can do to clear up the confusion (if that's what the emoji means!)? I may have written too fast and left out some context. |
I absolutely understand the "simple is better than complex" idea in this proposal and I support you in making maintenance easier. 👍🏻 I definitely worry about having to know the coverage CLI tool as well as setting up pytest-cov to generate terminal output though. At the moment it's one step in a very pytest-typical config setup (updating addopts in pytest.ini) and you're proposing two steps to get the same result, addopts and another way to run coverage report. Given how difficult some CI environments are (e.g. path problems, incompatible dependencies etc) I think this change would basically trade lower maintenance friction on your side with a higher barrier of ease of use. Not saying that's a bad trade, just that it would make sense to maybe find a way to alleviate that higher barrier for users. Alternatively to your suggestion would there be a way to instead implement a simplified version of the coverage API in pytest-cov? Only cover terminal reporting in pytest-cov for example and document how to do more complex reporting via the coverage CLI? |
I'd be in favour of the coverage API providing a method like |
would be some wiggle room in how cov_fail_under would be passed to summary of course, or it could handle the cov_fail_under entirely and raise an exception after having finished all the writing |
I could compromise on @jezdez's idea of keeping text reporting, and leaving other reporting to the coverage.py command line. How would other people feel about that?
BTW: I'm very glad for the discussion! :) |
As a frequent user of Pytest-cov, I'm strongly in favor of reducing its features. And I agree with @jezdez that keeping the console reports is worth thinking about. It's literally the only reporting feature I use with Pytest-cov, if I use it for reporting at all, and I can see how it's handy enough to keep around. The benefits of the Pytest-cov reporting UI always felt marginal to me, at best. At worst they just add bloat to my Pytest commands. |
simplifying the plugin would be great. Allowing text reporting would be preferable to not. |
You get my vote to go down this path with pytest cov 3 👍 |
What about two "versions" of plugin: pytest-cov and pytest-cov (enhanced).Ok, it does not much sense for me too, an enhanced version from a plugin, but the version could be like a branch from vcs project. Maybe, this would be like a experiment, to analyse how many people miss this "enhanced features". |
@maurobaraldi I'm not sure who would maintain the enhanced one. If I only contribute to the plain version (to add support for new coverage.py features), then the enhanced won't be a superset of the plain version. That sounds extra-confusing. :( |
It may be good for the discussion to talk about the affected API and options and common use cases and configurations. I think it should be differentiated between three major useages:
The direct useage of pytest is for simplicity and speedup of concurrent testing / subprocesses Usage of tox may follow two mayor config styles: as described in coverage docs: [testenv]
...
commands =
pytest --cov=src --cov=tests --cov-report=xml
setenv =
COVERAGE_FILE=.coverage.{envname}
deps =
pytest
pytest-cov
[testenv:coverage]
skip_install = true
deps =
coverage
setenv =
COVERAGE_FILE=.coverage
commands =
coverage erase
coverage combine
coverage html
coverage xml
coverage report --fail-under=100.0 or as documentet in pytest-cov docs: [testenv]
commands = pytest --cov --cov-append --cov-report=term-missing
deps =
pytest
pytest-cov
depends =
{py27,py36}: clean
report: py27,py36
[testenv:report]
deps = coverage
skip_install = true
commands =
coverage report
coverage html
[testenv:clean]
deps = coverage
skip_install = true
commands = coverage erase My question is, if cov-report options are removed, would both approches still work. Are other options as terminal and xml actually used? Could it be an option make pytest-cov just a commad wrapper around coverage and do noting itself anymore? |
@loechel, there are important things pytest-cov does that only it can do: managing the distribution and re-collection of coverage data under xdist; signaling when tests begin and end for who-tests-what; probably a few other things. It should continue to do those things. I don't see the value in pytest-cov being a wrapper around coverage.py's other functionality. |
There were 26 +1's on the top description. I don't know how to see the names in the GitHub UI. I had to use the API to get them: blueyed, davidism, sethmlarson, gvoysey, styvane, boxed, CodeMouse92, yeraydiazdiaz, ionelmc, RazerM, timofurrer, ivoflipse, somacdivad, obestwalter, okken, dcoles, otrenav, Stranger6667, gaborbernat, mblayman, eddieantonio, sivy, merwok, fschulze, jab, gkapfham. |
Mkay so I'll add my feedback:
I do have the same "things are not optimal" feeling, that's why I +1. |
I hear what you are saying about dealing with the fallout. I think probably everyone is on-board with dropping support for older everything (though I would like to keep Python 2.7 in the mix for now). Let's assume that is happening no matter what. Then we have to decide what else to do. Here are some possibilities, a gradient of severity: (Note: in the above, I used --cov-report=html as a proxy for any feature that has been dropped.) While D would mean the leanest resultant code, I would lean toward option C, to help people transition. |
+1 on C |
I'd personally want the current reporting to be available as an API then; I use pytest to drive coverage extensively for a variety of reasons, and if pytest-cov loses those features I'll need to reimplement them. Wrapping a command line when I'm already in Python is not ideal; I'd like to be able to invoke them as code. If so, that'd be fine; I'd be happy to have consistent coverage reporting control between test frameworks. |
@offbyone I'm not sure what you mean by an API. Coverage.py has an API. Would that work? Keeping all the code in pytest-cov but exposing it in an API from the plugin only adds to the complexity, it doesn't reduce it. |
coverage should inherit pytest-cov's ability to get coverage out of sub-proceses easily. Every time I (attempt) to do subprocess coverage, I re-discover a bunch of gotcha's with atexit, signals, environment and |
@meejah re subprocesses see: https://pypi.org/project/coverage_enable_subprocess/ - nedbat/coveragepy#367 |
@meejah I am in favor of exploring the parts of pytest-cov that could be moved into coverage.py itself. To keep the process a little simpler now, I'm only talking here about dropping code from pytest-cov that is already in coverage.py. The .pth/subprocess/etc stuff is gross and scary however it is done, but I understand that people find it frustrating. |
I'm a frequent user of pytest and pytest-cov. I use the various reporting options (both terminal in various versions, and HTML) through pytest-cov, currently, as a matter of my regular development cycle. Thank you for soliciting opinions and making the discussion so open for everybody. I, as a user, would be happy enough to deal with option C. I frequently work offline or on very bad internet, so being confronted with an error after an already tedious update woud be unpleasant, whereas having explicit instructions available would be an unexpectedly positive experience. (I assume I'm not speaking for a majority here, but maybe for a subset of devs that tend to be forgotten, occasionally.) I'd understand if you choose option D instead, but I'd feel less happy about it. If adding instructions for the different flags is hard, then please make that fact very clear in the changelog – it would definitely reduce my annoyance if I ran into it by accident. |
@nedbat yes, probably. I wasn't saying it didn't, only that I need one to replace pytest-cov's reporting for in-process work :) |
pytest-cov does much more than that, and much better. It supports all packaging and installation forms (coverage_enable_subprocess does not). Someone has suggested that there's no need for pytest-cov (ignoring subprocess stuff) since you can just run
|
FWIW: I always use I thought pytest-cov was sugar (only a shortcut) until I read #337 (comment), and tox already gives me a shortcut for Also, I also really enjoy the possibility of running bare @rixx If you have pytest-cov installed, you also have coverage, and |
Not exactly. My use case is that I'm used to running pytest with |
Throwing an idea out there: what if pytest-cov was not changed, and the new ideas implemented in a new plugin with another name? That way no automated upgrade can break command lines, maintainers will have to see warnings or announcements and decide to switch (the docs could help with translation of pytest-cov options to coverage.py commands). |
@merwok I think I like the idea of changing the name. We could make one more release of pytest-cov, where it prints a warning that there will be no more updates to it, and that you should switch to pytest-coverage. Then pytest-coverage can simply remove all the obsolete options. This has advantages: people have to explicitly switch over, and we don't need an awkward middle phase where the code is still complex, but doesn't do what people want. |
I think I like the proposal minus the warning part 👍 |
I am a user, and the name change @merwok suggested is my favorite option if the UI or functionality changes significantly, for the reasons he and @nedbat explained. I support any effort to notify the user of the changes and give clear directions about what to do. Seeing a warning about future changes would be a a nice touch and wouldn't bother me. |
For comparison, pep8 was renamed to pycodestyle in June 2016, with a runtime warning added to pep8 in October 2017. In September 2019, pep8 had nearly a million downloads, pycodestyle had 5.5m.
Source:
Source: Not sure what to conclude exactly! People still use the old one a lot. But renaming would be a clear signal and let you concentrate only on the new project and retire the old. |
I don't think that rename is necessary here but I cant help to notice that there is not even one downvote on the original issue. Rarely I see so much agreement. Go and and start removing stuff from it, I am myself busy with similar tasks on other projects. Less is more! ...especially for older projects. |
I only use Having all this removed, why should I keep using |
I disagree. I believe pytest-cov should integrate more closely with coverage and expose more of the new features - for example: #377 |
@graingert Can you explain why it should? What is difficult about using coverage reporting commands to do reporting? Why do those operations need tight integration with pytest? The only reason I've heard is, "I like just having one command to do everything." That command should be "tox" or "make" or "dostuff.sh", not "pytest". |
reply from #330
In my view pytest-cov's core feature is reporting coverage immediately after running the tests (potentially failing the suite if the coverage isn't high enough). Without this feature what is left? To me it seems a few workarounds for covering code in pytest-xdist and other subprocesses. What benefit would pytest-coverage have over I see the pytest-cov.pth workaround for subprocess as something that could be moved into coveragepy directly or another package |
@graingert You make a good point: it is easy to misunderstand what pytest-cov does that is essential. In my mind, pytest-cov's core features are those things that can be done best (or only) within a pytest plugin. This may not be a complete list but those features include:
You say:
This is exactly my point: pytest-cov reports have no benefit over coverage commands. Pytest-cov does nothing different or better when reporting than the coverage commands you can already run. Except the command-line interfaces are tortured (because the switches from "coverage report" have to all be crammed into It's fine for pytest-cov to only do a few small things. That's what plugins should do. If you want to add features like per-directory fail-under, let's add them to coverage. It's got nothing to do with pytest, and could benefit people using other test runners. |
Any update on this? |
I think that @ionelmc is the one that can make that decision. I personally started to remove pytest-cov from most of my projects and replace it with direct use of coverage-py, as I hit some several bugs/limitations. Considering the very limited maintenance time for the plugin, if pytest-cov wants to survive, it should redefine its scope as far more limited than before. Even so, in my case it was impossible to correctly record the coverage with pytest-cov because I has a pytest plugin inside my own codebase, and the imports was happening before even pytest-cov was fully initialize. I tried all the hacks that @ionelmc suggested but still failed. In the end I was able to use coveragepy while also enabling both subprocess and xdist support as I needed both, and apparently it did work well. I still wonder what could pytest-cov provide me that I cannot get from coveragepy? Or at least can it do to simplify the entire coverage configuration? Now, there is still one issue that I want to resolve on coverage part, navigation of the coverage reports online. At this moment I am using codecov but I am unpleased about their ability to fix reported bugs. For example they are not able recognize I also want to thank to all those involved in improving the coverage experience with Python! |
Link? 🙂 |
https://github.com/ansible/ansible-lint/blob/main/pyproject.toml#L13-L24 also check tox.ini for other settings related to coverage. Nothing in the code itself. |
If there's ever going to be a clean new pytest-cov without all the old cruft (that apparently can be easily replaced with |
A basic integration that will help with coverage requirements and early initialization might be really nice, i believe too much that's better served by coverage these days was added in the beginning I vaguely recall that when Pytest-cov started coverage was not nearly as nice and feature packed The fact that these days gutting Pytest-cov to a small shim seems like a good idea is a testament to the hard and tireless work @nedbat and others put into coverage |
There is also another option, but I am not sure what @nedbat thinks about it. What it coveragepy would also expose a coverage pytest plugin, one that would be used to add the very small useful integration with pytest. The only thing where an integration with pytest could help is related to discovery. If that is true, we could end-up retiring pytest-cov. I would love to see a way to simplify coverage setup on python projects, especially for pytest+tox+gha+codecov users, as I seen myself forced to add several hacks in order to make it work fine, most of them being not really obvious when you look at the config. If the others are interested about this subject, I could probably spend some time to document them, so we can review them and decide what can we do to minimize the amount of config someone would have to do do, to have a just-works approach. In fact I was wondering if a coverage plugin for tox would not be more suitable, because tox is already the wrapped the testing commands, so clearly in a better suited situation for inject coverage call. On this path, my only dilemma is that I do not know how to decide which tox commands should be run with coverage or not as from time to time I run multiple commands and some of them do not need to be covered. Maybe that bit should remain manual. |
Thanks for picking up the conversation. I am still interested in seeing this move forward. Ionel said:
I don't understand why a pytest plugin would do anything beyond what is needed to run the tests. Why should it do any coverage reporting? All of that can be handled by a coverage command after the pytest command. The coverage command can also handle the I think the urge to do more comes down to this idea that "people just want to type About including a plugin with coverage.py: I'm open to exploring that idea. Would people be willing to help me build and maintain it? |
I want to point out that python has a startup cost; if |
I've been working in the pytest-cov code recently to get it to support coverage.py 5.0 (pull request: #319), and also reviewing #330. The problems that have come up have brought me to a conclusion: pytest-cov does too many things.
The current code attempts to be a complete UI for coverage. I think this is a mistake. Pytest-cov should limit itself to those functions that can only be done within a pytest plugin. In particular, the reporting options of pytest-cov add no value over just using coverage directly to generate reports.
Pytest-cov is no one's passion project. There are a few of us that would like to move it forward, but want to do it efficiently. Removing functionality from pytest-cov would make it easier to maintain.
If we keep to the current model of pytest-cov being a complete UI for coverage.py, then every time something is added to coverage.py, the pytest-cov plugin needs to be updated. As an example, we just added a new report type to coverage.py (JSON). Why do we need to add options to pytest-cov to support it?
Pytest is a test runner. Its job is to run tests. The pytest-cov plugin should limit itself to integrating the run phase of coverage into the test running process. Reporting is a separate step that is better separately.
The current design leads to tortured option syntax in an attempt to squeeze everything into the pytest command line. There's no need. Remove options that aren't related to running, and let some options be specified in configuration files. Sophisticated test suites already have configuration files. Let's simplify pytest-cov.
Thoughts?
The text was updated successfully, but these errors were encountered: