Deprecate test command

[pganssle]

I think there's at least some agreement in #931 that we want to remove the test command. I think we should start by raising deprecation warnings pointing people at tox, the same way we've done for the upload and register commands.

The most likely stumbling block that I see is that I think a huge number of people have created their own test command that invokes their preferred test runner, pytest or whatever. Ideally we'd want to get the deprecation warning to them as well. Hopefully most of them are using TestCommand as their base class, though if we want to get really aggressive about it we could try parsing sys.argv directly.

I think we need to warn in the following situations:

1. If the setup.py test command is executed
2. If tests_require is specified
3. If aliases.test is specified in setup.cfg

It's likely that at least two of these will be specified, but I think two separate warnings would be useful.

CC: @gaborbernat @RonnyPfannschmidt @nicoddemus

[jaraco]

The pytest-runner project does subclass the test command, and there are projects actively depending on it. I do agree we want to deprecate each, and probably both together.

[di]

A bit off-topic, but do we have an overarching issue for deprecating/replacing the various incantations of setup.py? If not, would probably be good to have this, as well as some documentation about migrating on https://packaging.python.org/.

(I can create these issues if they don't already exist)

[jaraco]

do we have an overarching issue for deprecating/replacing the various incantations of setup.py?

Not to my knowledge.

[native-api]

I think there's at least some agreement in #931 that we want to remove the test command. I think we should start by raising deprecation warnings pointing people at tox

Is there, really? The feeedback there is basically that tox is a large dependency and is designed for a different use case than setup.py test.

[jayvdb]

This is a great way to kill sdists. (why have sdists if they have the same contents as wheels) How quickly the lesson of sourceforge.net has been forgotten. Or more recently Oracle changing the Term of Use of the Java downloads. IMO setuptools should be moving in the opposite direction, automatically including in sdists the test modules found using test_require or identified using pytest dry-run.

Anyways, unless setuptools also kills its plugin mechanism, someone will recreate the test command as soon as it is removed from setuptools. Maybe that is a good thing. (But if the goal is only to get that stuff out of the setuptools repo because it is hindering setuptools, why not split it off to a separate project now, rather than pretend that tox is a drop in solution)

[jaraco]

Is there, really? The feeedback there is basically that tox is a large dependency and is designed for a different use case than setup.py test.

Let me see if I can clarify/illustrate. There are two main concerns of setup.py test:

• install dependencies in an environment (the current one)
• invoke a test runner

Tox happens to do both of these in a user-friendly, declarative way, which is what makes it a recommended replacement. The user-experience of "run tox" is comparable to "run setup.py test".

tox does bring more sophistication (and thus friction) to the process, which is why it's somewhat not an exact match.

Users are welcome to seek out other solutions. I wrote pip-run as one such attempt. It has a different behavior in that it always installs dependencies, so doesn't have the second-run speed improvements seen with setup.py test.

why not split it off to a separate project now

You're welcome to consider that. The biggest issue with setup.py test now is that it relies on easy_install and setup_requires, both of which are superseded by pip and PEP 517. Another issue is that setup.py test has no mechanism to specify the version of setuptools under which to run... dependencies need to be resolved before setup.py is invoked. Any tools that attempts to use setuptools plugins to run tests will be forever be responsible to ensure the requisite version of setuptools is installed in advance (it's too late by the time setuptools is imported).

This is a great way to kill sdists. (why have sdists if they have the same contents as wheels) How quickly the lesson of sourceforge.net has been forgotten.

I'm not sure I follow. Can you elaborate on what the risks are?

IMO setuptools should be moving in the opposite direction, automatically including in sdists the test modules found using test_require or identified using pytest dry-run.

I don't think you want to bundle any dependencies, even those for tests, for the same reasons as you don't want to bundle dependencies (you don't want to have to re-release your sdists every time a dependency updates). You want a lightweight, declarative means to declare dependencies. If bundled dependencies are valuable, a higher-level tool should build on the fundamental building blocks (independent packages) to produce and publish those bundles (similar to how Dockerhub works to host materialized environments).

[pganssle]

This is a great way to kill sdists. (why have sdists if they have the same contents as wheels)

To be clear, this would change nothing about the distinction between sdists and wheels. I personally still recommend that you include your test code in your sdist, but not in the wheel. I also recommend that your tox.ini and any other test configuration should also go into the sdist.

python setup.py test has raised an error in dateutil for a few years now, but it has not affected the content or utility of the source distributions.

[jayvdb]

To be clear, this would change nothing about the distinction between sdists and wheels.

The reason to include tests in sdists, or even create sdists, becomes less obvious if the existing declarative syntax to describe them causes a deprecation warning. This will obviously mean packages with existing working metadata which describes their tests to sdist recipients will need to remove that metadata to be compatible with the latest setuptools. And the latest setuptools wont run the tests any more ... ? So again, why would those package maintainers bother with including tests in the sdist.

Add to that there are members of PyPA who were already pushing projects to not include tests in sdists (because GitHub!), who will use this deprecation as additional justification for being more vocal and aggressive in their stance.

These are not direct outcomes of this issue. However, they are fairly predictable side-effects.

I also recommend that your tox.ini and any other test configuration should also go into the sdist.

Great, but you are not the typical package maintainer. Unfortunately most sdists these days are not testable, usually missing test configuration or test data, until someone from a distro tries to get the sdist working, and then has to try to convince the package maintainer because the Python Packaging Guide says almost nothing about tests. Rather than trying to use @pypa resources and products to address that growing problem, it seems the developers are washing their hands of the problem, and write so many conflicting PEPs that replace each other before a significant segment of the user base has adopted the previous one usually because the implementations were all only partial completed before the authors started writing a different PEP. Compare with the Perl ecosystem, where running the tests is an automatic default part of the package installation experience, and buildbots can (and do!) run the tests of almost every package because they all use consistent metadata for finding and running the tests - packages unmodified for 10 years still build. Tox manages creating virtualenvs and running python(OS-agnostic) commands in them. It does not declaratively describe the test modules nor the test runner (I dont consider the testenv commands = to be declarative.) If setuptools isnt going to allow its metadata to describe unittest test modules, maybe pytest needs to become the recommended test runner, as its metadata in setup.cfg/pytest.ini/etc can at least hold the globs needed to know which modules are test modules and which are not.

[native-api]

The biggest issue with setup.py test now is that it relies on easy_install and setup_requires, both of which are superseded by pip and PEP 517.

If you are going to drastically reduce the setuptools' scope from a complete package management solution to just a build tool, please say that explicitly rather than trying to disguise it as a minor change.
(That said, then you probably want to deprecate setup.py install as well since it relies on the same "superseded" easy_install and bdist_egg. And other *requires arguments as well.)

The biggest issue with setup.py test now is that it relies on easy_install and setup_requires, both of which are superseded by pip and PEP 517

Likewise, if the real reason to deprecate it is that for all practical purposes, it has long been broken (which is true), and you have no desire to fix it, stating that openly is going to remove a whole lot of misunderstanding.

A fix would be to delegate "superseded" stuff to whatever it was superseded by -- to pip for installation and virtualenv (or whatever would work, this is just from the top of my head) to add a custom site-packages location.

---

@graingert

It would be good to drop sdists entirely for projects that ship pure python wheels

How would I get setup.py from a built wheel?

@jayvdb

Compare with the Perl ecosystem, where running the tests is an automatic default part of the package installation experience

Perl is much less scalable than Python and a a result, CPAN modules tend to be very small in comparison. I can't name a single Perl module whose tests consume a few gigabytes and take a few hours to run, making it impractical to run them as a part of every installation.

[pganssle]

If you are going to drastically reduce the setuptools' scope from a complete package management solution to just a build tool, please say that explicitly rather than trying to disguise it as a minor change.

I don't think we've ever tried to disguise this. For several years now we have been actively warning people not to invoke setup.py directly at every opportunity, and we've been quite explicit that the end goal is for setuptools to lose as many of its extraneous features as possible and become a standard build tool.

(That said, then you probably want to deprecate setup.py install as well since it relies on the same "superseded" easy_install and bdist_egg. And other *requires arguments as well.)

Yes, we fully intend to remove setup.py install. At the moment bdist_wheel relies on the install command, which is the only reason we have not started raising warnings.

[native-api]

I don't think we've ever tried to disguise this. For several years now we have been actively warning people

Could you reference some overarching issue on this initiative in the top post then? That would work much better as justification than "some agreement" in some close-circle discussion.

---

At the moment bdist_wheel relies on the install command, which is the only reason we have not started raising warnings.

Raising a warning only if it's invoked via public interface should address that.

[jayvdb]

I can't name a single Perl module whose tests [compare with opencv] consume a few gigabytes and take a few hours to run, making it impractical to run them as a part of every installation.

I am not suggesting that Python packages should run tests by default when installing packages (, or that anyone should use Perl). Sure, PyPI hold entries for much larger codebases than are typically in CPAN modules. Large Perl codebases are often not in CPAN at all, and while there are noble attempts like PDL and BioPerl, most people working on large datasets have moved from Perl to R or Python, so there is more Python development occurring now when data sizes considered almost impossible 10 years ago are common place now. fwiw, R has tests also occurring during installation, and CRAN has similar buildbots, and there are some large R testsuites - not sure if there are any as big as opencv . So it is likely more useful to contrast with R instead of Perl.

It would be nice if there was a new "standard" way to discover tests for any Python packages before setuptools deprecates the existing mechanism that packages are exposing that metadata. Deprecation in software engineering usually means there is a documented replacement which is (roughly) fit for purpose.

[pganssle]

It would be nice if there was a new "standard" way to discover tests for any Python packages before setuptools deprecates the existing mechanism that packages are exposing that metadata. Deprecation in software engineering usually means there is a documented replacement which is (roughly) fit for purpose.

The problem that I think most people in this thread are having is that there currently is no standard way to execute tests in Python. To the extent that setup.py test was ever that way, it has dramatically declined in popularity, partially due to the rise in popularity of pytest and tox and partially due to the fact that setup.py test never really worked all that well to start with. Some projects use make test and some use some custom shell script.

To the extent that we could fix the problems with setup.py test, we would essentially just be re-inventing tox, because the requirements of a non-broken test target would be that it has a declarative configuration format (tox.ini), a clear way to declare dependencies (deps=) and can build and install those dependencies in an isolated environment (this is what tox does).

In any case, pretending that setup.py test is supported or non-broken is not helping anyone. There are multiple good replacements for setup.py test, I think it's time that we start warning people that they should migrate to one of those.

[jayvdb]

The problem that I think most people in this thread are having is that there currently is no standard way to execute tests in Python.

setup.py test is that standard way to execute tests. It has been like that for one and a half decades. It has been baked into uni courses and peoples scripts. It is the basis of Django's manage.py test.

Failing that, unittest test discovery is also standard https://docs.python.org/3/library/unittest.html#unittest-test-discovery

pytest is the very large elephant in the room. A lot of Python direction problems would be solved by declaring that pytest is the new standard (in addition to unittest). Wrap it in a PEP. ;-)

setup.py test never really worked all that well to start with.

It works very well for lots of packages.

It doesnt work well for all packages, of course, but that isnt reason to break it for all packages before there is a good replacement.
PEP 517 didnt cover test running, but it has laid the groundwork for it, and it wont be long before PEP 517/8 build systems incorporate solutions for that problem and/or a PEP is accepted which does standardised the hooks for test running.

setuptools should keep supporting its existing user-base, keep them working correctly, with all of the features they rely on.

With PEP 517/8, new build systems can flourish. The goal should be for setuptools to be deprecating itself, and the deprecation notice provide a list of all high quality replacement PEP 517/8 build systems which have implemented drop-in solutions for the many varied types of projects which setuptools has supported for the last decade or more. That would be in the spirit of PEP 517, and would avoid setuptools existing near-monopoly implicitly preventing other build systems from being used and supported.

[RonnyPfannschmidt]

status quo is setuptools maintaining legacy things with a really thinly stretched set of maintainers, all while nobody actually steps up to make things better

from my pov a "working" solution is the main impediment for having someone step up and do a better/good solution

[native-api]

from my pov a "working" solution is the main impediment for having someone step up and do a better/good solution

A fix would be to delegate "superseded" stuff to whatever it was superseded by -- to pip for installation and virtualenv (or whatever would work, this is just from the top of my head) to add a custom site-packages location.

Any volunteers?

[pganssle]

A fix would be to delegate "superseded" stuff to whatever it was superseded by -- to pip for installation and virtualenv (or whatever would work, this is just from the top of my head) to add a custom site-packages location.

This is also something that we considered, but at the end of the day I think it would be much better to just deprecate these functions because we want to train people to use the right tool for the job, and invoking setup.py is essentially never the right tool for the job - at best it will end up being a thin wrapper around our preferred tool. What's worse is that if you have an old version of setuptools it will end up not being a pass-through to a supported tool, which can screw up your environment.

If we continue supporting these things instead of just warning and eventually erroring out with a message that tells you about the migration path, people will continue with the erroneous belief that executing setup.py is the standard and/or recommended way to do things. They will put it in their documentation and teach it in classes. Instead it's better for us to start communicating clearly that these things are and have been unsupported for a long time and start migrating to the new way of doing things.