Include tests in PyPI source tarball

[jluttine]

It would be nice if the PyPI source tarball included the tests so one can run them after installing to check that it works. Running package tests is extremely useful and important when packaging for a Linux distro as I'm doing at the moment, so I'd love to have the tests included. I think this is good practice.

[wholmgren]

I'm not opposed but is this the standard for other "pydata" projects? I usually look to pandas (big), xarray (medium), metpy (smaller) as examples to try to follow. How do they manage this?

[jluttine]

Pandas and xarray include tests in PyPI source tarball. For some reason, metpy doesn't.

Some references I found quickly:

• https://docs.python.org/2/distutils/sourcedist.html#specifying-the-files-to-distribute
• https://stackoverflow.com/a/20180000

[jluttine]

Also NumPy and SciPy include tests in the source tarball in PyPI.

[mikofski]

I usually put tests and documentation source in the sdist or source distribution which is uploaded to PyPI as a tarball or zip file, but not in the wheels or other binary distributions.

If the tests include a lot of test data then it would be an unnecessary burden to put the in wheels, but I suppose if they are smell enough then it doesn't matter.

There was a discussion about this on the SciPy Dev mail list, in their case the tests were so big that they're considering moving the test data to a separate repo altogether, and either accessing it through an API or having testers clone the repo. There is an upside to have a shared public test data repo for multiple projects with similar test data, then the data can also be used in the docs as examples and the examples can be more realistic or rocket in their demonstrative power

[wholmgren]

With only a quick read of our manifest/setup.py and comparing with xarray/pandas, I am surprised that our tests are not already bundled. What do we need to do differently? Pull request welcome. Or edit the release procedures wiki.

Note that the test_forecast.py tests frequently fail due to issues with the data on the server (I think) so you'd probably want to exclude those and/or improve them.

A separate test repo sounds like a lot of work to me.

[mikofski]

There already is a shared test data repo for R, I think maybe it's datasets there are efforts in Python to read them I think? As I said you can read the thread in SciPy devs starting in March

[wholmgren]

closed by #580