Support GitHub tarballs

[jayvdb]

Sometimes it is useful/necessary to fetch the source tarballs from the git/hg/etc repo, especially to get tests which are excluded from the PyPi package.

Also, it would be nice if py2pack could ensure this approach is done in a safe manner, using PyPi for the main content, and then splicing in extra content in the source repo's tarball.

[jayvdb]

The format I have used for https://build.opensuse.org/package/show/home:jayvdb:pyexcel/python-pyexcel-io

Source1:        https://github.com/pyexcel/pyexcel-io/archive/v%{version}.tar.gz
...
%prep
%setup -q -n pyexcel-io-%{version}
tar --wildcards --strip-components=1 -xzf %SOURCE1 pyexcel-io-%{version}/tests

[jayvdb]

Example decline for tests not being included from the GitHub repo : https://build.opensuse.org/request/show/670793

Another package I recently ran into where it would be nice to have the tests automatically added https://build.opensuse.org/request/show/666395 , though in that case the tests were not passing.

[jayvdb]

An example where the maintainer has shown, in the past, to be reticent to including tests, citing that they are available on GitHub ryanmcgrath/twython#507

[scarabeusiv]

Nah, it is easier to just switch to github tags if the upstream is semi-dead or refusing to ship the tests rather than use syntax like this. We already have cca 100 packages in main repo doing that.

For us it really matters not where from the first source is obtained from, esp. if upstream is stubborn.

[scarabeusiv]

Also one of the things why we should just switch is space consideration, less we consume on the VCS the better, thus 2 almost exact copies of archive are way worse than 1.

[toabctl]

Also one of the things why we should just switch is space consideration, less we consume on the VCS the better, thus 2 almost exact copies of archive are way worse than 1.

@scarabeusiv with "just switch" you mean we should use github tag tarballs instead of pypi tarballs?

[scarabeusiv]

Yep in a case where nobody wants to tweak the package on pypi for our needs just use the github tag (of course you are out of luck if even those tags are missing).

By no means this should be default behaviour, but if it allows us to execute the tests...

[toabctl]

By no means this should be default behaviour, but if it allows us to execute the tests...

ok. good. we are in sync then :)

[jayvdb]

SO py2pack can look for tests in the pypi package, and if there are none then follow a GitHub link to find tests in the repo? And then use GitHub is that is the only place the tests were found?

[jayvdb]

https://github.com/jayvdb/pypidb finds the GitHub repo for almost any PyPI package.

Currently I am checking all packages in devel:languages:python* prefixed python[23]-*, and there are about 40 such packages it cant resolve to an SCM.

The packages I couldnt/didnt map to PyPI names are in https://github.com/jayvdb/pypidb/blob/ba7740e/tests/datasets.py#L287 . There is a mapping of openSUSE names to PyPI names directly above that, which may be incomplete.

The list of those PyPI packages in openSUSE that do not yet find an SCM is at https://github.com/jayvdb/pypidb/blob/master/tests/test_osc.py as "expected_failures" . I suspect some of those will be missing mappings from openSUSE names to PyPI names, which the team here might be able to spot quickly. The tests do check packages in devel:languages:python:misc, and I tried to resolve most of them which could help with analysing/updating them and moving them into devel:languages:python where appropriate, but I am ignoring failures because there are quite a few that are mostly old / no longer relevant, so effort was better spent elsewhere.

[mcepl]

The tests do check packages in devel:languages:python:misc, and I tried to resolve most of them which could help with analysing/updating them and moving them into devel:languages:python where appropriate, but I am ignoring failures because there are quite a few that are mostly old / no longer relevant, so effort was better spent elsewhere.

Actually, if you see packages which are too obsolete/broken/whatever even for d:l:p:misc don't hesitate to delete them/suggest their deletion, please.

[mcepl]

I don't see this ticket leading to any actionable item in my TODO list. Please, reopen with the appropriate argument, if you feel otherwise.