Builds on Travis CI not passing

[xolox]

While developing py2deb I've been extending and running the test suite continuously for months now and that was working great. When I published py2deb on GitHub a couple of days ago I got my first chance to run the py2deb test suite on Travis CI and I found out and that it fails consistently on all Python versions :-(.

Because there are multiple ways to fix this problem and all of them are somewhat involved / complex, I will use this GitHub issue to summarize my thoughts and weigh the pros and cons of each approach.

My local testing environment

To summarize my local testing environment:

• Ubuntu Linux 12.04, 64 bits
• Default Python version is Python 2.7.3 installed from official repositories.
• Additional Python versions installed from deadsnakes PPA:
  • Python 2.6.9
  • Python 3.4.2
• Using Tox to run test suite against Python 2.6, 2.7 and 3.4

The Travis CI testing environment

To summarize what I know and have inferred (assumed) about the Travis CI testing environment:

• Travis CI workers are also running Ubuntu 12.04
• I believe that in the past Travis CI was using the deadsnakes PPA
  • I can't find anything about this in the online documentation (just dead references from external sites)
  • My findings with the py2deb test suite seem to point out that Travis CI is using custom compiled Python binaries, even for Python 2.7 (the system default on Ubuntu 12.04)

At present it seems like Travis CI is using a Chef cookbook that uses pyenv to install custom compiled Python binaries to run test suites against. Some references:

• https://github.com/travis-ci/travis-cookbooks/blob/master/vm_templates/common/python.yml references python::pyenv
• https://github.com/travis-ci/travis-cookbooks/blob/master/ci_environment/python/recipes/pyenv.rb shows what python::pyenv does
• All of this eventually leads to https://github.com/yyuu/pyenv which gives more background information about pyenv in general

Why the py2deb test suite fails

Due to its goals py2deb obviously integrates quite a bit with Debian Python packaging. One such bit of integration is my (ab)use of dpkg-shlibdeps. It seems to me that the main issue on Travis CI right now is that dpkg-shlibdeps immediately errors out because *.so files are compiled against a Python binary that is not packaged. The same basic issue is described in the StackOverflow thread dpkg-shlibdeps: error: no dependency information found for.

Ways to get the test suite running

1. Don't use the Travis CI installed Python versions, instead start from a clean Ubuntu 12.04 install, manually use the deadsnakes PPA to get Python 2.6 and 3.4 installed and use Tox to run the test suite against all of them.
   • Pro: Fairly quickly to get going again, allowing me to focus on "more important things".
   • Con: When done the naive way this would force the 2.6/2.7/3.4 tests to run in serial, but that can be fixed by creative use of a a build matrix based on custom environment variables.
   • Con: Not using Travis CI the way it's meant to be used. Then again, Travis CI is just a tool, and their defaults don't necessarily hold for every project in the world (which they acknowledge).
2. Instead of side stepping Travis CI's use of custom compiled Python binaries, dive deeper into dpkg-shlibdeps and see if we can get it to ignore these issues but keep looking for other dependencies.
   • Pro: I looked into the Perl sources a bit and that seemed impossible, but the previously linked StackOverflow thread suggests using the --ignore-missing-info option. Maybe I'll use that as my first option (easiest to try out).
   • Con: Users of py2deb should realize when proper dependency information is not generated. In other words, when dpkg-shlibdeps fails I want py2deb to fail, because blindly marching on could mean a binary Debian package is generated which is missing essential dependencies on system libraries! So just ignoring error codes could get awkward.
3. The feature provided by the integration of dpkg-shlibdeps is very useful and I don't want to lose that, but clearly the way I'm running dpkg-shlibdeps is a huge hack waiting to bite me in the ass. I've been itching to replace dpkg-shlibdeps because of this, but as I mentioned I looked into its sources and there are literally thousands of lines of (for me) indecipherable Perl code underneath.
   • Pro: A reimplementation dedicated to py2deb could provide exactly what py2deb means, without the gross hacks required to get dpkg-shlibdeps running.
   • Con: Can I really be excused for wasting time porting all of that to Python, making the same bugs that inevitably occurred during the development of dpkg-shlibdeps? (a rhetorical question; I know the answer ;-) I'll also need to maintain all of this in later Debian/Ubuntu releases...

Right now I will investigate option two because it's the most light weight approach with the least potential to bite me (the py2deb project) in the ass later on.

[xolox]

Conclusion:

• py2deb expects to be running against a Python binary installed via Debian packages (because of dpkg-shlibdeps). Running within a virtual environment based on a Python binary installed via Debian packages is fine, which explains why I've never run into this. The problem only manifests when running against a Python that was compiled from source and installed outside the Debian packaging system.
• Because of the intended use case of py2deb I don't see this as a real problem, it was just a bit awkward to get the automated test suite going on Travis CI. In the end I went with option one: I used the deadsnakes PPA in combination with Tox to enforce consistency.
• I'll include a mention of this in the readme so potential users of py2deb don't trip over this behavior, or at least when they do they can find an explanation of the problem.

Because the test suite is now properly running on Travis CI I'm closing this issue.