Improving Python packaging

[jakirkham]

I recently packaged caffe for conda-forge. I noticed a few things about the Python install that could be improved to make it easier for users to install and maintain the package thereof.

One of the things I noticed is there wasn't a proper setup.py and that the Python files were not getting placed in site-packages. Further the executables were not being placed in the right directory and lacked the right permissions.

Another issue related to the executables were designed as if to be full-blown importable modules. In practice, it is better to separate out the executable portion into a small shim that gets installed elsewhere and has proper permissions. This can be done by either using entry_points or making your own shims. For doing the former, take a look at this example from conda itself. Here is an example of the latter with Jupyter's notebook.

As this stuff can be a bit tedious to get right, I would recommend trying something like cookiecutter to get this all setup. This can be pip installed. We also have it conda-forge if you wish to go that route.

[lukeyeager]

Here's a little context, some of which may be relevant:

there wasn't a proper setup.py

See discussion at #2440.

the Python files were not getting placed in site-packages

See discussion at #3925.

[jakirkham]

there wasn't a proper setup.py

See discussion at #2440.

On the first point, I would like to clarify I also think pip install caffe or similar is laughably ridiculous (mainly thinking about the long list of non-Python dependencies). There is a reason I built it with conda instead. 😉

Maybe if conda packages could be converted to wheels (though probably many changes on the wheels side of the fence) this might be possible, but that is a bit of a pipe dream. 💭 Sorry, I digress.

Mainly I was think of using setup.py as tool to place everything where it suppose to live. Honestly I don't care if this is done in CMakeLists.txt or something else if that is easier.

the Python files were not getting placed in site-packages

See discussion at #3925.

Happy to see this is an idea you are willing to accept (at least according to the cited issue).

Again any strategy that is picked here is an improvement. This could be done with setup.py, but it could be done other ways too. I did something to get this packaged, which is a bit of a hack, but feel free to lift it if it helps you. Here's the snippet. The ugliest part is handling the executables from bin. Still some renaming and possibly splitting of things here would do the trick and obviate that.

[lukeyeager]

I'm pretty sure the @BVLC team would welcome any pull requests you have to add a working setup.py script or to help install pycaffe to the correct system location. I know I'd like to see some improvements in these areas.

[jakirkham]

As a follow-up on our previous conversation, @shelhamer, I have added this issue on packaging improvements.

[jakirkham]

Any thoughts on this from the @BVLC team? I'd like to help improve this, but I don't want to start until we have some consensus about what needs to be done.

[willyd]

For a windows conda package to be possible some dependencies are missing on conda-forge:

• glog (should be pretty easy to add since glog as a pretty good CMake build for windows)
• leveldb (optional and lmdb is now the default backend)
• lmdb
• openblas?

[jakirkham]

While that is an interesting question and I certainly would be interested to hear your thoughts, that is off topic w.r.t. this issue. The concern this issue is focused on is how to properly handle the installation of Python scripts. This is not an OS or conda-forge specific question.

As you seem interested in the Windows packaging of Caffe on conda-forge, I invite you to share your thoughts in issue ( conda-forge/caffe-feedstock#3 ), @willyd . 😉

[shelhamer]

To echo @lukeyeager we would welcome pull requests for helping package up pycaffe the right way.