Why is caffes Python binding not a package (hosted on PyPI)?

[MartinThoma]

I am wondering why caffe has these installation instructions instead of simply creating a Python package and allowing the user to install it via

$ pip install caffe

If the only reason is time / knowledge how to create such a package, I could do that.

[lukeyeager]

👍

But, you can't really do this without choosing a version number (#486).

[MartinThoma]

Hmmm ... I thought this might be as easy as creating a setup.py like

from setuptools import find_packages
from setuptools import setup
import pip.download
from pip.req import parse_requirements

# parse_requirements() returns generator of pip.req.InstallRequirement objects
install_reqs = parse_requirements('requirements.txt',
                                  session=pip.download.PipSession())

# reqs is a list of requirement
# e.g. ['django==1.5.1', 'mezzanine==1.4.6']
reqs = [str(ir.req) for ir in install_reqs]

config = {
    'name': 'caffe',
    'version': '0.1.0',
    'author': 'BVLC members and the open-source community',
    'packages': find_packages(),
    'scripts': ['bin/classify.py', 'bin/detect.py',
                'bin/draw_net.py'],
    'platforms': ['Linux', 'MacOS X', 'Windows'],
    'url': 'https://github.com/BVLC/caffe',
    'license': 'BSD',
    'description': ('Caffe is a deep learning framework made with expression, '
                    'speed, and modularity in mind.'),
    'long_description': ('Caffe is a deep learning framework made with '
                         'expression,  speed, and modularity in mind. It is '
                         'developed by the Berkeley Vision and Learning '
                         'Center (BVLC) and community contributors.'),
    'install_requires': reqs,
    'keywords': ['caffe', 'deep learning'],
    'download_url': 'https://github.com/BVLC/caffe',
    'classifiers': ['Development Status :: 3 - Alpha',
                    'Environment :: Console',
                    'Intended Audience :: Developers',
                    'Intended Audience :: Science/Research',
                    'License :: OSI Approved :: BSD License',
                    'Natural Language :: English',
                    'Programming Language :: Python :: 2.7',
                    'Programming Language :: Python :: 3',
                    'Programming Language :: Python :: 3.3',
                    'Programming Language :: Python :: 3.4',
                    'Topic :: Scientific/Engineering :: Artificial Intelligence',
                    'Topic :: Software Development',
                    'Topic :: Utilities'],
    'zip_safe': False
}

setup(**config)

adjusting the directory structure and it would be done. It turns out that the _caffe.so is still important. As it can be anywhere, I don't see a way to install caffe without having to give at least this path.

I am not sure if it would still be better to have this on pip (and give instrucutions what to do when the caffe python package detects that the _caffe.so cannot be found).

[MartinThoma]

@lukeyeager I could create the package and add an error message with instructions / a link to the docs when the package does not find _caffe.so. This would make it possible / easier to put the Python documentation online, take a little bit of the work it needs to deal with the _caffe.so file from smarter people than I am (who can eventually make this work only with pip like it's done with numpy) and it could guide Python developers who want to install caffe (as their first reflex might be pip install caffe).

What do you (or other caffe developers) think about that?

[lukeyeager]

Here's what I hear you saying, correct me if I'm wrong:

1. You want to let pycaffe run without the .so file so that you can still generate the documentation
   • That's probably fine. Just make sure it still throws a helpful error if you try to actually use pycaffe without the _caffe.so
2. You want to ship the caffe source with the pip package so that the .so file can be compiled locally with pip
   • That's going to require a lot of testing (since it should work on multiple platforms) and I'd suggest using the CMake build (even though it's still "unofficial") to auto-detect packages installed on the user's system.
3. You're not worried about getting the versioning figured out.
   • I know it seems like a small thing, but I don't think the caffe developers have much of an interest in committing to stable releases with documentation on new features since the last release. I feel like those things are required if you're setting version numbers.

[MartinThoma]

@lukeyeager

1. Yes.
2. No. I don't want to ship any other source code than the Python code as I don't know how this could be automatically compiled.
3. Yes. The Python package would have its own version, starting with 0.1.0. This is independent of the caffe version.

The Python documentation would only document the Python code. So in many cases it would still be necessary to look at the "normal" documentation.

[lukeyeager]

No. I don't want to ship any other source code than the Python code as I don't know how this could be automatically compiled.

Hang on. If they have to get the C++ source from GitHub (which comes with the python source), why is it helpful to also get the python source through pip?

[MartinThoma]

It probably isn't helpful to get the source through pip. But it is my first reflex to try it with pip. If the package (when loaded) tells me that I have to compile the caffe code and add caffe to the PYTHONPATH, I know what to do without searching the right point in the docs.

Advantages of creating a package and putting it on PyPI are:

1. It would automatically install requirements.
2. I hope that somebody who knows how to create a Python package which does also compile C++ code could extend the package to automatically compile the C++ code.
3. Reserving the name. Currently, nobody does use caffe, but that could change. (For example, there is a gi package on PyPI which is not related to GTK ... it took me quite a while to figure that out, as many Python/GTK tutorials begin with from gi.repository import Gtk)
4. Creating the documentation and hosting it on pythonhosted / readthedocs is easier

[lukeyeager]

I see, so this is mostly a documentation/placeholder thing - it wouldn't be super helpful for installing caffe at first, but could become so in the future. Thanks for the clarification.

I think for now we should tell people not to use the pip module, even though it will be there, for these reasons:

1. Coming back to the versioning issue again, this is going to be hell if you have pycaffe v0.1.0 installed, and it's just looking for a _caffe.so, hoping that it's the right version. Ideally, it should be looking for _caffe.so.1.[>=1] or something.
2. You'd be asking the user to put two mismatched versions of the python code on their system, one from PyPI and one from GitHub. They may not understand which one they're supposed to be using.

[MartinThoma]

Coming back to the versioning issue again, this is going to be hell if you have pycaffe v0.1.0 installed, and it's just looking for a _caffe.so, hoping that it's the right version. Ideally, it should be looking for _caffe.so.1.[>=1] or something.

One could make a ~/.cafferc file where one can set configuration parameters. Or an environment variable.

You'd be asking the user to put two mismatched versions of the python code on their system, one from PyPI and one from GitHub. They may not understand which one they're supposed to be using.

The user should not set the PYTHONPATH if he already installed it via pip. Eventually it would be better to make a new Python branch to avoid this issue. So when people check out the master, they don't have the Python code.

[lukeyeager]

One could make a ~/.cafferc file where one can set configuration parameters. Or an environment variable.

How does that help pycaffe find the correct library?

Eventually it would be better to make a new Python branch to avoid this issue. So when people check out the master, they don't have the Python code.

I disagree. I only see pip install caffe being useful if you can also apt-get install caffe, at which point you wouldn't be expected to grab the source at all. Don't hold your breath on that, though.

Maybe there is another direction that we can move this discussion in. If we add the setup.py script to GitHub, we can instruct users to install the python module like this:

$ cd python && python setup.py install

When you install with setup.py (as I understand it), python uses the files in your local directory (e.g. ~/caffe/python/caffe), which is in the same directory as _caffe.so. That will make import caffe work without having to edit PYTHONPATH.

[lukeyeager]

FYI, I tried your setup.py script at ~/caffe/python/setup.py. Is that not where I'm supposed to put it? I got this error:

$ python setup.py install
running install
running bdist_egg
running egg_info
creating caffe.egg-info
(...)
running install_scripts
running build_scripts
creating build/scripts-2.7
error: file '/home/lyeager/caffe/python/bin/classify.py' does not exist

[micahcc]

Alright, so I have a more complete setup.py here: https://github.com/MicahChambers/caffe/commit/024997fd019f43f8f2b2f5cb932a2492b5d2793f

Its probably not what most people want, but anyone wants to build off it, they can. To use you can just point your pip to the branch (probably from a virtualenv)

pip install https://github.com/MicahChambers/caffe/archive/master.zip

This doesn't use your existing binaries, it just recompiles them and installs them into your python site-packages. Not ideal, but its not like the shared objects come out to be that large. OpenCV and other multi-language packages typically include all the bindings in a package (as in in the .deb file, not as pip packages). That would probably be the best here too; but since there isn't a debian package, and I needed a quick and easy way to install caffe without the fuss of creating my own .deb and hosting it myself, this is the best solution for me right now.

No support for GPU or non-atlas blas either.

Edit: Correction about OpenCV.

[arthur-tacca]

This issue conflates two different issues. Even the title reflects that:

1. Make Caffe's Python binding in to a proper Python package.
2. Host it on PyPI.

The first request is fairly easy. It can be restated like this: please include a setup.py in the Python directory. (Moving scripts to a bin/ subdirectory and refactoring them is a related enhancement, but makes no difference from a user's point of view.) As @lukeyeager says, users can then be instructed to install the Python package with setup.py install. Note that they can still access ilsvrc_2012_mean.npy using pkg_resources.resource_stream('caffe', 'imagenet/ilsvrc_2012_mean.npy').

To correct a statement made by @lukeyeager (if I understood it correctly), it is not true that Python then uses the files in the original directory; instead they get copied to the virtual environment that is currently active (which is usually also somewhere in your home directory). But unlike @MartinThoma I suggest assuming that setup.py will only be run after the usual compilation process, so it could just copy everything needed, including the _caffe.so (but not including non-Python stuff like libcaffe.so). A suitably adjusted setup.py can be found here:

Gist of setup.py for Caffe

I have used a slight variant of it (without the scripts line) to install the Python part of Caffe on my own machine and it works well, avoiding PYTHONPATH twiddling and allowing visibility of the caffe package in Python tools like PyCharm.

The second request is quite different. Personally, I don't like the idea of including a Python-only package on PyPI that just tells you that you haven't really installed caffe after all, especially if it complicates implementing the first request. A more ambitious alternative is possible in principle: Python packages can include C dependencies that get built as part of the installation procedure (and pre-built versions can be uploaded that get selected automatically based on the platform). This is at least as difficult as it sounds; OpenCV hasn't managed it either.

[lukeyeager]

@arthur-tacca: well said. Adding a working setup.py is helpful, but uploading to PyPI is not.

@cdluminate: I expect you would be happy to simplify your debian packaging scripts by using dh-python, correct?

[cdluminate]

setup.py is always better than hardcoded installation path. It helps caffe to standarlize its python installation. I'll be happy to adopt this change whatever it simplifies packaging script or not.

[shelhamer]

I agree that a setup.py is welcome. Hosting on PyPI does not seem like it'd help since it requires the Caffe library too. Packaging by apt #2601 or snap #5300 seems more helpful.

[wikier]

Still no progress on this?

[Jayxiaowu]

@MicahChambers
can you teach me how to use pip method to install caffe? thank you very much

[Cloudmersive]

This should really be on Pip. A lot more people would use Caffe if it was one command to install, instead of 25.

[naibaf7]

@Cloudmersive I think it gets further complicated by different "flavours" of Caffe:
Do you want Caffe for the CPU, for CUDA or for OpenCL or for HIP? These need different compiled libraries & dependencies. Additionally... what BLAS should be linked by default, cuDNN etc. it's not an easy question and you don't want to have countless precompiled binaries for every case, I think.

[gfursin]

I agree with @naibaf7 – one may need many different flavors of Caffe with different libraries for different platforms (Intel, ARM, NVIDIA) and different OS to get the best performance. It’s actually the same for Caffe2, TensorFlow, MXNet, etc.

If it’s of interest, we are now working on an alternative approach: we created a simple cross-platform package manager in Python which automates installation/building of Caffe, Caffe2, MXNet, TensorFlow with various sub-dependencies on Linux, Windows, MacOS and Android in a unified way.

You can find examples here:

• https://github.com/dividiti/ck-caffe/wiki/Installation#Installing_Caffe_packages_with_different_dependencies_via_CK
• https://github.com/dividiti/ck-caffe
• https://github.com/ctuning/ck-caffe2
• https://github.com/ctuning/ck-tensorflow/wiki/Installation
• https://github.com/ctuning/ck-mxnet

We also integrated this package manager with scientific workflows – for example it supports recent ACM tournaments on collaborative optimization of various deep learning algorithms across different data sets, models, libs, frameworks and platforms: http://cKnowledge.org/request .

It’s an on-going community project so please stay tuned ;)… We also plan a related event at SuperComputing'18 ...

CC: @psyhtest, @fvella, @ens-lg4

[Dinuz]

@gfursin I checked your work, and honestly is impressive, thanks for it. However, you mentioned in your comment that the package support also MacOs, but I couldn't find any reference in the installation direction.

I did install caffe for example, on MacOs with all the dependency, and it was kind of painful (and honestly very long and time consuming). I got excited by your approach, and I thought it could be incredible to speed things up, but apparently MacOs is not present yet, or am I wrong?

[psyhtest]

@Dinuz Thanks for your interest and kind words about CK! At the moment, CK considers MacOS as a flavour of Linux. With this in mind, you can actually try the Linux instructions, as @ens-lg4 recently fixed quite a few showstoppers (e.g. see closed issues)

[Dinuz]

@psyhtest My pleasure, it's always important recognize the value and the work of people. I am going to follow your suggestions:)

[psyhtest]

@Dinuz thanks again!

One gotcha is that you need to build with LLVM. I will ask @ens-lg4 if he remembers more and can share here (so we can update the docs in due course).

[gfursin]

Hi @Dinuz!

Thank you for your interest and sorry that MacOS support is still weak in the CK. The thing is that our main focus was to create a customizable cross-platform package manager with a common API which can be extended by the community, while focusing on various Linux flavour, Windows and Android which was required by our partners and first users.

Since we started receiving more requests to support MacOS recently, we just started providing it. @ens-lg4 started porting CK-Caffe (and other DNN frameworks) to MacOS so if something is still not working or there are some bugs, please tell us and/or feel free to open tickets to let us reproduce issues and fix them - CK is a community project and we rely a lot on such feedback to improve and fix things there ;) ! This, in turn, will allow CK users to immediately reuse such improved cross-platform packages for MacOS in their high level and platform independent experimental workflows ...

[Dinuz]

@psyhtest thank you so much, it will be really appreciated.

@gfursin sorry for what? Thank you for the work you have done already. I am sure the MacOs support will come soon (at the end of the day MacOS is unix flavored system). I am going to test the suit, and I will be more than happy to provide feedbacks and eventually help.
You guys are doing an interesting and fascinating work, and I am already very grateful.
BTW Nice to virtually meet the two of you @psyhtest and @gfursin

[gfursin]

Nice to virtually meet you too, @Dinuz, and thanks for your encouragement - very appreciated! I am sorry because I am myself often frustrated when things are not working as expected ;) ! I believe @ens-lg4 had a good progress last week building Caffe on MacOS via CK while I updated all ck-caffe* packages to support latest compilers and libraries on Windows, Linux and Android.
@ens-lg4 - may I ask you to send us an update when to try CK-Caffe on MacOS, please?

BTW, I also added support for virtual environments similar to Python but for binary packages built via CK (CK enables co-existence of multiple versions of tools useful to test various functionality or compare versions):

$ ck install package --tags=lib,caffe --env.CAFFE_BUILD_PYTHON=ON
$ ck show env
$ ck virtual env --tags=lib,caffe
$ python
$ > import caffe

Have a good week!

[wilderfield]

Hey, its 2018 now. Have we lost passion for pip install caffe ?

[rawells14]

Caffe user here chiming in from 2019 - Still no pip support?

[400lbhacker]

becase the people who made caffee themselves were neglectful , purposely sabotaged their own project and ease of installation by yanking the plug on their fast no dependancy pre built win32 binary caffe exe's and instead, due to increasing instabilty (as a result of their own malifence) have decided that the program is more suited to being built from scratch from increasinglier heavier and bloated enviroments, frameworks and enviromental preconfiguraitions rather than 1 click pure win32/64 lean c++ exe

why do i need python, to run a npm, to run a scipy, to run a conda to run a pip, to run a ipython-/jupyter notebook to run an pycaffe or pytorch or luajit or caffe?!?!?! That IS the problem with caffe. if their was a 1 step pip3 or conda install of caffe theird be millions of people using it of course, would be useful, but lets rerember you are going out of your way to do extra steps to accomplish something that is superior with much less effort (focus on win32 precompiled binaries and low level hook/wrappers to keep stability, and than once the prebuilt exes are made public again, you write your Pip3 install, which would only be like 6 lines long and the press of the (y) button to get it all installed

[arianaa30]

OMG! Why I can't install Caffe on Windows?! 2019 is ending and still no pip/conda support?!

[arthur-tacca]

@arianaa30 Caffe has been superseded by Caffe2 (and PyTorch, which merged with Caffe2). I believe there is no new development on the old Caffe 1.x, so it seems particularly unlikely anyone will go to the effort of producing Windows wheels.

BTW, I don't believe that liking your own comment gives it any more weight.

[arianaa30]

@arthur-tacca thanks for the info. So caffe2 is already easy to install? (w/ conda or pip?)
Yeah I was testing the like feature...looks like one can like itself, and that multiple times!