Caffe preset for Android and Windows

[bhack]

Do you think will be possible to add preset for Caffe?

[saudet]

Sure, that's actually a topic @paul-hammant would be interested in working on. Paul, comments?

[saudet]

Please let me know if you encounter any issues with the new presets for Caffe! Thanks

[bhack]

Thanks for the effort. I share this to caffe community to test it.

[bhack]

API DOC seems has not been generated.

[saudet]

Sure, not released yet. Are you having problems generating it yourself?

[bhack]

I will try tomorrow at work

[saudet]

It's been released with version 0.11! Thanks for bringing this to my attention. Currently, there are binary files only for linux-x86_64, but I hope you and others will soon contribute builds for other platforms as well.

[futurely]

The Caffe master now acts as the only development branch which means that it may change the public APIs without warnings. For example, the compute mode is no longer global but belongs to a net or a layer. So the current caffe tool example of this preset does not work against the latest master branch of Caffe.

It's better to build against a specific released version of Caffe.

[benjaminklein]

Thank you for creating this! From a brief look at your code - I see how one can train a new model.

Could one also use this in order to classify images with an existing (pre-trained) model of caffe. For example - download the VGG model (http://www.robots.ox.ac.uk/~vgg/research/very_deep/), load it in Java and use your code in order to classify new images? Could you give a code example of how do to this?

[futurely]

https://github.com/bytedeco/javacpp-presets/tree/master/caffe#the-srcmainjavacaffejava-source-file

[saudet]

@futurely I see there's rc2 that's not too old indeed. I'll keep tracking master for now, but will switch to a release, on the next release. Thanks for the feedback! BTW, I've just rebuilt with the latest source code, and the sample still works just fine.

@benjaminklein I'm new to Caffe myself, but if you have some working code in C++, I can help porting that to Java!

[futurely]

The native library cannot be loaded using Caffe.set_mode(Caffe.CPU); // Instantiate the caffe net. or Loader.load(org.bytedeco.javacpp.caffe.class);. The loader has to extract the native library from the jar to a temporary path like @adamheinrich did.

Exception in thread "main" java.lang.UnsatisfiedLinkError: no jnicaffe in java.library.path
    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1886)
    at java.lang.Runtime.loadLibrary0(Runtime.java:849)
    at java.lang.System.loadLibrary(System.java:1088)
    at org.bytedeco.javacpp.Loader.loadLibrary(Loader.java:536)
    at org.bytedeco.javacpp.Loader.load(Loader.java:411)
    at org.bytedeco.javacpp.Loader.load(Loader.java:354)
    at org.bytedeco.javacpp.caffe.<clinit>(caffe.java:12)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:270)
    at org.bytedeco.javacpp.Loader.load(Loader.java:386)
    at org.bytedeco.javacpp.Loader.load(Loader.java:354)
    at org.bytedeco.javacpp.caffe$Caffe.<clinit>(caffe.java:548)
    at gr.iti.mklab.visual.examples.CaffeFeatureExtractionAndIndexing.main(CaffeFeatureExtractionAndIndexing.java:79)
Caused by: java.lang.UnsatisfiedLinkError: /tmp/javacpp39595883066632/libjnicaffe.so: libhdf5_hl.so.8: cannot open shared object file: No such file or directory
    at java.lang.ClassLoader$NativeLibrary.load(Native Method)
    at java.lang.ClassLoader.loadLibrary1(ClassLoader.java:1965)
    at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1890)
    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1851)
    at java.lang.Runtime.load0(Runtime.java:795)
    at java.lang.System.load(System.java:1062)
    at org.bytedeco.javacpp.Loader.loadLibrary(Loader.java:525)
    ... 9 more

Second problem is that it will stop 99% if not all of the users from trying this library if they have to manually (re-)install all of jnicaffe's dependencies everywhere they need to deploy their Java programs. The dependent libraries have to be released all together. The following two scripts can extract all of them.

list_dependencies.sh

#!/usr/bin/env bash

ROOT_DIR="$( cd "$(dirname "$0")"/.. ; pwd -P )"
SCRIPTS_DIR=$ROOT_DIR/scripts
BUILD_DIR=$ROOT_DIR/build
BIN=your_binary

# cd $BUILD_DIR

LD_LIBRARY_PATH=........

ldd ../$BIN > ldd.out

#strace -f -o strace.out 2>&1 $ROOT_DIR/$BIN

strace -f $ROOT_DIR/$BIN &> strace.out 

cat strace.out | grep -v '(No such file or directory)' | grep -v '<unfinished ...>' | grep 'open(.*\.so.*\"' > strace.txt

$SCRIPTS_DIR/parse_dependencies.py ldd.out strace.txt | sort > libs.txt

parse_dependencies.py

#!/usr/bin/env python

import re
import sys
import os
import sets
from subprocess import Popen, PIPE

def is_file(file):
    if file is None:
        return false
    return os.path.isfile(file)

def extract_path(pattern,  line,  group_nr):
    match = pattern.search(line)
    path = None
    if match:
        file = match.group(group_nr)
        if is_file(file):
            path = file 
    return path

def run(cmd,  std = 1):
    try :
        retline = []
        cmdList = cmd.split()
        if std == 1:
            ret = Popen(cmdList,  stdout = PIPE,  stderr = open('/dev/null')).stdout
        elif std == 2:
            ret = Popen(cmdList,  stderr = PIPE,  stdout = open('/dev/null')).stderr
        for x in ret.readlines() :
            retline.append(x.strip())
        return retline
    except:
        pass
##    except IOError as Ierr :
##        sys.stderr.write('%s not found : '%cmd+str(Ierr)+'\n')

def ldd_dependencies(file_path):
    lines = open(file_path).readlines()
    libs = set()
    for line in lines:
        if not(line):
            continue
        # Not all ldd lines include => they might
        # reference the absolute path of the library
        if line.find('=>') == -1 and line.find('/') == -1:
            continue
        if line.find('=>') != -1:
            lib = line.split('=>')[1].split('(')[0].strip()
        else:
            lib = line.split('(')[0].strip()
        libs.add(lib)
    return libs

def strace_dependencies(file_path):
    lines = open(file_path).readlines()
    libs = set()
    for line in lines:
        line = line.strip()
        if line:
            line = line.split()
            if len(line) > 1:
                line = line[1]
                if line.startswith('open("'):
                    start = len('open("')
                    lib = line[start : line.find('"', start)]
                    libs.add(lib)
    return libs

def main():
    if len(sys.argv) < 2:
        print 'usage: parse_dependencies ldd.out strace.out'
        exit(1)

    libs = ldd_dependencies(sys.argv[1])

    strace = strace_dependencies(sys.argv[2])

    libs = libs.union(strace)
    for lib in libs:
        lib = lib.strip()
        if lib.startswith('/'):
            print lib
    return

if __name__ == '__main__':
    main()

[futurely]

Some users reported that other javacpp libraries may also suffer from the above two problems. My personal experience was that projects usually worked well in my development environments because I've spent a lot time installing dependencies and properly working around various corners cases. But there were always surprises when the projects were deployed into production environments. A lot of the efforts in the Caffe communities aimed to keep cross-platform compatibility. Even after they set up a Travis continuous integration matrix to build for the combinations of popular platform options, new compatibility issues were still reported frequently. It is far from a trivial problem. But a CI infrastructure is certainly the most effective and efficient way to find out various test failures as early as possible.

[saudet]

@futurely Of course, I totally agree with you: We absolutely need CI. We just need someone to actually do it! :) There's some info here: #22. Would you be interested in starting with Caffe on Linux? The script would be good for CI actually. Let's put it in the build! Thanks

BTW, to bundle dependencies in a cross-platform manner, we currently need to list them manually in @Platform(preload=...), like here for example:
https://github.com/bytedeco/javacpp-presets/blob/master/leptonica/src/main/java/org/bytedeco/javacpp/presets/lept.java#L33
But Caffe has so many dependencies, I gave up. We could automate that if we find a nice way to port your script to Android, Mac OS X, and Windows. Anyway, I feel we should prioritize CI for now. What do you think?

[futurely]

Not that many dependencies. Here are the script to build a library simply wrapping Caffe and the required libraries. The versions of the libraries are more important than the number of them to run on different platforms.

#!/usr/bin/env bash

ROOT_DIR="$( cd "$(dirname "$0")"/.. ; pwd -P )"
BUILD_DIR=$ROOT_DIR/build
NCPUS=`grep -c ^processor /proc/cpuinfo`
PREFIX=$(eval echo ~${SUDO_USER})/usr
PATH=$PREFIX/bin:$PATH
LD_LIBRARY_PATH=$PREFIX/lib:$LD_LIBRARY_PATH

rm -rf $BUILD_DIR
mkdir -p $BUILD_DIR
cd $BUILD_DIR
cmake -DCMAKE_PREFIX_PATH=$PREFIX -DCMAKE_BUILD_TYPE=Debug -DCPU_ONLY=ON ..
make -j $NCPUS

RHEL 5.7, gcc 4.1

/home/username/usr/lib/libboost_system.so.1.57.0
/home/username/usr/lib/libboost_thread.so.1.57.0
/home/username/usr/lib/libcaffe.so
/home/username/usr/lib/libgflags.so
/home/username/usr/lib/libglog.so.0
/home/username/usr/lib/libhdf5_hl.so.9
/home/username/usr/lib/libhdf5.so.9
/home/username/usr/lib/libleveldb.so.1
/home/username/usr/lib/liblmdb.so
/home/username/usr/lib/libopenblas.so.0
/home/username/usr/lib/libopencv_core.so.2.4
/home/username/usr/lib/libopencv_highgui.so.2.4
/home/username/usr/lib/libopencv_imgproc.so.2.4
/home/username/usr/lib/libprotobuf.so.8
/home/username/usr/lib/libsnappy.so.1
/lib64/ld-linux-x86-64.so.2
/lib64/libc.so.6
/lib64/libdl.so.2
/lib64/libexpat.so.0
/lib64/libgcc_s.so.1
/lib64/libglib-2.0.so.0
/lib64/libgmodule-2.0.so.0
/lib64/libgobject-2.0.so.0
/lib64/libgthread-2.0.so.0
/lib64/libm.so.6
/lib64/libpthread.so.0
/lib64/librt.so.1
/lib64/libz.so.1
/usr/lib64/libatk-1.0.so.0
/usr/lib64/libcairo.so.2
/usr/lib64/libfontconfig.so.1
/usr/lib64/libfreetype.so.6
/usr/lib64/libgdk_pixbuf-2.0.so.0
/usr/lib64/libgdk-x11-2.0.so.0
/usr/lib64/libgfortran.so.1
/usr/lib64/libgomp.so.1
/usr/lib64/libgtk-x11-2.0.so.0
/usr/lib64/libidn.so.11
/usr/lib64/libjpeg.so.62
/usr/lib64/libjson.so.0
/usr/lib64/libpango-1.0.so.0
/usr/lib64/libpangocairo-1.0.so.0
/usr/lib64/libpangoft2-1.0.so.0
/usr/lib64/libpng12.so.0
/usr/lib64/libstdc++.so.6
/usr/lib64/libX11.so.6
/usr/lib64/libXau.so.6
/usr/lib64/libXcursor.so.1
/usr/lib64/libXdmcp.so.6
/usr/lib64/libXext.so.6
/usr/lib64/libXfixes.so.3
/usr/lib64/libXinerama.so.1
/usr/lib64/libXi.so.6
/usr/lib64/libXrandr.so.2
/usr/lib64/libXrender.so.1
/usr/local/lib/libjson-c.so.2

[futurely]

Yes, CI is definitely a top priority. I experimented with ElasticSearch and Solr Cloud last week. Elastic{ON} featured many very high profile users. But starting the latest release of ElasticSearch ended up with ElasticsearchIllegalStateException: Can't recover from node. Googling led to the unsolved GitHub issues discussing the causes of the exception. On the other hand, Solr Cloud runs out of the box in a few minutes without any issue. A cluster comprised of two nodes was set up automatically, the example scripts indexed the docs of various formats smoothly and the management portal provided convenient faceted searching capability. You know which one I chose.

[saudet]

Caffe adds a lot more dependencies than that when built on Fedora. It would appear to be a good thing to have it built on CentOS/RHEL. :)

So, let me know if you need anything from me to get started with CI. Thanks a lot!!

[futurely]

To make Caffe preset work with the native libraries, some issues have to resolved by Caffe itself because the dependencies inevitably evolve with the official project, e.g. BVLC/caffe#1188, BVLC/caffe#1738 (BVLC/caffe#1074), BVLC/caffe#2240, and BVLC/caffe#1782. If you can promote the priority of the above related issues or introduce some specialists on them to the Caffe owners to get them solved, the dependencies will no longer be a messy nightmare and this issue will be much easier to handle with. Thanks!

[saudet]

These issues don't necessarily need to be fixed within Caffe. When a piece of software starts depending on so many things, we usually resort to package managers. Under Linux, we have things like RPM that work fine, Homebrew is the recent fad under Mac OS X, and Windows doesn't have anything yet, but they're going to put something in Windows 10, apparently. CMake isn't a package manager, so that's not going to work (well). Here with the JavaCPP Presets, we're trying to do that on the Java platform, which means across platforms on the native side, something that no one has ever attempted to do before. So, any ideas about how to go about it are welcome, indeed. For example, we could port your Python script to Groovy, and use that within Gradle, making it available to all platforms without any additional dependencies! That sounds like a step in the right direction, but I'm open to anything really.

One big problem though is CUDA. The way it's designed doesn't play well with package managers, but we can start without considering CUDA for now. That's how I build OpenCV at the moment and have not received any complaints yet, but the situation might be different for Caffe...

[bhack]

@saudet @futurely @sh1r0 Do you see other modularity that could be added to this PR that could simplify android build and caffe-preset?

[saudet]

Well, I think we should first try to make a build with the least number of dependencies so we don't have to port everything to Android right away...

[bhack]

@saudet If can take a look at @sh1r0 work he has already test with a little jni interface in https://github.com/sh1r0/caffe-android-lib and modified Caffe https://github.com/sh1r0/caffe/tree/mobile with some manual hack to let compile on NDK but his Caffe branch it isn't in sync with master. I think that giving feedback in this PR we could prepare the path to let to have an integrated android build in official Caffe master branch.

[saudet]

Sure, it would be great. Could you give it a try and add the build commands to the cppbuild.sh file yourself?

[bhack]

I think that an android-arm) case cannot be already covered actually. Other than improving this modularity for android we need to let caffe build with https://github.com/taka-no-me/android-cmake as you do in opencv cppbuild.sh in android cases.

[futurely]

@bhack
Ideally, Caffe could use mature multi-platform matrix/tensor libraries such as ArrayFire which supports Android. But that wouldn't happen in the near future since it is quite difficult to replace one of the most fundamental classes of Caffe without introducing quite a lot of compatibility and probably also stability issues.

[xjock]

The window Caffe is now available.

https://github.com/Microsoft/caffe