Unable to fully build (Docker image), tests failed for caffe-distri #205

baristahell · 2016-12-06T10:00:35Z

So, I'm trying to build a docker image for CaffeOnSpark, but the make step doesn't work. I based my Dockerfile on the CaffeOnSpark tutorial and added OpenBlas and Protobuf (had issues because they were missing). Still, I have issues about the test step that prevent me from successfully building the image. Can you help me? I don't really get it.

RUN cd ${CAFFE_ON_SPARK} &&   make build
cd caffe-public; make proto; make -j4 -e distribute; cd ..

(It seems to work until there but i start getting some warnings. Not sure if they're that problematic though)

[INFO] Scanning for projects...
(Downloading everything)
[WARNING] 
[WARNING] Some problems were encountered while building the effective model for com.yahoo.ml:caffe-grid:jar:0.1-SNAPSHOT
[WARNING] The expression ${version} is deprecated. Please use ${project.version} instead.
[WARNING] 
[WARNING] It is highly recommended to fix these problems because they threaten the stability of your build.
[WARNING] 
[WARNING] For this reason, future Maven versions might no longer support building such malformed projects.
[WARNING] 
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO] 
[INFO] caffe
[INFO] caffe-distri
[INFO] caffe-grid
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building caffe 0.1-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building caffe-distri 0.1-SNAPSHOT
[INFO] ------------------------------------------------------------------------
(Downloading from repo.maven)
[INFO] Executing tasks

protoc:
     [exec] make[1]: Entering directory `/root/CaffeOnSpark/caffe-distri'
     [exec] PROTOC ../caffe-public/distribute/proto/caffe.proto
     [exec] protoc --proto_path=../caffe-public/distribute/proto --java_out=src/main/java ../caffe-public/distribute/proto/caffe.proto
     [exec] make[1]: Leaving directory `/root/CaffeOnSpark/caffe-distri'
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-compiler-plugin:3.0:compile (compile-generator) @ caffe-distri ---
(Downloading from repo.maven)
[INFO] Changes detected - recompiling the module!
[WARNING] File encoding has not been set, using platform encoding ANSI_X3.4-1968, i.e. build is platform dependent!
[INFO] Compiling 10 source files to /root/CaffeOnSpark/caffe-distri/target/classes
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[39561,19] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[39561,20] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[39571,19] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[39571,20] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[39581,19] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[39581,20] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[40371,19] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[40371,20] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[40384,19] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[40384,20] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[40396,19] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[40396,20] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[41716,21] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[41716,22] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[41729,21] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[41729,22] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[41741,21] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[41741,22] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[41753,21] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[41753,22] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[41769,21] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[41769,22] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[41784,21] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[41784,22] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[41800,21] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[41800,22] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java: /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java uses or overrides a deprecated API.
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java: Recompile with -Xlint:deprecation for details.
[INFO] 
[INFO] --- maven-resources-plugin:2.3:resources (default-resources) @ caffe-distri ---
(Downloading from repo.maven)
[WARNING] Using platform encoding (ANSI_X3.4-1968 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory /root/CaffeOnSpark/caffe-distri/src/main/resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.0:compile (default-compile) @ caffe-distri ---
[INFO] Changes detected - recompiling the module!
[WARNING] File encoding has not been set, using platform encoding ANSI_X3.4-1968, i.e. build is platform dependent!
[INFO] Compiling 10 source files to /root/CaffeOnSpark/caffe-distri/target/classes
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[39561,19] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[39561,20] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[39571,19] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[39571,20] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[39581,19] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[39581,20] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[40371,19] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[40371,20] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[40384,19] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[40384,20] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[40396,19] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[40396,20] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[41716,21] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[41716,22] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[41729,21] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[41729,22] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[41741,21] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[41741,22] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[41753,21] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[41753,22] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[41769,21] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[41769,22] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[41784,21] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[41784,22] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[41800,21] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[41800,22] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java: /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java uses or overrides a deprecated API.
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java: Recompile with -Xlint:deprecation for details.
[INFO] 
[INFO] --- maven-antrun-plugin:1.7:run (make) @ caffe-distri ---
[INFO] Executing tasks

make:
     [exec] make[1]: Entering directory `/root/CaffeOnSpark/caffe-distri'
     [exec] javah -d include/jni -force -cp target/classes \
     [exec] 	com.yahoo.ml.jcaffe.Mat
     [exec] javah -d include/jni -force -cp target/classes \
     [exec] 	com.yahoo.ml.jcaffe.FloatBlob
     [exec] javah -d include/jni -force -cp target/classes \
     [exec] 	com.yahoo.ml.jcaffe.CaffeNet
     [exec] javah -d include/jni -force -cp target/classes \
     [exec] 	com.yahoo.ml.jcaffe.FloatDataTransformer
     [exec] javah -d include/jni -force -cp target/classes \
     [exec] 	com.yahoo.ml.jcaffe.FloatArray
     [exec] javah -d include/jni -force -cp target/classes \
     [exec] 	com.yahoo.ml.jcaffe.MatVector
     [exec] CXX src/main/cpp/util/parallel_cpu.cpp
     [exec] g++ src/main/cpp/util/parallel_cpu.cpp -MMD -MP -pthread -fPIC -DDEBUG -g -O0 -DUSE_OPENCV -DCPU_ONLY -I/usr/include/python2.7 -I/usr/lib/python2.7/dist-packages/numpy/core/include -I/usr/local/include -I/usr/lib/jvm/java-8-oracle/include -I.build_debug/src -I./include -I../caffe-public/distribute/include -I../caffe-public/src -I/usr/lib/jvm/java-8-oracle/include/linux -Wall -Wno-sign-compare -c -o .build_debug/src/main/cpp/util/parallel_cpu.o 2> .build_debug/src/main/cpp/util/parallel_cpu.o.warnings.txt \
     [exec] 		|| (cat .build_debug/src/main/cpp/util/parallel_cpu.o.warnings.txt; exit 1)
     [exec] CXX src/main/cpp/util/socket.cpp
     [exec] g++ src/main/cpp/util/socket.cpp -MMD -MP -pthread -fPIC -DDEBUG -g -O0 -DUSE_OPENCV -DCPU_ONLY -I/usr/include/python2.7 -I/usr/lib/python2.7/dist-packages/numpy/core/include -I/usr/local/include -I/usr/lib/jvm/java-8-oracle/include -I.build_debug/src -I./include -I../caffe-public/distribute/include -I../caffe-public/src -I/usr/lib/jvm/java-8-oracle/include/linux -Wall -Wno-sign-compare -c -o .build_debug/src/main/cpp/util/socket.o 2> .build_debug/src/main/cpp/util/socket.o.warnings.txt \
     [exec] 		|| (cat .build_debug/src/main/cpp/util/socket.o.warnings.txt; exit 1)
     [exec] CXX src/main/cpp/util/InputAdapter.cpp
     [exec] g++ src/main/cpp/util/InputAdapter.cpp -MMD -MP -pthread -fPIC -DDEBUG -g -O0 -DUSE_OPENCV -DCPU_ONLY -I/usr/include/python2.7 -I/usr/lib/python2.7/dist-packages/numpy/core/include -I/usr/local/include -I/usr/lib/jvm/java-8-oracle/include -I.build_debug/src -I./include -I../caffe-public/distribute/include -I../caffe-public/src -I/usr/lib/jvm/java-8-oracle/include/linux -Wall -Wno-sign-compare -c -o .build_debug/src/main/cpp/util/InputAdapter.o 2> .build_debug/src/main/cpp/util/InputAdapter.o.warnings.txt \
     [exec] 		|| (cat .build_debug/src/main/cpp/util/InputAdapter.o.warnings.txt; exit 1)
     [exec] CXX src/main/cpp/util/CoSInputAdapter.cpp
     [exec] g++ src/main/cpp/util/CoSInputAdapter.cpp -MMD -MP -pthread -fPIC -DDEBUG -g -O0 -DUSE_OPENCV -DCPU_ONLY -I/usr/include/python2.7 -I/usr/lib/python2.7/dist-packages/numpy/core/include -I/usr/local/include -I/usr/lib/jvm/java-8-oracle/include -I.build_debug/src -I./include -I../caffe-public/distribute/include -I../caffe-public/src -I/usr/lib/jvm/java-8-oracle/include/linux -Wall -Wno-sign-compare -c -o .build_debug/src/main/cpp/util/CoSInputAdapter.o 2> .build_debug/src/main/cpp/util/CoSInputAdapter.o.warnings.txt \
     [exec] 		|| (cat .build_debug/src/main/cpp/util/CoSInputAdapter.o.warnings.txt; exit 1)
     [exec] CXX src/main/cpp/util/rdma.cpp
     [exec] g++ src/main/cpp/util/rdma.cpp -MMD -MP -pthread -fPIC -DDEBUG -g -O0 -DUSE_OPENCV -DCPU_ONLY -I/usr/include/python2.7 -I/usr/lib/python2.7/dist-packages/numpy/core/include -I/usr/local/include -I/usr/lib/jvm/java-8-oracle/include -I.build_debug/src -I./include -I../caffe-public/distribute/include -I../caffe-public/src -I/usr/lib/jvm/java-8-oracle/include/linux -Wall -Wno-sign-compare -c -o .build_debug/src/main/cpp/util/rdma.o 2> .build_debug/src/main/cpp/util/rdma.o.warnings.txt \
     [exec] 		|| (cat .build_debug/src/main/cpp/util/rdma.o.warnings.txt; exit 1)
     [exec] CXX src/main/cpp/util/mini_cluster.cpp
     [exec] g++ src/main/cpp/util/mini_cluster.cpp -MMD -MP -pthread -fPIC -DDEBUG -g -O0 -DUSE_OPENCV -DCPU_ONLY -I/usr/include/python2.7 -I/usr/lib/python2.7/dist-packages/numpy/core/include -I/usr/local/include -I/usr/lib/jvm/java-8-oracle/include -I.build_debug/src -I./include -I../caffe-public/distribute/include -I../caffe-public/src -I/usr/lib/jvm/java-8-oracle/include/linux -Wall -Wno-sign-compare -c -o .build_debug/src/main/cpp/util/mini_cluster.o 2> .build_debug/src/main/cpp/util/mini_cluster.o.warnings.txt \
     [exec] 		|| (cat .build_debug/src/main/cpp/util/mini_cluster.o.warnings.txt; exit 1)
     [exec] CXX src/main/cpp/util/socket_sync_cpu.cpp
     [exec] g++ src/main/cpp/util/socket_sync_cpu.cpp -MMD -MP -pthread -fPIC -DDEBUG -g -O0 -DUSE_OPENCV -DCPU_ONLY -I/usr/include/python2.7 -I/usr/lib/python2.7/dist-packages/numpy/core/include -I/usr/local/include -I/usr/lib/jvm/java-8-oracle/include -I.build_debug/src -I./include -I../caffe-public/distribute/include -I../caffe-public/src -I/usr/lib/jvm/java-8-oracle/include/linux -Wall -Wno-sign-compare -c -o .build_debug/src/main/cpp/util/socket_sync_cpu.o 2> .build_debug/src/main/cpp/util/socket_sync_cpu.o.warnings.txt \
     [exec] 		|| (cat .build_debug/src/main/cpp/util/socket_sync_cpu.o.warnings.txt; exit 1)
     [exec] CXX src/main/cpp/util/blocking_queue_extra.cpp
     [exec] g++ src/main/cpp/util/blocking_queue_extra.cpp -MMD -MP -pthread -fPIC -DDEBUG -g -O0 -DUSE_OPENCV -DCPU_ONLY -I/usr/include/python2.7 -I/usr/lib/python2.7/dist-packages/numpy/core/include -I/usr/local/include -I/usr/lib/jvm/java-8-oracle/include -I.build_debug/src -I./include -I../caffe-public/distribute/include -I../caffe-public/src -I/usr/lib/jvm/java-8-oracle/include/linux -Wall -Wno-sign-compare -c -o .build_debug/src/main/cpp/util/blocking_queue_extra.o 2> .build_debug/src/main/cpp/util/blocking_queue_extra.o.warnings.txt \
     [exec] 		|| (cat .build_debug/src/main/cpp/util/blocking_queue_extra.o.warnings.txt; exit 1)
     [exec] CXX src/main/cpp/util/rdma_sync.cpp
     [exec] g++ src/main/cpp/util/rdma_sync.cpp -MMD -MP -pthread -fPIC -DDEBUG -g -O0 -DUSE_OPENCV -DCPU_ONLY -I/usr/include/python2.7 -I/usr/lib/python2.7/dist-packages/numpy/core/include -I/usr/local/include -I/usr/lib/jvm/java-8-oracle/include -I.build_debug/src -I./include -I../caffe-public/distribute/include -I../caffe-public/src -I/usr/lib/jvm/java-8-oracle/include/linux -Wall -Wno-sign-compare -c -o .build_debug/src/main/cpp/util/rdma_sync.o 2> .build_debug/src/main/cpp/util/rdma_sync.o.warnings.txt \
     [exec] 		|| (cat .build_debug/src/main/cpp/util/rdma_sync.o.warnings.txt; exit 1)
     [exec] CXX src/main/cpp/util/MemoryInputAdapter.cpp
     [exec] g++ src/main/cpp/util/MemoryInputAdapter.cpp -MMD -MP -pthread -fPIC -DDEBUG -g -O0 -DUSE_OPENCV -DCPU_ONLY -I/usr/include/python2.7 -I/usr/lib/python2.7/dist-packages/numpy/core/include -I/usr/local/include -I/usr/lib/jvm/java-8-oracle/include -I.build_debug/src -I./include -I../caffe-public/distribute/include -I../caffe-public/src -I/usr/lib/jvm/java-8-oracle/include/linux -Wall -Wno-sign-compare -c -o .build_debug/src/main/cpp/util/MemoryInputAdapter.o 2> .build_debug/src/main/cpp/util/MemoryInputAdapter.o.warnings.txt \
     [exec] 		|| (cat .build_debug/src/main/cpp/util/MemoryInputAdapter.o.warnings.txt; exit 1)
     [exec] CXX src/main/cpp/util/socket_sync.cpp
     [exec] g++ src/main/cpp/util/socket_sync.cpp -MMD -MP -pthread -fPIC -DDEBUG -g -O0 -DUSE_OPENCV -DCPU_ONLY -I/usr/include/python2.7 -I/usr/lib/python2.7/dist-packages/numpy/core/include -I/usr/local/include -I/usr/lib/jvm/java-8-oracle/include -I.build_debug/src -I./include -I../caffe-public/distribute/include -I../caffe-public/src -I/usr/lib/jvm/java-8-oracle/include/linux -Wall -Wno-sign-compare -c -o .build_debug/src/main/cpp/util/socket_sync.o 2> .build_debug/src/main/cpp/util/socket_sync.o.warnings.txt \
     [exec] 		|| (cat .build_debug/src/main/cpp/util/socket_sync.o.warnings.txt; exit 1)
     [exec] CXX src/main/cpp/CaffeNet.cpp
     [exec] g++ src/main/cpp/CaffeNet.cpp -MMD -MP -pthread -fPIC -DDEBUG -g -O0 -DUSE_OPENCV -DCPU_ONLY -I/usr/include/python2.7 -I/usr/lib/python2.7/dist-packages/numpy/core/include -I/usr/local/include -I/usr/lib/jvm/java-8-oracle/include -I.build_debug/src -I./include -I../caffe-public/distribute/include -I../caffe-public/src -I/usr/lib/jvm/java-8-oracle/include/linux -Wall -Wno-sign-compare -c -o .build_debug/src/main/cpp/CaffeNet.o 2> .build_debug/src/main/cpp/CaffeNet.o.warnings.txt \
     [exec] 		|| (cat .build_debug/src/main/cpp/CaffeNet.o.warnings.txt; exit 1)
     [exec] CXX src/main/cpp/jni/JniMat.cpp
     [exec] g++ src/main/cpp/jni/JniMat.cpp -MMD -MP -pthread -fPIC -DDEBUG -g -O0 -DUSE_OPENCV -DCPU_ONLY -I/usr/include/python2.7 -I/usr/lib/python2.7/dist-packages/numpy/core/include -I/usr/local/include -I/usr/lib/jvm/java-8-oracle/include -I.build_debug/src -I./include -I../caffe-public/distribute/include -I../caffe-public/src -I/usr/lib/jvm/java-8-oracle/include/linux -Wall -Wno-sign-compare -c -o .build_debug/src/main/cpp/jni/JniMat.o 2> .build_debug/src/main/cpp/jni/JniMat.o.warnings.txt \
     [exec] 		|| (cat .build_debug/src/main/cpp/jni/JniMat.o.warnings.txt; exit 1)
     [exec] CXX src/main/cpp/jni/JniFloatBlob.cpp
     [exec] g++ src/main/cpp/jni/JniFloatBlob.cpp -MMD -MP -pthread -fPIC -DDEBUG -g -O0 -DUSE_OPENCV -DCPU_ONLY -I/usr/include/python2.7 -I/usr/lib/python2.7/dist-packages/numpy/core/include -I/usr/local/include -I/usr/lib/jvm/java-8-oracle/include -I.build_debug/src -I./include -I../caffe-public/distribute/include -I../caffe-public/src -I/usr/lib/jvm/java-8-oracle/include/linux -Wall -Wno-sign-compare -c -o .build_debug/src/main/cpp/jni/JniFloatBlob.o 2> .build_debug/src/main/cpp/jni/JniFloatBlob.o.warnings.txt \
     [exec] 		|| (cat .build_debug/src/main/cpp/jni/JniFloatBlob.o.warnings.txt; exit 1)
     [exec] CXX src/main/cpp/jni/JniCaffeNet.cpp
     [exec] g++ src/main/cpp/jni/JniCaffeNet.cpp -MMD -MP -pthread -fPIC -DDEBUG -g -O0 -DUSE_OPENCV -DCPU_ONLY -I/usr/include/python2.7 -I/usr/lib/python2.7/dist-packages/numpy/core/include -I/usr/local/include -I/usr/lib/jvm/java-8-oracle/include -I.build_debug/src -I./include -I../caffe-public/distribute/include -I../caffe-public/src -I/usr/lib/jvm/java-8-oracle/include/linux -Wall -Wno-sign-compare -c -o .build_debug/src/main/cpp/jni/JniCaffeNet.o 2> .build_debug/src/main/cpp/jni/JniCaffeNet.o.warnings.txt \
     [exec] 		|| (cat .build_debug/src/main/cpp/jni/JniCaffeNet.o.warnings.txt; exit 1)
     [exec] CXX src/main/cpp/jni/JniFloatDataTransformer.cpp
     [exec] g++ src/main/cpp/jni/JniFloatDataTransformer.cpp -MMD -MP -pthread -fPIC -DDEBUG -g -O0 -DUSE_OPENCV -DCPU_ONLY -I/usr/include/python2.7 -I/usr/lib/python2.7/dist-packages/numpy/core/include -I/usr/local/include -I/usr/lib/jvm/java-8-oracle/include -I.build_debug/src -I./include -I../caffe-public/distribute/include -I../caffe-public/src -I/usr/lib/jvm/java-8-oracle/include/linux -Wall -Wno-sign-compare -c -o .build_debug/src/main/cpp/jni/JniFloatDataTransformer.o 2> .build_debug/src/main/cpp/jni/JniFloatDataTransformer.o.warnings.txt \
     [exec] 		|| (cat .build_debug/src/main/cpp/jni/JniFloatDataTransformer.o.warnings.txt; exit 1)
     [exec] CXX src/main/cpp/jni/JniFloatArray.cpp
     [exec] g++ src/main/cpp/jni/JniFloatArray.cpp -MMD -MP -pthread -fPIC -DDEBUG -g -O0 -DUSE_OPENCV -DCPU_ONLY -I/usr/include/python2.7 -I/usr/lib/python2.7/dist-packages/numpy/core/include -I/usr/local/include -I/usr/lib/jvm/java-8-oracle/include -I.build_debug/src -I./include -I../caffe-public/distribute/include -I../caffe-public/src -I/usr/lib/jvm/java-8-oracle/include/linux -Wall -Wno-sign-compare -c -o .build_debug/src/main/cpp/jni/JniFloatArray.o 2> .build_debug/src/main/cpp/jni/JniFloatArray.o.warnings.txt \
     [exec] 		|| (cat .build_debug/src/main/cpp/jni/JniFloatArray.o.warnings.txt; exit 1)
     [exec] CXX src/main/cpp/jni/JniMatVector.cpp
     [exec] g++ src/main/cpp/jni/JniMatVector.cpp -MMD -MP -pthread -fPIC -DDEBUG -g -O0 -DUSE_OPENCV -DCPU_ONLY -I/usr/include/python2.7 -I/usr/lib/python2.7/dist-packages/numpy/core/include -I/usr/local/include -I/usr/lib/jvm/java-8-oracle/include -I.build_debug/src -I./include -I../caffe-public/distribute/include -I../caffe-public/src -I/usr/lib/jvm/java-8-oracle/include/linux -Wall -Wno-sign-compare -c -o .build_debug/src/main/cpp/jni/JniMatVector.o 2> .build_debug/src/main/cpp/jni/JniMatVector.o.warnings.txt \
     [exec] 		|| (cat .build_debug/src/main/cpp/jni/JniMatVector.o.warnings.txt; exit 1)
     [exec] CXX src/main/cpp/cos_data_layer.cpp
     [exec] g++ src/main/cpp/cos_data_layer.cpp -MMD -MP -pthread -fPIC -DDEBUG -g -O0 -DUSE_OPENCV -DCPU_ONLY -I/usr/include/python2.7 -I/usr/lib/python2.7/dist-packages/numpy/core/include -I/usr/local/include -I/usr/lib/jvm/java-8-oracle/include -I.build_debug/src -I./include -I../caffe-public/distribute/include -I../caffe-public/src -I/usr/lib/jvm/java-8-oracle/include/linux -Wall -Wno-sign-compare -c -o .build_debug/src/main/cpp/cos_data_layer.o 2> .build_debug/src/main/cpp/cos_data_layer.o.warnings.txt \
     [exec] 		|| (cat .build_debug/src/main/cpp/cos_data_layer.o.warnings.txt; exit 1)
     [exec] CXX src/main/cpp/common.cpp
     [exec] g++ src/main/cpp/common.cpp -MMD -MP -pthread -fPIC -DDEBUG -g -O0 -DUSE_OPENCV -DCPU_ONLY -I/usr/include/python2.7 -I/usr/lib/python2.7/dist-packages/numpy/core/include -I/usr/local/include -I/usr/lib/jvm/java-8-oracle/include -I.build_debug/src -I./include -I../caffe-public/distribute/include -I../caffe-public/src -I/usr/lib/jvm/java-8-oracle/include/linux -Wall -Wno-sign-compare -c -o .build_debug/src/main/cpp/common.o 2> .build_debug/src/main/cpp/common.o.warnings.txt \
     [exec] 		|| (cat .build_debug/src/main/cpp/common.o.warnings.txt; exit 1)
     [exec] AR -o .build_debug/lib/libcaffedistri.a
     [exec] ar rcs .build_debug/lib/libcaffedistri.a  .build_debug/src/main/cpp/util/parallel_cpu.o .build_debug/src/main/cpp/util/socket.o .build_debug/src/main/cpp/util/InputAdapter.o .build_debug/src/main/cpp/util/CoSInputAdapter.o .build_debug/src/main/cpp/util/rdma.o .build_debug/src/main/cpp/util/mini_cluster.o .build_debug/src/main/cpp/util/socket_sync_cpu.o .build_debug/src/main/cpp/util/blocking_queue_extra.o .build_debug/src/main/cpp/util/rdma_sync.o .build_debug/src/main/cpp/util/MemoryInputAdapter.o .build_debug/src/main/cpp/util/socket_sync.o .build_debug/src/main/cpp/CaffeNet.o .build_debug/src/main/cpp/jni/JniMat.o .build_debug/src/main/cpp/jni/JniFloatBlob.o .build_debug/src/main/cpp/jni/JniCaffeNet.o .build_debug/src/main/cpp/jni/JniFloatDataTransformer.o .build_debug/src/main/cpp/jni/JniFloatArray.o .build_debug/src/main/cpp/jni/JniMatVector.o .build_debug/src/main/cpp/cos_data_layer.o .build_debug/src/main/cpp/common.o
     [exec] LD -o .build_debug/lib/libcaffedistri.so .build_debug/src/main/cpp/util/parallel_cpu.o .build_debug/src/main/cpp/util/socket.o .build_debug/src/main/cpp/util/InputAdapter.o .build_debug/src/main/cpp/util/CoSInputAdapter.o .build_debug/src/main/cpp/util/rdma.o .build_debug/src/main/cpp/util/mini_cluster.o .build_debug/src/main/cpp/util/socket_sync_cpu.o .build_debug/src/main/cpp/util/blocking_queue_extra.o .build_debug/src/main/cpp/util/rdma_sync.o .build_debug/src/main/cpp/util/MemoryInputAdapter.o .build_debug/src/main/cpp/util/socket_sync.o .build_debug/src/main/cpp/CaffeNet.o .build_debug/src/main/cpp/jni/JniMat.o .build_debug/src/main/cpp/jni/JniFloatBlob.o .build_debug/src/main/cpp/jni/JniCaffeNet.o .build_debug/src/main/cpp/jni/JniFloatDataTransformer.o .build_debug/src/main/cpp/jni/JniFloatArray.o .build_debug/src/main/cpp/jni/JniMatVector.o .build_debug/src/main/cpp/cos_data_layer.o .build_debug/src/main/cpp/common.o
     [exec] g++ -shared -o .build_debug/lib/libcaffedistri.so  .build_debug/src/main/cpp/util/parallel_cpu.o .build_debug/src/main/cpp/util/socket.o .build_debug/src/main/cpp/util/InputAdapter.o .build_debug/src/main/cpp/util/CoSInputAdapter.o .build_debug/src/main/cpp/util/rdma.o .build_debug/src/main/cpp/util/mini_cluster.o .build_debug/src/main/cpp/util/socket_sync_cpu.o .build_debug/src/main/cpp/util/blocking_queue_extra.o .build_debug/src/main/cpp/util/rdma_sync.o .build_debug/src/main/cpp/util/MemoryInputAdapter.o .build_debug/src/main/cpp/util/socket_sync.o .build_debug/src/main/cpp/CaffeNet.o .build_debug/src/main/cpp/jni/JniMat.o .build_debug/src/main/cpp/jni/JniFloatBlob.o .build_debug/src/main/cpp/jni/JniCaffeNet.o .build_debug/src/main/cpp/jni/JniFloatDataTransformer.o .build_debug/src/main/cpp/jni/JniFloatArray.o .build_debug/src/main/cpp/jni/JniMatVector.o .build_debug/src/main/cpp/cos_data_layer.o .build_debug/src/main/cpp/common.o -pthread -fPIC -DDEBUG -g -O0 -DUSE_OPENCV -DCPU_ONLY -I/usr/include/python2.7 -I/usr/lib/python2.7/dist-packages/numpy/core/include -I/usr/local/include -I/usr/lib/jvm/java-8-oracle/include -I.build_debug/src -I./include -I../caffe-public/distribute/include -I../caffe-public/src -I/usr/lib/jvm/java-8-oracle/include/linux -Wall -Wno-sign-compare -L/usr/lib/python3.4 -L/usr/local/lib -L/usr/lib -L/usr/lib/x86_64-linux-gnu -L/usr/include/google -L../caffe-public/distribute/lib -L.build_debug/lib  -lglog -lgflags -lprotobuf -lboost_system -lm -lcaffe -lhdf5_hl -lhdf5 -lopencv_core -lopencv_highgui -lopencv_imgproc -lboost_thread -lstdc++ -lcblas -latlas 
     [exec] CXX src/main/cpp/tools/caffe_mini_cluster.cpp
     [exec] g++ src/main/cpp/tools/caffe_mini_cluster.cpp -MMD -MP -pthread -fPIC -DDEBUG -g -O0 -DUSE_OPENCV -DCPU_ONLY -I/usr/include/python2.7 -I/usr/lib/python2.7/dist-packages/numpy/core/include -I/usr/local/include -I/usr/lib/jvm/java-8-oracle/include -I.build_debug/src -I./include -I../caffe-public/distribute/include -I../caffe-public/src -I/usr/lib/jvm/java-8-oracle/include/linux -Wall -Wno-sign-compare -c -o .build_debug/src/main/cpp/tools/caffe_mini_cluster.o 2> .build_debug/src/main/cpp/tools/caffe_mini_cluster.o.warnings.txt \
     [exec] 		|| (cat .build_debug/src/main/cpp/tools/caffe_mini_cluster.o.warnings.txt; exit 1)
     [exec] CXX/LD -o .build_debug/src/main/cpp/tools/caffe_mini_cluster.bin
     [exec] g++ .build_debug/src/main/cpp/tools/caffe_mini_cluster.o -o .build_debug/src/main/cpp/tools/caffe_mini_cluster.bin -pthread -fPIC -DDEBUG -g -O0 -DUSE_OPENCV -DCPU_ONLY -I/usr/include/python2.7 -I/usr/lib/python2.7/dist-packages/numpy/core/include -I/usr/local/include -I/usr/lib/jvm/java-8-oracle/include -I.build_debug/src -I./include -I../caffe-public/distribute/include -I../caffe-public/src -I/usr/lib/jvm/java-8-oracle/include/linux -Wall -Wno-sign-compare -lcaffedistri -L/usr/lib/python3.4 -L/usr/local/lib -L/usr/lib -L/usr/lib/x86_64-linux-gnu -L/usr/include/google -L../caffe-public/distribute/lib -L.build_debug/lib  -lglog -lgflags -lprotobuf -lboost_system -lm -lcaffe -lhdf5_hl -lhdf5 -lopencv_core -lopencv_highgui -lopencv_imgproc -lboost_thread -lstdc++ -lcblas -latlas \
     [exec] 		-Wl,-rpath,\$ORIGIN/../lib
     [exec] # add include
     [exec] cp -r include distribute/
     [exec] # add libraries
     [exec] cp .build_debug/lib/libcaffedistri.a distribute/lib
     [exec] cp .build_debug/lib/libcaffedistri.so distribute/lib
     [exec] make[1]: Leaving directory `/root/CaffeOnSpark/caffe-distri'
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-resources-plugin:2.3:testResources (default-testResources) @ caffe-distri ---
[WARNING] Using platform encoding (ANSI_X3.4-1968 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.0:testCompile (default-testCompile) @ caffe-distri ---
[INFO] Changes detected - recompiling the module!
[WARNING] File encoding has not been set, using platform encoding ANSI_X3.4-1968, i.e. build is platform dependent!
[INFO] Compiling 8 source files to /root/CaffeOnSpark/caffe-distri/target/test-classes
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ caffe-distri ---
(Downloading from repo.maven)
[INFO] META-INF/ already added, skipping
[INFO] META-INF/MANIFEST.MF already added, skipping
[INFO] org/ already added, skipping
[INFO] org/slf4j/ already added, skipping
[INFO] META-INF/maven/ already added, skipping
[INFO] META-INF/maven/org.slf4j/ already added, skipping
[INFO] META-INF/MANIFEST.MF already added, skipping
[INFO] META-INF/ already added, skipping
[INFO] META-INF/maven/ already added, skipping
[INFO] org/ already added, skipping
[INFO] Building jar: /root/CaffeOnSpark/caffe-distri/target/caffe-distri-0.1-SNAPSHOT-jar-with-dependencies.jar
[INFO] META-INF/ already added, skipping
[INFO] META-INF/MANIFEST.MF already added, skipping
[INFO] org/ already added, skipping
[INFO] org/slf4j/ already added, skipping
[INFO] META-INF/maven/ already added, skipping
[INFO] META-INF/maven/org.slf4j/ already added, skipping
[INFO] META-INF/MANIFEST.MF already added, skipping
[INFO] META-INF/ already added, skipping
[INFO] META-INF/maven/ already added, skipping
[INFO] org/ already added, skipping
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building caffe-grid 0.1-SNAPSHOT
[INFO] ------------------------------------------------------------------------
(Downloading from repo.maven)
[INFO] 
[INFO] --- maven-compiler-plugin:3.0:compile (compile-generator) @ caffe-grid ---
[INFO] Changes detected - recompiling the module!
[WARNING] File encoding has not been set, using platform encoding ANSI_X3.4-1968, i.e. build is platform dependent!
[INFO] Compiling 1 source file to /root/CaffeOnSpark/caffe-grid/target/classes
[INFO] 
[INFO] --- maven-resources-plugin:2.3:resources (default-resources) @ caffe-grid ---
[WARNING] Using platform encoding (ANSI_X3.4-1968 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] Copying 1 resource
[INFO] 
[INFO] --- maven-scala-plugin:2.14.2:add-source (scala-compile-first) @ caffe-grid ---
(Downloading from repo.maven)
[INFO] Add Source directory: /root/CaffeOnSpark/caffe-grid/src/main/scala
[INFO] Add Test Source directory: /root/CaffeOnSpark/caffe-grid/src/test/scala
[INFO] 
[INFO] --- maven-scala-plugin:2.14.2:compile (scala-compile-first) @ caffe-grid ---
[INFO] Checking for multiple versions of scala
[WARNING]  Expected all dependencies to require Scala version: 2.10.4
[WARNING]  com.twitter:chill_2.10:0.5.0 requires scala version: 2.10.4
[WARNING]  com.typesafe.akka:akka-remote_2.10:2.3.11 requires scala version: 2.10.4
[WARNING]  com.typesafe.akka:akka-actor_2.10:2.3.11 requires scala version: 2.10.4
[WARNING]  com.typesafe.akka:akka-slf4j_2.10:2.3.11 requires scala version: 2.10.4
[WARNING]  org.apache.spark:spark-core_2.10:1.6.0 requires scala version: 2.10.5
[WARNING] Multiple versions of scala libraries detected!
[INFO] includes = [**/*.java,**/*.scala,]
[INFO] excludes = []
[INFO] /root/CaffeOnSpark/caffe-grid/src/main/java:-1: info: compiling
[INFO] /root/CaffeOnSpark/caffe-grid/src/main/scala:-1: info: compiling
[INFO] Compiling 22 source files to /root/CaffeOnSpark/caffe-grid/target/classes at 1481012525365
(Downloading from repo.maven)
[INFO] Compiling 1 source file to /root/CaffeOnSpark/caffe-grid/target/classes
[INFO] 
[INFO] --- maven-resources-plugin:2.3:testResources (default-testResources) @ caffe-grid ---
[WARNING] Using platform encoding (ANSI_X3.4-1968 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] Copying 7 resources
[INFO] 
[INFO] --- maven-scala-plugin:2.14.2:testCompile (scala-test-compile) @ caffe-grid ---
[INFO] Checking for multiple versions of scala
[WARNING]  Expected all dependencies to require Scala version: 2.10.4
[WARNING]  com.twitter:chill_2.10:0.5.0 requires scala version: 2.10.4
[WARNING]  com.typesafe.akka:akka-remote_2.10:2.3.11 requires scala version: 2.10.4
[WARNING]  com.typesafe.akka:akka-actor_2.10:2.3.11 requires scala version: 2.10.4
[WARNING]  com.typesafe.akka:akka-slf4j_2.10:2.3.11 requires scala version: 2.10.4
[WARNING]  org.apache.spark:spark-core_2.10:1.6.0 requires scala version: 2.10.5
[WARNING] Multiple versions of scala libraries detected!
[INFO] includes = [**/*.java,**/*.scala,]
[INFO] excludes = []
[INFO] /root/CaffeOnSpark/caffe-grid/src/test/scala:-1: info: compiling
[INFO] Compiling 4 source files to /root/CaffeOnSpark/caffe-grid/target/test-classes at 1481012536393
[INFO] prepare-compile in 0 s
[INFO] compile in 4 s
[INFO] 
[INFO] --- maven-compiler-plugin:3.0:testCompile (default-testCompile) @ caffe-grid ---
[INFO] Changes detected - recompiling the module!
[WARNING] File encoding has not been set, using platform encoding ANSI_X3.4-1968, i.e. build is platform dependent!
[INFO] 
[INFO] --- maven-surefire-plugin:2.10:test (default-test) @ caffe-grid ---
(Downloading from repo.maven)
[INFO] Tests are skipped.
[INFO] 
[INFO] --- scalatest-maven-plugin:1.0:test (test) @ caffe-grid ---
(Downloading from repo.maven)
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ caffe-grid ---
[INFO] Building jar: /root/CaffeOnSpark/caffe-grid/target/caffe-grid-0.1-SNAPSHOT.jar
[INFO] 
[INFO] --- maven-assembly-plugin:2.2-beta-5:single (make-assembly) @ caffe-grid ---
[INFO] META-INF/ already added, skipping
[INFO] META-INF/MANIFEST.MF already added, skipping
[INFO] META-INF/maven/ already added, skipping
[INFO] META-INF/ already added, skipping
[INFO] META-INF/MANIFEST.MF already added, skipping
[INFO] org/ already added, skipping
[INFO] org/slf4j/ already added, skipping
[INFO] META-INF/maven/ already added, skipping
[INFO] META-INF/maven/org.slf4j/ already added, skipping
[INFO] META-INF/MANIFEST.MF already added, skipping
[INFO] META-INF/ already added, skipping
[INFO] META-INF/maven/ already added, skipping
[INFO] org/ already added, skipping
[INFO] META-INF/MANIFEST.MF already added, skipping
[INFO] META-INF/ already added, skipping
[INFO] META-INF/maven/ already added, skipping
[INFO] META-INF/ already added, skipping
[INFO] META-INF/MANIFEST.MF already added, skipping
[INFO] org/ already added, skipping
[INFO] META-INF/maven/ already added, skipping
[INFO] META-INF/ already added, skipping
[INFO] META-INF/MANIFEST.MF already added, skipping
[INFO] META-INF/maven/ already added, skipping
[INFO] META-INF/maven/org.deephacks.lmdbjni/ already added, skipping
[INFO] META-INF/ already added, skipping
[INFO] META-INF/MANIFEST.MF already added, skipping
[INFO] META-INF/native/ already added, skipping
[INFO] META-INF/maven/ already added, skipping
[INFO] META-INF/maven/org.deephacks.lmdbjni/ already added, skipping
[INFO] Building jar: /root/CaffeOnSpark/caffe-grid/target/caffe-grid-0.1-SNAPSHOT-jar-with-dependencies.jar
[INFO] META-INF/ already added, skipping
[INFO] META-INF/MANIFEST.MF already added, skipping
[INFO] META-INF/maven/ already added, skipping
[INFO] META-INF/ already added, skipping
[INFO] META-INF/MANIFEST.MF already added, skipping
[INFO] org/ already added, skipping
[INFO] org/slf4j/ already added, skipping
[INFO] META-INF/maven/ already added, skipping
[INFO] META-INF/maven/org.slf4j/ already added, skipping
[INFO] META-INF/MANIFEST.MF already added, skipping
[INFO] META-INF/ already added, skipping
[INFO] META-INF/maven/ already added, skipping
[INFO] org/ already added, skipping
[INFO] META-INF/MANIFEST.MF already added, skipping
[INFO] META-INF/ already added, skipping
[INFO] META-INF/maven/ already added, skipping
[INFO] META-INF/ already added, skipping
[INFO] META-INF/MANIFEST.MF already added, skipping
[INFO] org/ already added, skipping
[INFO] META-INF/maven/ already added, skipping
[INFO] META-INF/ already added, skipping
[INFO] META-INF/MANIFEST.MF already added, skipping
[INFO] META-INF/maven/ already added, skipping
[INFO] META-INF/maven/org.deephacks.lmdbjni/ already added, skipping
[INFO] META-INF/ already added, skipping
[INFO] META-INF/MANIFEST.MF already added, skipping
[INFO] META-INF/native/ already added, skipping
[INFO] META-INF/maven/ already added, skipping
[INFO] META-INF/maven/org.deephacks.lmdbjni/ already added, skipping
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] caffe ............................................. SUCCESS [0.002s]
[INFO] caffe-distri ...................................... SUCCESS [53.051s]
[INFO] caffe-grid ........................................ SUCCESS [5:07.389s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 6:02.154s
[INFO] Finished at: Tue Dec 06 08:22:28 UTC 2016
[INFO] Final Memory: 64M/691M
[INFO] ------------------------------------------------------------------------
jar -xvf caffe-grid/target/caffe-grid-0.1-SNAPSHOT-jar-with-dependencies.jar META-INF/native/linux64/liblmdbjni.so
 inflated: META-INF/native/linux64/liblmdbjni.so
mv META-INF/native/linux64/liblmdbjni.so /root/CaffeOnSpark/caffe-distri/distribute/lib
/root/CaffeOnSpark/scripts/setup-mnist.sh

You can see here that the building step was successful

(Downloading mnist from yann.lecun.com)

Creating lmdb...
libdc1394 error: Failed to initialize libdc1394
I1206 08:22:56.832409  3334 db_lmdb.cpp:35] Opened lmdb examples/mnist/mnist_train_lmdb
I1206 08:22:56.832577  3334 convert_mnist_data.cpp:88] A total of 60000 items.
I1206 08:22:56.832584  3334 convert_mnist_data.cpp:89] Rows: 28 Cols: 28
I1206 08:22:56.850394  3334 db_lmdb.cpp:107] Doubling LMDB map size to 2MB ...
I1206 08:22:56.861675  3334 db_lmdb.cpp:107] Doubling LMDB map size to 4MB ...
I1206 08:22:56.885169  3334 db_lmdb.cpp:107] Doubling LMDB map size to 8MB ...
I1206 08:22:56.931920  3334 db_lmdb.cpp:107] Doubling LMDB map size to 16MB ...
I1206 08:22:57.035118  3334 db_lmdb.cpp:107] Doubling LMDB map size to 32MB ...
I1206 08:22:57.236804  3334 db_lmdb.cpp:107] Doubling LMDB map size to 64MB ...
I1206 08:22:57.595010  3334 convert_mnist_data.cpp:108] Processed 60000 files.
libdc1394 error: Failed to initialize libdc1394
I1206 08:22:57.647197  3335 db_lmdb.cpp:35] Opened lmdb examples/mnist/mnist_test_lmdb
I1206 08:22:57.647366  3335 convert_mnist_data.cpp:88] A total of 10000 items.
I1206 08:22:57.647383  3335 convert_mnist_data.cpp:89] Rows: 28 Cols: 28
I1206 08:22:57.659924  3335 db_lmdb.cpp:107] Doubling LMDB map size to 2MB ...
I1206 08:22:57.672019  3335 db_lmdb.cpp:107] Doubling LMDB map size to 4MB ...
I1206 08:22:57.698387  3335 db_lmdb.cpp:107] Doubling LMDB map size to 8MB ...
I1206 08:22:57.752008  3335 db_lmdb.cpp:107] Doubling LMDB map size to 16MB ...
I1206 08:22:57.781086  3335 convert_mnist_data.cpp:108] Processed 10000 files.
Done.
export LD_LIBRARY_PATH="/home/y/lib64:/home/y/lib64/mkl/intel64:/root/CaffeOnSpark/caffe-public/distribute/lib:/root/CaffeOnSpark/caffe-distri/distribute/lib:/usr/lib64:/lib64 "; mvn  -B package
[INFO] Scanning for projects...
[WARNING] 
[WARNING] Some problems were encountered while building the effective model for com.yahoo.ml:caffe-grid:jar:0.1-SNAPSHOT
[WARNING] The expression ${version} is deprecated. Please use ${project.version} instead.
[WARNING] 
[WARNING] It is highly recommended to fix these problems because they threaten the stability of your build.
[WARNING] 
[WARNING] For this reason, future Maven versions might no longer support building such malformed projects.
[WARNING] 
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO] 
[INFO] caffe
[INFO] caffe-distri
[INFO] caffe-grid
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building caffe 0.1-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building caffe-distri 0.1-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- maven-antrun-plugin:1.7:run (proto) @ caffe-distri ---
[INFO] Executing tasks

protoc:
     [exec] make[1]: Entering directory `/root/CaffeOnSpark/caffe-distri'
     [exec] make[1]: Nothing to be done for `proto'.
     [exec] make[1]: Leaving directory `/root/CaffeOnSpark/caffe-distri'
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-compiler-plugin:3.0:compile (compile-generator) @ caffe-distri ---
[INFO] Changes detected - recompiling the module!
[WARNING] File encoding has not been set, using platform encoding ANSI_X3.4-1968, i.e. build is platform dependent!
[INFO] Compiling 10 source files to /root/CaffeOnSpark/caffe-distri/target/classes
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[39561,19] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[39561,20] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[39571,19] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[39571,20] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[39581,19] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[39581,20] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[40371,19] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[40371,20] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[40384,19] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[40384,20] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[40396,19] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[40396,20] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[41716,21] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[41716,22] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[41729,21] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[41729,22] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[41741,21] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[41741,22] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[41753,21] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[41753,22] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[41769,21] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[41769,22] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[41784,21] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[41784,22] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[41800,21] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[41800,22] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java: /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java uses or overrides a deprecated API.
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java: Recompile with -Xlint:deprecation for details.
[INFO] 
[INFO] --- maven-resources-plugin:2.3:resources (default-resources) @ caffe-distri ---
[WARNING] Using platform encoding (ANSI_X3.4-1968 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory /root/CaffeOnSpark/caffe-distri/src/main/resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.0:compile (default-compile) @ caffe-distri ---
[INFO] Changes detected - recompiling the module!
[WARNING] File encoding has not been set, using platform encoding ANSI_X3.4-1968, i.e. build is platform dependent!
[INFO] Compiling 10 source files to /root/CaffeOnSpark/caffe-distri/target/classes
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[39561,19] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[39561,20] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[39571,19] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[39571,20] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[39581,19] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[39581,20] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[40371,19] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[40371,20] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[40384,19] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[40384,20] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[40396,19] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[40396,20] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[41716,21] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[41716,22] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[41729,21] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[41729,22] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[41741,21] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[41741,22] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[41753,21] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[41753,22] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[41769,21] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[41769,22] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[41784,21] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[41784,22] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[41800,21] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[41800,22] unmappable character for encoding ASCII
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java: /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java uses or overrides a deprecated API.
[WARNING] /root/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java: Recompile with -Xlint:deprecation for details.
[INFO] 
[INFO] --- maven-antrun-plugin:1.7:run (make) @ caffe-distri ---
[INFO] Executing tasks

make:
     [exec] make[1]: Entering directory `/root/CaffeOnSpark/caffe-distri'
     [exec] javah -d include/jni -force -cp target/classes \
     [exec] 	com.yahoo.ml.jcaffe.Mat
     [exec] javah -d include/jni -force -cp target/classes \
     [exec] 	com.yahoo.ml.jcaffe.FloatBlob
     [exec] javah -d include/jni -force -cp target/classes \
     [exec] 	com.yahoo.ml.jcaffe.CaffeNet
     [exec] javah -d include/jni -force -cp target/classes \
     [exec] 	com.yahoo.ml.jcaffe.FloatDataTransformer
     [exec] javah -d include/jni -force -cp target/classes \
     [exec] 	com.yahoo.ml.jcaffe.FloatArray
     [exec] javah -d include/jni -force -cp target/classes \
     [exec] 	com.yahoo.ml.jcaffe.MatVector
     [exec] CXX src/main/cpp/CaffeNet.cpp
     [exec] g++ src/main/cpp/CaffeNet.cpp -MMD -MP -pthread -fPIC -DDEBUG -g -O0 -DUSE_OPENCV -DCPU_ONLY -I/usr/include/python2.7 -I/usr/lib/python2.7/dist-packages/numpy/core/include -I/usr/local/include -I/usr/lib/jvm/java-8-oracle/include -I.build_debug/src -I./include -I../caffe-public/distribute/include -I../caffe-public/src -I/usr/lib/jvm/java-8-oracle/include/linux -Wall -Wno-sign-compare -c -o .build_debug/src/main/cpp/CaffeNet.o 2> .build_debug/src/main/cpp/CaffeNet.o.warnings.txt \
     [exec] 		|| (cat .build_debug/src/main/cpp/CaffeNet.o.warnings.txt; exit 1)
     [exec] CXX src/main/cpp/jni/JniMat.cpp
     [exec] g++ src/main/cpp/jni/JniMat.cpp -MMD -MP -pthread -fPIC -DDEBUG -g -O0 -DUSE_OPENCV -DCPU_ONLY -I/usr/include/python2.7 -I/usr/lib/python2.7/dist-packages/numpy/core/include -I/usr/local/include -I/usr/lib/jvm/java-8-oracle/include -I.build_debug/src -I./include -I../caffe-public/distribute/include -I../caffe-public/src -I/usr/lib/jvm/java-8-oracle/include/linux -Wall -Wno-sign-compare -c -o .build_debug/src/main/cpp/jni/JniMat.o 2> .build_debug/src/main/cpp/jni/JniMat.o.warnings.txt \
     [exec] 		|| (cat .build_debug/src/main/cpp/jni/JniMat.o.warnings.txt; exit 1)
     [exec] CXX src/main/cpp/jni/JniFloatBlob.cpp
     [exec] g++ src/main/cpp/jni/JniFloatBlob.cpp -MMD -MP -pthread -fPIC -DDEBUG -g -O0 -DUSE_OPENCV -DCPU_ONLY -I/usr/include/python2.7 -I/usr/lib/python2.7/dist-packages/numpy/core/include -I/usr/local/include -I/usr/lib/jvm/java-8-oracle/include -I.build_debug/src -I./include -I../caffe-public/distribute/include -I../caffe-public/src -I/usr/lib/jvm/java-8-oracle/include/linux -Wall -Wno-sign-compare -c -o .build_debug/src/main/cpp/jni/JniFloatBlob.o 2> .build_debug/src/main/cpp/jni/JniFloatBlob.o.warnings.txt \
     [exec] 		|| (cat .build_debug/src/main/cpp/jni/JniFloatBlob.o.warnings.txt; exit 1)
     [exec] CXX src/main/cpp/jni/JniCaffeNet.cpp
     [exec] g++ src/main/cpp/jni/JniCaffeNet.cpp -MMD -MP -pthread -fPIC -DDEBUG -g -O0 -DUSE_OPENCV -DCPU_ONLY -I/usr/include/python2.7 -I/usr/lib/python2.7/dist-packages/numpy/core/include -I/usr/local/include -I/usr/lib/jvm/java-8-oracle/include -I.build_debug/src -I./include -I../caffe-public/distribute/include -I../caffe-public/src -I/usr/lib/jvm/java-8-oracle/include/linux -Wall -Wno-sign-compare -c -o .build_debug/src/main/cpp/jni/JniCaffeNet.o 2> .build_debug/src/main/cpp/jni/JniCaffeNet.o.warnings.txt \
     [exec] 		|| (cat .build_debug/src/main/cpp/jni/JniCaffeNet.o.warnings.txt; exit 1)
     [exec] CXX src/main/cpp/jni/JniFloatDataTransformer.cpp
     [exec] g++ src/main/cpp/jni/JniFloatDataTransformer.cpp -MMD -MP -pthread -fPIC -DDEBUG -g -O0 -DUSE_OPENCV -DCPU_ONLY -I/usr/include/python2.7 -I/usr/lib/python2.7/dist-packages/numpy/core/include -I/usr/local/include -I/usr/lib/jvm/java-8-oracle/include -I.build_debug/src -I./include -I../caffe-public/distribute/include -I../caffe-public/src -I/usr/lib/jvm/java-8-oracle/include/linux -Wall -Wno-sign-compare -c -o .build_debug/src/main/cpp/jni/JniFloatDataTransformer.o 2> .build_debug/src/main/cpp/jni/JniFloatDataTransformer.o.warnings.txt \
     [exec] 		|| (cat .build_debug/src/main/cpp/jni/JniFloatDataTransformer.o.warnings.txt; exit 1)
     [exec] CXX src/main/cpp/jni/JniFloatArray.cpp
     [exec] g++ src/main/cpp/jni/JniFloatArray.cpp -MMD -MP -pthread -fPIC -DDEBUG -g -O0 -DUSE_OPENCV -DCPU_ONLY -I/usr/include/python2.7 -I/usr/lib/python2.7/dist-packages/numpy/core/include -I/usr/local/include -I/usr/lib/jvm/java-8-oracle/include -I.build_debug/src -I./include -I../caffe-public/distribute/include -I../caffe-public/src -I/usr/lib/jvm/java-8-oracle/include/linux -Wall -Wno-sign-compare -c -o .build_debug/src/main/cpp/jni/JniFloatArray.o 2> .build_debug/src/main/cpp/jni/JniFloatArray.o.warnings.txt \
     [exec] 		|| (cat .build_debug/src/main/cpp/jni/JniFloatArray.o.warnings.txt; exit 1)
     [exec] CXX src/main/cpp/jni/JniMatVector.cpp
     [exec] g++ src/main/cpp/jni/JniMatVector.cpp -MMD -MP -pthread -fPIC -DDEBUG -g -O0 -DUSE_OPENCV -DCPU_ONLY -I/usr/include/python2.7 -I/usr/lib/python2.7/dist-packages/numpy/core/include -I/usr/local/include -I/usr/lib/jvm/java-8-oracle/include -I.build_debug/src -I./include -I../caffe-public/distribute/include -I../caffe-public/src -I/usr/lib/jvm/java-8-oracle/include/linux -Wall -Wno-sign-compare -c -o .build_debug/src/main/cpp/jni/JniMatVector.o 2> .build_debug/src/main/cpp/jni/JniMatVector.o.warnings.txt \
     [exec] 		|| (cat .build_debug/src/main/cpp/jni/JniMatVector.o.warnings.txt; exit 1)
     [exec] AR -o .build_debug/lib/libcaffedistri.a
     [exec] ar rcs .build_debug/lib/libcaffedistri.a  .build_debug/src/main/cpp/util/parallel_cpu.o .build_debug/src/main/cpp/util/socket.o .build_debug/src/main/cpp/util/InputAdapter.o .build_debug/src/main/cpp/util/CoSInputAdapter.o .build_debug/src/main/cpp/util/rdma.o .build_debug/src/main/cpp/util/mini_cluster.o .build_debug/src/main/cpp/util/socket_sync_cpu.o .build_debug/src/main/cpp/util/blocking_queue_extra.o .build_debug/src/main/cpp/util/rdma_sync.o .build_debug/src/main/cpp/util/MemoryInputAdapter.o .build_debug/src/main/cpp/util/socket_sync.o .build_debug/src/main/cpp/CaffeNet.o .build_debug/src/main/cpp/jni/JniMat.o .build_debug/src/main/cpp/jni/JniFloatBlob.o .build_debug/src/main/cpp/jni/JniCaffeNet.o .build_debug/src/main/cpp/jni/JniFloatDataTransformer.o .build_debug/src/main/cpp/jni/JniFloatArray.o .build_debug/src/main/cpp/jni/JniMatVector.o .build_debug/src/main/cpp/cos_data_layer.o .build_debug/src/main/cpp/common.o
     [exec] LD -o .build_debug/lib/libcaffedistri.so .build_debug/src/main/cpp/util/parallel_cpu.o .build_debug/src/main/cpp/util/socket.o .build_debug/src/main/cpp/util/InputAdapter.o .build_debug/src/main/cpp/util/CoSInputAdapter.o .build_debug/src/main/cpp/util/rdma.o .build_debug/src/main/cpp/util/mini_cluster.o .build_debug/src/main/cpp/util/socket_sync_cpu.o .build_debug/src/main/cpp/util/blocking_queue_extra.o .build_debug/src/main/cpp/util/rdma_sync.o .build_debug/src/main/cpp/util/MemoryInputAdapter.o .build_debug/src/main/cpp/util/socket_sync.o .build_debug/src/main/cpp/CaffeNet.o .build_debug/src/main/cpp/jni/JniMat.o .build_debug/src/main/cpp/jni/JniFloatBlob.o .build_debug/src/main/cpp/jni/JniCaffeNet.o .build_debug/src/main/cpp/jni/JniFloatDataTransformer.o .build_debug/src/main/cpp/jni/JniFloatArray.o .build_debug/src/main/cpp/jni/JniMatVector.o .build_debug/src/main/cpp/cos_data_layer.o .build_debug/src/main/cpp/common.o
     [exec] g++ -shared -o .build_debug/lib/libcaffedistri.so  .build_debug/src/main/cpp/util/parallel_cpu.o .build_debug/src/main/cpp/util/socket.o .build_debug/src/main/cpp/util/InputAdapter.o .build_debug/src/main/cpp/util/CoSInputAdapter.o .build_debug/src/main/cpp/util/rdma.o .build_debug/src/main/cpp/util/mini_cluster.o .build_debug/src/main/cpp/util/socket_sync_cpu.o .build_debug/src/main/cpp/util/blocking_queue_extra.o .build_debug/src/main/cpp/util/rdma_sync.o .build_debug/src/main/cpp/util/MemoryInputAdapter.o .build_debug/src/main/cpp/util/socket_sync.o .build_debug/src/main/cpp/CaffeNet.o .build_debug/src/main/cpp/jni/JniMat.o .build_debug/src/main/cpp/jni/JniFloatBlob.o .build_debug/src/main/cpp/jni/JniCaffeNet.o .build_debug/src/main/cpp/jni/JniFloatDataTransformer.o .build_debug/src/main/cpp/jni/JniFloatArray.o .build_debug/src/main/cpp/jni/JniMatVector.o .build_debug/src/main/cpp/cos_data_layer.o .build_debug/src/main/cpp/common.o -pthread -fPIC -DDEBUG -g -O0 -DUSE_OPENCV -DCPU_ONLY -I/usr/include/python2.7 -I/usr/lib/python2.7/dist-packages/numpy/core/include -I/usr/local/include -I/usr/lib/jvm/java-8-oracle/include -I.build_debug/src -I./include -I../caffe-public/distribute/include -I../caffe-public/src -I/usr/lib/jvm/java-8-oracle/include/linux -Wall -Wno-sign-compare -L/usr/lib/python3.4 -L/usr/local/lib -L/usr/lib -L/usr/lib/x86_64-linux-gnu -L/usr/include/google -L../caffe-public/distribute/lib -L.build_debug/lib  -lglog -lgflags -lprotobuf -lboost_system -lm -lcaffe -lhdf5_hl -lhdf5 -lopencv_core -lopencv_highgui -lopencv_imgproc -lboost_thread -lstdc++ -lcblas -latlas 
     [exec] # add include
     [exec] cp -r include distribute/
     [exec] # add libraries
     [exec] cp .build_debug/lib/libcaffedistri.a distribute/lib
     [exec] cp .build_debug/lib/libcaffedistri.so distribute/lib
     [exec] make[1]: Leaving directory `/root/CaffeOnSpark/caffe-distri'
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-resources-plugin:2.3:testResources (default-testResources) @ caffe-distri ---
[WARNING] Using platform encoding (ANSI_X3.4-1968 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.0:testCompile (default-testCompile) @ caffe-distri ---
[INFO] Changes detected - recompiling the module!
[WARNING] File encoding has not been set, using platform encoding ANSI_X3.4-1968, i.e. build is platform dependent!
[INFO] Compiling 8 source files to /root/CaffeOnSpark/caffe-distri/target/test-classes
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ caffe-distri ---
[INFO] Surefire report directory: /root/CaffeOnSpark/caffe-distri/target/surefire-reports
(Downloading from repo.maven)

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running TestSuite
Configuring TestNG with: org.apache.maven.surefire.testng.conf.TestNG652Configurator@3830f1c0
Tests run: 66, Failures: 25, Errors: 0, Skipped: 40, Time elapsed: 0.473 sec <<< FAILURE!
setUp(com.yahoo.ml.jcaffe.CaffeNetTest)  Time elapsed: 0.148 sec  <<< FAILURE!
java.lang.UnsatisfiedLinkError: no caffedistri in java.library.path
	at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
	at java.lang.Runtime.loadLibrary0(Runtime.java:870)
	at java.lang.System.loadLibrary(System.java:1122)
	at com.yahoo.ml.jcaffe.BaseObject.<clinit>(BaseObject.java:10)
	at com.yahoo.ml.jcaffe.CaffeNetTest.setUp(CaffeNetTest.java:39)

floatarraygetnegative(com.yahoo.ml.jcaffe.FloatArrayTest)  Time elapsed: 0 sec  <<< FAILURE!
java.lang.NoClassDefFoundError: com.yahoo.ml.jcaffe.BaseObject
	at com.yahoo.ml.jcaffe.FloatArrayTest.floatarraygetnegative(FloatArrayTest.java:13)

floatarraysetinvalid(com.yahoo.ml.jcaffe.FloatArrayTest)  Time elapsed: 0 sec  <<< FAILURE!
java.lang.NoClassDefFoundError: com.yahoo.ml.jcaffe.BaseObject
	at com.yahoo.ml.jcaffe.FloatArrayTest.floatarraysetinvalid(FloatArrayTest.java:26)

copyfromnull(com.yahoo.ml.jcaffe.FloatBlobTest)  Time elapsed: 0 sec  <<< FAILURE!
java.lang.NoClassDefFoundError: com.yahoo.ml.jcaffe.BaseObject
	at com.yahoo.ml.jcaffe.FloatBlobTest.copyfromnull(FloatBlobTest.java:26)

floatblobnull(com.yahoo.ml.jcaffe.FloatBlobTest)  Time elapsed: 0 sec  <<< FAILURE!
java.lang.NoClassDefFoundError: com.yahoo.ml.jcaffe.BaseObject
	at com.yahoo.ml.jcaffe.FloatBlobTest.floatblobnull(FloatBlobTest.java:35)

reshapeinvalid(com.yahoo.ml.jcaffe.FloatBlobTest)  Time elapsed: 0 sec  <<< FAILURE!
java.lang.NoClassDefFoundError: com.yahoo.ml.jcaffe.BaseObject
	at com.yahoo.ml.jcaffe.FloatBlobTest.reshapeinvalid(FloatBlobTest.java:45)

setcpudatanull(com.yahoo.ml.jcaffe.FloatBlobTest)  Time elapsed: 0 sec  <<< FAILURE!
java.lang.NoClassDefFoundError: com.yahoo.ml.jcaffe.BaseObject
	at com.yahoo.ml.jcaffe.FloatBlobTest.setcpudatanull(FloatBlobTest.java:19)

testAllocate(com.yahoo.ml.jcaffe.FloatBlobTest)  Time elapsed: 0 sec  <<< FAILURE!
java.lang.NoClassDefFoundError: com.yahoo.ml.jcaffe.BaseObject
	at com.yahoo.ml.jcaffe.FloatBlobTest.testAllocate(FloatBlobTest.java:13)

testBasic(com.yahoo.ml.jcaffe.FloatBlobTest)  Time elapsed: 0 sec  <<< FAILURE!
java.lang.NoClassDefFoundError: com.yahoo.ml.jcaffe.BaseObject
	at com.yahoo.ml.jcaffe.FloatBlobTest.testBasic(FloatBlobTest.java:57)

testCopy(com.yahoo.ml.jcaffe.FloatBlobTest)  Time elapsed: 0.001 sec  <<< FAILURE!
java.lang.NoClassDefFoundError: com.yahoo.ml.jcaffe.BaseObject
	at com.yahoo.ml.jcaffe.FloatBlobTest.testCopy(FloatBlobTest.java:74)

basicDataTest(com.yahoo.ml.jcaffe.MatTest)  Time elapsed: 0 sec  <<< FAILURE!
java.lang.NoClassDefFoundError: com.yahoo.ml.jcaffe.BaseObject
	at com.yahoo.ml.jcaffe.MatTest.basicDataTest(MatTest.java:148)

basicTest(com.yahoo.ml.jcaffe.MatTest)  Time elapsed: 0.001 sec  <<< FAILURE!
java.lang.NoClassDefFoundError: com.yahoo.ml.jcaffe.BaseObject
	at com.yahoo.ml.jcaffe.MatTest.basicTest(MatTest.java:57)

getMatDecodeWithInvalidFlag(com.yahoo.ml.jcaffe.MatTest)  Time elapsed: 0.003 sec  <<< FAILURE!
java.lang.NoClassDefFoundError: com.yahoo.ml.jcaffe.BaseObject
	at com.yahoo.ml.jcaffe.MatTest.getMatDecodeWithInvalidFlag(MatTest.java:172)

matChannelsTest(com.yahoo.ml.jcaffe.MatTest)  Time elapsed: 0 sec  <<< FAILURE!
java.lang.NoClassDefFoundError: com.yahoo.ml.jcaffe.BaseObject
	at com.yahoo.ml.jcaffe.MatTest.matChannelsTest(MatTest.java:141)

matNull(com.yahoo.ml.jcaffe.MatTest)  Time elapsed: 0 sec  <<< FAILURE!
java.lang.NoClassDefFoundError: com.yahoo.ml.jcaffe.BaseObject
	at com.yahoo.ml.jcaffe.MatTest.matNull(MatTest.java:48)

matResizeDecodeTest(com.yahoo.ml.jcaffe.MatTest)  Time elapsed: 0.001 sec  <<< FAILURE!
java.lang.NoClassDefFoundError: com.yahoo.ml.jcaffe.BaseObject
	at com.yahoo.ml.jcaffe.MatTest.matResizeDecodeTest(MatTest.java:132)

matResizeInvalid(com.yahoo.ml.jcaffe.MatTest)  Time elapsed: 0.001 sec  <<< FAILURE!
java.lang.NoClassDefFoundError: com.yahoo.ml.jcaffe.BaseObject
	at com.yahoo.ml.jcaffe.MatTest.matResizeInvalid(MatTest.java:185)

getMatVecDatafromInvalidIndex(com.yahoo.ml.jcaffe.MatVectorTest)  Time elapsed: 0 sec  <<< FAILURE!
java.lang.NoClassDefFoundError: com.yahoo.ml.jcaffe.BaseObject
	at com.yahoo.ml.jcaffe.MatVectorTest.getMatVecDatafromInvalidIndex(MatVectorTest.java:103)

invalidchannel(com.yahoo.ml.jcaffe.MatVectorTest)  Time elapsed: 0 sec  <<< FAILURE!
java.lang.NoClassDefFoundError: com.yahoo.ml.jcaffe.BaseObject
	at com.yahoo.ml.jcaffe.MatVectorTest.invalidchannel(MatVectorTest.java:131)

invalidheight(com.yahoo.ml.jcaffe.MatVectorTest)  Time elapsed: 0.001 sec  <<< FAILURE!
java.lang.NoClassDefFoundError: com.yahoo.ml.jcaffe.BaseObject
	at com.yahoo.ml.jcaffe.MatVectorTest.invalidheight(MatVectorTest.java:113)

invalidwidth(com.yahoo.ml.jcaffe.MatVectorTest)  Time elapsed: 0.001 sec  <<< FAILURE!
java.lang.NoClassDefFoundError: com.yahoo.ml.jcaffe.BaseObject
	at com.yahoo.ml.jcaffe.MatVectorTest.invalidwidth(MatVectorTest.java:122)

matNullInMatVec(com.yahoo.ml.jcaffe.MatVectorTest)  Time elapsed: 0 sec  <<< FAILURE!
java.lang.NoClassDefFoundError: com.yahoo.ml.jcaffe.BaseObject
	at com.yahoo.ml.jcaffe.MatVectorTest.matNullInMatVec(MatVectorTest.java:57)

matOnWrongMatVecIndex(com.yahoo.ml.jcaffe.MatVectorTest)  Time elapsed: 0 sec  <<< FAILURE!
java.lang.NoClassDefFoundError: com.yahoo.ml.jcaffe.BaseObject
	at com.yahoo.ml.jcaffe.MatVectorTest.matOnWrongMatVecIndex(MatVectorTest.java:69)

matVecNegativeIndex(com.yahoo.ml.jcaffe.MatVectorTest)  Time elapsed: 0 sec  <<< FAILURE!
java.lang.NoClassDefFoundError: com.yahoo.ml.jcaffe.BaseObject
	at com.yahoo.ml.jcaffe.MatVectorTest.matVecNegativeIndex(MatVectorTest.java:48)

setUp(com.yahoo.ml.jcaffe.TransformTest)  Time elapsed: 0.164 sec  <<< FAILURE!
java.lang.NoClassDefFoundError: com.yahoo.ml.jcaffe.BaseObject
	at com.yahoo.ml.jcaffe.TransformTest.setUp(TransformTest.java:43)


Results :

Failed tests:   setUp(com.yahoo.ml.jcaffe.CaffeNetTest): no caffedistri in java.library.path
  floatarraygetnegative(com.yahoo.ml.jcaffe.FloatArrayTest): com.yahoo.ml.jcaffe.BaseObject
  floatarraysetinvalid(com.yahoo.ml.jcaffe.FloatArrayTest): com.yahoo.ml.jcaffe.BaseObject
  copyfromnull(com.yahoo.ml.jcaffe.FloatBlobTest): com.yahoo.ml.jcaffe.BaseObject
  floatblobnull(com.yahoo.ml.jcaffe.FloatBlobTest): com.yahoo.ml.jcaffe.BaseObject
  reshapeinvalid(com.yahoo.ml.jcaffe.FloatBlobTest): com.yahoo.ml.jcaffe.BaseObject
  setcpudatanull(com.yahoo.ml.jcaffe.FloatBlobTest): com.yahoo.ml.jcaffe.BaseObject
  testAllocate(com.yahoo.ml.jcaffe.FloatBlobTest): com.yahoo.ml.jcaffe.BaseObject
  testBasic(com.yahoo.ml.jcaffe.FloatBlobTest): com.yahoo.ml.jcaffe.BaseObject
  testCopy(com.yahoo.ml.jcaffe.FloatBlobTest): com.yahoo.ml.jcaffe.BaseObject
  basicDataTest(com.yahoo.ml.jcaffe.MatTest): com.yahoo.ml.jcaffe.BaseObject
  basicTest(com.yahoo.ml.jcaffe.MatTest): com.yahoo.ml.jcaffe.BaseObject
  getMatDecodeWithInvalidFlag(com.yahoo.ml.jcaffe.MatTest): com.yahoo.ml.jcaffe.BaseObject
  matChannelsTest(com.yahoo.ml.jcaffe.MatTest): com.yahoo.ml.jcaffe.BaseObject
  matNull(com.yahoo.ml.jcaffe.MatTest): com.yahoo.ml.jcaffe.BaseObject
  matResizeDecodeTest(com.yahoo.ml.jcaffe.MatTest): com.yahoo.ml.jcaffe.BaseObject
  matResizeInvalid(com.yahoo.ml.jcaffe.MatTest): com.yahoo.ml.jcaffe.BaseObject
  getMatVecDatafromInvalidIndex(com.yahoo.ml.jcaffe.MatVectorTest): com.yahoo.ml.jcaffe.BaseObject
  invalidchannel(com.yahoo.ml.jcaffe.MatVectorTest): com.yahoo.ml.jcaffe.BaseObject
  invalidheight(com.yahoo.ml.jcaffe.MatVectorTest): com.yahoo.ml.jcaffe.BaseObject
  invalidwidth(com.yahoo.ml.jcaffe.MatVectorTest): com.yahoo.ml.jcaffe.BaseObject
  matNullInMatVec(com.yahoo.ml.jcaffe.MatVectorTest): com.yahoo.ml.jcaffe.BaseObject
  matOnWrongMatVecIndex(com.yahoo.ml.jcaffe.MatVectorTest): com.yahoo.ml.jcaffe.BaseObject
  matVecNegativeIndex(com.yahoo.ml.jcaffe.MatVectorTest): com.yahoo.ml.jcaffe.BaseObject
  setUp(com.yahoo.ml.jcaffe.TransformTest): com.yahoo.ml.jcaffe.BaseObject

Tests run: 66, Failures: 25, Errors: 0, Skipped: 40

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] caffe ............................................. SUCCESS [0.001s]
[INFO] caffe-distri ...................................... FAILURE [22.769s]
[INFO] caffe-grid ........................................ SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 22.917s
[INFO] Finished at: Tue Dec 06 08:23:21 UTC 2016
[INFO] Final Memory: 27M/552M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.12.4:test (default-test) on project caffe-distri: There are test failures.
[ERROR] 
[ERROR] Please refer to /root/CaffeOnSpark/caffe-distri/target/surefire-reports for the individual test results.
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :caffe-distri
make: *** [build] Error 1
The command '/bin/sh -c cd ${CAFFE_ON_SPARK} &&   make build' returned a non-zero code: 2

The text was updated successfully, but these errors were encountered:

arundasan91 · 2016-12-06T16:06:57Z

Hi @baristahell ,

How did you set up Hadoop and Spark for the Docker Image ? Were you able to ssh into your localhost ?

baristahell · 2016-12-07T08:28:46Z

Hello @arundasan91 , i have a similar Dockerfile as https://hub.docker.com/r/cimenx/caffeonspark/~/dockerfile/ , so the Hadoop and Spark installations are managed by the CaffeOnSpark scripts, i didn't change this part.

I haven't done any kind of ssh into localhost yet.

junshi15 · 2016-12-07T18:08:39Z

@baristahell
I see this:

java.lang.UnsatisfiedLinkError: no caffedistri in java.library.path

It's likely your LD_LIBRARY_PATH was not set properly. somewhere you should have libcaffedistri.so after compilation, then LD_LIBRARY_PATH should be set accordingly.

arundasan91 · 2016-12-07T22:09:10Z

@baristahell , Please check whether:

Your hadoop (HDFS) is properly configured and running by logging into docker container and running jps.
Please check whether necessary ports are open

I have a working docker container. I had to make sure that Hadoop is working fine.

Also, in the dockerfile you shared, please correct the ENV variables. Some of them have = sign in them. That is not necessary. Please make sure all the environment variables required are set.

arundasan91 · 2016-12-08T06:25:22Z

To set the LD_LIBRARY_PATH, please include this to your docker file

ENV LD_LIBRARY_PATH $LD_LIBRARY_PATH:$CAFFE_ON_SPARK/caffe-public/distribute/lib:$CAFFE_ON_SPARK/caffe-distri/distribute/lib

baristahell · 2016-12-08T10:29:12Z

Thanks for the answers. Ok so i already had the library path already set but after the make build, i put it before this step but it didn't change the output. When i run jps in the temporary image i get 15 Jps, i don't really get what it means.

If you have the time to check the Dockerfile i use i'll put it there, i don't really see what's wrong and i bet there's a lot. If you're able to share your Dockerfile it would be amazing, i'm struggling with this thing.

## Reference:
## https://github.com/yahoo/CaffeOnSpark/wiki/GetStarted_local
## https://github.com/yahoo/CaffeOnSpark/blob/master/caffe-grid/src/test/python/PythonTest.sh

FROM ubuntu:14.04
ENV pwd /root

## Install git, wget, bc and dependencies
# https://github.com/Kaixhin/dockerfiles/blob/master/caffe/deps/Dockerfile
RUN apt-get update && apt-get install -y \
  git \
  wget \
  bc \
  cmake \
  libatlas-base-dev \
  libatlas-dev \
  libboost-all-dev \
  libopencv-dev \
  libprotobuf-dev \
  libgoogle-glog-dev \
  libgflags-dev \
  protobuf-compiler \
  libhdf5-dev \
  libleveldb-dev \
  liblmdb-dev \
  libsnappy-dev \
  python-dev \
  python-pip \
  python-numpy \
  maven \
  software-properties-common \
  gfortran > /dev/null && \
  pip install --upgrade pip

## Install Oracle Java 8
# https://github.com/dockerfile/java/blob/master/oracle-java8/Dockerfile
RUN \
  echo oracle-java8-installer shared/accepted-oracle-license-v1-1 select true | debconf-set-selections && \
  add-apt-repository -y ppa:webupd8team/java && \
  apt-get update && \
  apt-get install -y oracle-java8-installer && \
  rm -rf /var/lib/apt/lists/* && \
  rm -rf /var/cache/oracle-jdk8-installer

ENV JAVA_HOME /usr/lib/jvm/java-8-oracle

# Get CaffeOnSpark
RUN cd ${pwd} && git clone https://github.com/cimenx/CaffeOnSpark.git --recursive
ENV CAFFE_ON_SPARK ${pwd}/CaffeOnSpark

# Configure CaffeOnSpark
ADD requirements.txt ${CAFFE_ON_SPARK}/caffe-public/python/
RUN cd ${CAFFE_ON_SPARK}/caffe-public/ && \
  pip install -U -r python/requirements.txt

# Add OpenBlas & protobuf
RUN cd root && mkdir temp && cd temp && mkdir OpenBlas-0.2.19 && mkdir protobuf-2.5.0
ADD OpenBLAS-0.2.19 /OpenBlas-0.2.19
ADD protobuf-2.5.0 /protobuf-2.5.0
RUN cd OpenBlas-0.2.19 && make
RUN cd ../protobuf-2.5.0 && ./configure && make && make check && make install

# Add settings xml for Maven
ADD settings.xml /root/.m2/settings.xml
RUN cd ${CAFFE_ON_SPARK}/caffe-public/

# Install Spark and Hadoop
ADD Makefile.config ${CAFFE_ON_SPARK}/caffe-public/
RUN cd ${pwd} && bash ${CAFFE_ON_SPARK}/scripts/local-setup-hadoop.sh && \
  bash ${CAFFE_ON_SPARK}/scripts/local-setup-spark.sh \

ENV HADOOP_HOME ${pwd}/hadoop-2.6.4
ENV SPARK_HOME ${pwd}/spark-1.6.0-bin-hadoop2.6
ENV PATH ${HADOOP_HOME}/bin:${SPARK_HOME}/bin:${SPARK_HOME}/sbin:${PATH}
ENV LD_LIBRARY_PATH $LD_LIBRARY_PATH:$CAFFE_ON_SPARK/caffe-public/distribute/lib:$CAFFE_ON_SPARK/caffe-distri/distribute/lib


# Build CaffeOnSpark
RUN cd ${CAFFE_ON_SPARK} && \
  make build

# ENV LD_LIBRARY_PATH ${CAFFE_ON_SPARK}/caffe-public/distribute/lib #:${CAFFE_ON_SPARK}/caffe-distri/distribute/lib

RUN cd ${CAFFE_ON_SPARK}/data/ && unzip ${CAFFE_ON_SPARK}/caffe-grid/target/caffeonsparkpythonapi.zip

EXPOSE 8080 7077 8081
WORKDIR /root/spark-1.6.0-bin-hadoop2.6/

ENTRYPOINT ["IPYTHON=1 pyspark", \
"--driver-library-path", "${CAFFE_ON_SPARK}/caffe-grid/target/caffe-grid-0.1-SNAPSHOT-jar-with-dependencies.jar", \
"--driver-class-path", "${CAFFE_ON_SPARK}/caffe-grid/target/caffe-grid-0.1-SNAPSHOT-jar-with-dependencies.jar", \
"--jars", "${CAFFE_ON_SPARK}/caffe-grid/target/caffe-grid-0.1-SNAPSHOT-jar-with-dependencies.jar", \
"--py-files", "${CAFFE_ON_SPARK}/caffe-grid/target/caffeonsparkpythonapi.zip", \
"--files", "${CAFFE_ON_SPARK}/data/caffe/_caffe.so", \
"--conf", "spark.driver.extraLibraryPath=${LD_LIBRARY_PATH}", \
"--conf", "spark.executorEnv.LD_LIBRARY_PATH=${LD_LIBRARY_PATH}", \
"--conf", "spark.executorEnv.DYLD_LIBRARY_PATH=${LD_LIBRARY_PATH}"
]

(I just realized i cloned the repo from cimenx and not from yahoo, but same error again)

arundasan91 · 2016-12-08T15:54:47Z

@baristahell , I will take a look at the Dockerfile.

I installed CaffeOnSpark with Hadoop 2.7.1 and Spark 2.0 since I have to use Openstack object storage for accessing data. However, I had a tough time running basic examples in the docker container because of the Spark version. Maven was not building for Spark 2.

Please add this to your docker file for passwordless SSH.

# Passwordless SSH
RUN ssh-keygen -y -q -N "" -t dsa -f /etc/ssh/ssh_host_dsa_key
RUN ssh-keygen -y -q -N "" -t rsa -f /etc/ssh/ssh_host_rsa_key
RUN ssh-keygen -q -N "" -t rsa -f /root/.ssh/id_rsa
RUN cp /root/.ssh/id_rsa.pub ~/.ssh/authorized_keys

From what I found, I you should also copy the xml config files in CaffeOnSpark\scripts to your HADOOP_HOME\etc\hadoop manually. That ensures your hadoop environment have all the defaults set.
libprotobuf-dev and libopenblas-dev are enough for installing protobuf and openblas. If you are not doing heavy programming using them, there is no necessity to build them from source.
Even if I had the SPARK_HOME\bin in my $PATH variable, I was not able to run spark-submit from root. So it will be better to pass them to the docker file like this:

ENV HADOOP_HOME=/usr/local/hadoop
ENV SPARK_HOME=/usr/local/spark
ENV PATH $PATH:$JAVA_HOME/bin
ENV PATH $PATH:$HADOOP_HOME/bin
ENV PATH $PATH:$SPARK_HOME/bin
ENV PATH $PATH:$SPARK_HOME/sbin

About jps, it lists the java processes running. If you have properly set-up hadoop, when you run start-dfs.sh and start-yarn.sh you should get something like this when you run jps.

root@85e8ee9d8f41:~# jps
273 DataNode
145 NameNode
452 SecondaryNameNode
740 NodeManager
1240 Jps
633 ResourceManager

Since you are only getting jps, I doubt there are problems with hadoop config. Getting hadoop to work on docker is not trivial. Please check this link for more info on installing hadoop on Docker.

https://hub.docker.com/r/sequenceiq/hadoop-docker/~/dockerfile/

I hope I will be able to push a Dockerfile to CaffeOnSpark repo soon. Will definitely update here as well.

arundasan91 · 2016-12-08T18:11:02Z

In the mean time, please check whether this docker image works for you.

docker pull arundas/caffeonspark

Tag is v1.

You'll have to continue from Step 7) Install mnist and cifar10 dataset into its HDFS of the Getting Started Guide "https://github.com/yahoo/CaffeOnSpark/wiki/GetStarted_yarn"

arundasan91 · 2016-12-08T19:03:44Z

@baristahell , If you do not want to download the image as such, there is a Dockerfile attached with PR #208. Please use it to create a container and let us know if it worked.

baristahell · 2016-12-08T21:52:35Z

Thanks for the help! I'll check that tomorrow and will keep you updated.
I already have the cluster working (well, one never knows) so i should be able to tell you soon, if i managed to get it to run.
I'll be trying both the image and the Dockerfile and will tell you how it went

arundasan91 · 2016-12-08T21:56:07Z

Great. Take your time.

baristahell · 2016-12-09T09:59:13Z

First update: the image works and i am able to create a working container.

The only issue for now was that YARN_CONFIG_DIR wasn't defined and i had to set it manually before submitting the test job.

arundasan91 · 2016-12-09T14:54:12Z

@baristahell , Thanks for the update. I had to give either HADOOP_CONF_DIR or YARN_CONF_DIR. So I chose the former. Should have defined both. Will surely update the image.

While using the dockerfile mentioned in the PR, you will have to copy the supporting files in config directory as well. YARN_CONF_DIR should be updated in the dockerfile too, if required.

Thanks.

arundasan91 · 2016-12-09T16:24:36Z

Adding this to the docker file will mitigate the error that you faced:

ENV YARN_CONF_DIR /usr/local/hadoop/etc/hadoop

I should have missed the HADOOP_CONF_DIR in the image uploaded to docker hub.
I have already included HADOOP_CONF_DIR in the dockerfile in #208. This docker file should work out of the box without any errors since I am getting it to work.

baristahell changed the title ~~Unable to fully build, tests failed for caffe-distri~~ Unable to fully build (Docker image), tests failed for caffe-distri Dec 6, 2016

baristahell closed this as completed Dec 13, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to fully build (Docker image), tests failed for caffe-distri #205

Unable to fully build (Docker image), tests failed for caffe-distri #205

baristahell commented Dec 6, 2016

arundasan91 commented Dec 6, 2016

baristahell commented Dec 7, 2016

junshi15 commented Dec 7, 2016

arundasan91 commented Dec 7, 2016

arundasan91 commented Dec 8, 2016

baristahell commented Dec 8, 2016 •

edited

Loading

arundasan91 commented Dec 8, 2016

arundasan91 commented Dec 8, 2016 •

edited

Loading

arundasan91 commented Dec 8, 2016

baristahell commented Dec 8, 2016

arundasan91 commented Dec 8, 2016

baristahell commented Dec 9, 2016 •

edited

Loading

arundasan91 commented Dec 9, 2016 •

edited

Loading

arundasan91 commented Dec 9, 2016

Unable to fully build (Docker image), tests failed for caffe-distri #205

Unable to fully build (Docker image), tests failed for caffe-distri #205

Comments

baristahell commented Dec 6, 2016

arundasan91 commented Dec 6, 2016

baristahell commented Dec 7, 2016

junshi15 commented Dec 7, 2016

arundasan91 commented Dec 7, 2016

arundasan91 commented Dec 8, 2016

baristahell commented Dec 8, 2016 • edited Loading

arundasan91 commented Dec 8, 2016

arundasan91 commented Dec 8, 2016 • edited Loading

arundasan91 commented Dec 8, 2016

baristahell commented Dec 8, 2016

arundasan91 commented Dec 8, 2016

baristahell commented Dec 9, 2016 • edited Loading

arundasan91 commented Dec 9, 2016 • edited Loading

arundasan91 commented Dec 9, 2016

baristahell commented Dec 8, 2016 •

edited

Loading

arundasan91 commented Dec 8, 2016 •

edited

Loading

baristahell commented Dec 9, 2016 •

edited

Loading

arundasan91 commented Dec 9, 2016 •

edited

Loading