Implement Faster R-CNN on CaffeOnSpark

[gnosisyuw]

I am triyng to implement Faster R-CNN on CaffeOnSpark. Since there are several customized layers in the Faster R-CNN. I modified the caffe-public part to include these additional layers into the CaffeOnSpark. Currently, the build process was successful, but the subsequent test (caffe-distri) reported error. The weird thing was that every time I tried to resume the test, it would give me different errors.

ERROR 1:
I0629 18:29:09.404780 19341 net.cpp:226] pool1 needs backward computation.
I0629 18:29:09.404783 19341 net.cpp:226] relu1 needs backward computation.
I0629 18:29:09.404786 19341 net.cpp:226] conv1 needs backward computation.
I0629 18:29:09.404789 19341 net.cpp:228] label_data_1_split does not need backward computation.
I0629 18:29:09.404793 19341 net.cpp:228] data does not need backward computation.
I0629 18:29:09.404795 19341 net.cpp:270] This network produces output accuracy
I0629 18:29:09.404798 19341 net.cpp:270] This network produces output loss
I0629 18:29:09.404817 19341 net.cpp:283] Network initialization done.
I0629 18:29:09.404875 19341 solver.cpp:60] Solver scaffolding done.
CaffeNetTest training:.I0629 18:29:09.515323 19341 MemoryInputAdapter.cpp:15] MemoryInputAdapter is used
F0629 18:29:09.515372 19341 memory_data_layer.cpp:88] Check failed: n % batch_size_ == 0 (-3 vs. 0) n must be a multiple of batch size
*** Check failure stack trace: ***
Aborted (core dumped)
ERROR 2:

T E S T S

Running TestSuite
Configuring TestNG with: org.apache.maven.surefire.testng.conf.TestNG652Configurator@4d112e85
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0629 18:32:26.088688 21973 CaffeNet.cpp:78] set root solver device id to 0

A fatal error has been detected by the Java Runtime Environment:

SIGSEGV (0xb) at pc=0x00002b2296b764c0, pid=21970, tid=47427528435456

JRE version: OpenJDK Runtime Environment (7.0_101) (build 1.7.0_101-b00)
Java VM: OpenJDK 64-Bit Server VM (24.95-b01 mixed mode linux-amd64 compressed oops)
Derivative: IcedTea 2.6.6
Distribution: Ubuntu 14.04 LTS, package 7u101-2.6.6-0ubuntu0.14.04.1
Problematic frame:
C [libstdc++.so.6+0xbb4c0] std::string::assign(std::string const&)+0x10

Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again

ERROR 3:
I0629 17:40:57.415719 14776 net.cpp:228] label_data_1_split does not need backward computation.
I0629 17:40:57.415721 14776 net.cpp:228] data does not need backward computation.
I0629 17:40:57.415724 14776 net.cpp:270] This network produces output accuracy
I0629 17:40:57.415725 14776 net.cpp:270] This network produces output loss
I0629 17:40:57.415736 14776 net.cpp:283] Network initialization done.
I0629 17:40:57.415789 14776 solver.cpp:60] Solver scaffolding done.

A fatal error has been detected by the Java Runtime Environment:

SIGSEGV (0xb) at pc=0x00002ac1671efe60, pid=14773, tid=47008695981824

JRE version: OpenJDK Runtime Environment (7.0_101) (build 1.7.0_101-b00)
Java VM: OpenJDK 64-Bit Server VM (24.95-b01 mixed mode linux-amd64 compressed oops)
Derivative: IcedTea 2.6.6
Distribution: Ubuntu 14.04 LTS, package 7u101-2.6.6-0ubuntu0.14.04.1
Problematic frame:
C [libcaffe.so.1.0.0-rc3+0x13de60] caffe::MemoryDataParameter::MergeFrom(caffe::MemoryDataParameter const&)+0x70

Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again.

Can someone give me some advice about this error? More generally, if I want to include customized layer into CaffeOnSpark, what modification do I need to do on the CaffeOnSpark?

Thanks

[junshi15]

Is fast-RNN already at BVLC caffe? If so, we can update caffe-public.

Regarding customized layers, as long as they don't touch input layer, it should fine. For example, your own inner-product layer should be just fine. But there may be issues if you introduce your own data layer.

[gnosisyuw]

@junshi15 Thanks for your reply. Currently , Faster RCNN is not included in BVLC caffe. But these customized layers didn't touch the input layer. Do you have any clue that what result in this error?

BTW, it would be very helpful if your team update caffe-public. There are several additional layers added recently.

[mriduljain]

I think we recently updated the LSTM's in BVLC caffe when Jeff Donahue's
branch was merged in BVLC. That was the last merge for CaffeOnSpark as
well, as far as I remember.

On Fri, Jul 1, 2016 at 7:34 PM, GnosisYu notifications@github.com wrote:

@junshi15 https://github.com/junshi15 Thanks for your reply. Currently
, Faster RCNN is not included in BVLC caffe. But these customized layers
didn't touch the input layer. Do you have any clue that what result in this
error?

BTW, it would be very helpful if your team update caffe-public. There are
several additional layers added recently.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#99 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/ACCTVfW4SSVUVd9KRh-H3P04e1z9FNypks5qRc45gaJpZM4JBtcA
.

[junshi15]

We have not updated caffe-public since it is open-sourced. https://github.com/yahoo/caffe/commits/e107fb7a0d737868aff401ef954c1d8059e5e450
We will update it shortly.
@mriduljain You must be talking about a different version.

[mriduljain]

Ok I forgot. I updated it internally. Will send a pull request shortly

On Saturday, July 2, 2016, Jun Shi notifications@github.com wrote:

We have not updated caffe-public since it is open-sourced.
https://github.com/yahoo/caffe/commits/e107fb7a0d737868aff401ef954c1d8059e5e450
We will update it shortly.
@mriduljain https://github.com/mriduljain You must be talking about a
different version.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#99 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/ACCTVXZsDbruVe9oV-RWbYY33n7L6V7Kks5qRnBXgaJpZM4JBtcA
.

[gnosisyuw]

@junshi15 @mriduljain Problem is solved. Thx.