Classification performance is very slow like half a minute on AWS G2 instance?

[5argon]

I looked through the paper but I could not find any speed benchmark that I can compare with. I tried crfasrnn_demo.py on a single instance of g2.2xlarge Amazon AWS and got unexpectedly slow classification speed and wondering if something is wrong or not.

The code below in crfasrnn_demo.py that I modified to time :

# Get predictions
start = time.time()
segmentation = net.predict([im])
end = time.time()
segmentation2 = segmentation[0:cur_h, 0:cur_w]
output_im = PILImage.fromarray(segmentation2)
output_im.putpalette(pallete)

plt.imshow(output_im)
plt.savefig('output.png')
print(end-start)

The result is 29.7011039257 seconds. What is the average time required for this demo?

[bittnt]

Because you are using CPU. Change the setting to make it running on GPU please.

[5argon]

Alright, I saw the line that says set_mode_gpu and uncommented it but however now it says

AttributeError: 'module' object has no attribute 'set_mode_gpu'

Could this be a problem of the make pycaffe build not building the gpu support? But I followed steps in the guide exactly. (Turn on USE_CUDNN and nothing else)

[5argon]

I found out that set_mode_gpu is an attribute of a net, not the Caffe itself.

The performance I got with GPU enabled is now better, 5.24521899223 seconds. Still, I expect something faster since I think usually classification is faster than this since I heard mobile device can also use the trained network efficiently. How long does it take for you?

[dalafer]

Hi guys, I have the same situation, taking about 30 seconds to finish.
I have uncommented USE_CUDNN and set as 1. I then recompiled and ran the python script. It still took 30 seconds. How can one confirm if GPU is being used?

Note: I did not uncomment set_mode_gpu, when I did it gave me the same error as 5argon.

[bittnt]

Hi. @abdalafamsee

This means the code is running entirely on single CPU.

The current code on this repository works with cudnn version <=3. But you should make the code works with other branch: https://github.com/mtourne/crfasrnn or even integrate the multi-stage-mean-field layer with the caffe upstream code.

[dalafer]

@bittnt thank you I will give that one a try and report back with results

[dalafer]

@bittnt I did not have success.
These were my steps:

• git clone --recursive https://github.com/mtourne/crfasrnn.git
• mv Makefile.config.example Makefile.config, then edited to uncomment USE_CUDNN
• make
• make pycaffe
• cd python-scripts
• sh download_trained_model.sh
• python crfasrnn_demo.py

It still took 30 seconds to finish. What else do you suggest I look to make sure its running on the GPU?

I ran cmake to see the configurations and versions of everything, this is the output:

******************* Caffe Configuration Summary *******************
-- General:
-- Version : (Caffe doesn't declare its version in headers)
-- Git : 0.1-10-g39ded56-dirty
-- System : Linux
-- C++ compiler : /usr/bin/c++
-- Release CXX flags : -O3 -DNDEBUG -fPIC -Wall -Wno-sign-compare -Wno-uninitialized
-- Debug CXX flags : -g -fPIC -Wall -Wno-sign-compare -Wno-uninitialized

-- Build type : Release

-- BUILD_SHARED_LIBS : ON
-- BUILD_python : ON
-- BUILD_matlab : OFF
-- BUILD_docs : ON

-- CPU_ONLY : OFF

-- Dependencies:
-- BLAS : Yes (Atlas)
-- Boost : Yes (ver. 1.54)
-- glog : Yes
-- gflags : Yes
-- protobuf : Yes (ver. 2.5.0)
-- lmdb : Yes (ver. 0.9.10)
-- Snappy : Yes (ver. 1.1.0)
-- LevelDB : Yes (ver. 1.15)
-- OpenCV : Yes (ver. 2.4.8)

-- CUDA : Yes (ver. 6.5)

-- NVIDIA CUDA:
-- Target GPU(s) : Auto
-- GPU arch(s) : sm_30

-- cuDNN : Yes

-- Python:
-- Interpreter : /usr/bin/python2.7 (ver. 2.7.6)
-- Libraries : /usr/lib/x86_64-linux-gnu/libpython2.7.so (ver 2.7.6)

-- NumPy : /usr/local/lib/python2.7/dist-packages/numpy/core/include (ver 1.9.2)

-- Documentaion:
-- Doxygen : No

-- config_file :

-- Install:

-- Install path : /var/www/crfasrnn_oringal/caffe-crfrnn/install

-- Configuring done
-- Generating done

[bittnt]

Check the comments :#13

To activate gpu mode you can just add a True at the end of the Segmenter
initialization:
net = caffe.Segmenter(MODEL_FILE, PRETRAINED, True)

On Wed, Feb 17, 2016 at 7:01 PM abdalafamsee notifications@github.com
wrote:

@bittnt https://github.com/bittnt I did not have success.
These were my steps:

• git clone --recursive https://github.com/mtourne/crfasrnn.git
• mv Makefile.config.example Makefile.config, then edited to uncomment
  USE_CUDNN
• make
• make pycaffe
• cd python-scripts
• sh download_trained_model.sh
• python crfasrnn_demo.py

It still took 30 seconds to finish. What else do you suggest I look to
make sure its running on the GPU?

—
Reply to this email directly or view it on GitHub
#25 (comment).

[dalafer]

@bittnt Great I think it worked however I have a new error:
F0217 20:19:43.300274 31951 syncedmem.cpp:66] Check failed: error == cudaSuccess (2 vs. 0) out of memory.

Would this be associated with the warning:
Memory required for data: 1287634208
[libprotobuf WARNING google/protobuf/io/coded_stream.cc:505] Reading dangerously large protocol message. If the message turns out to be larger than 2147483647 bytes...... ?

This is what I am working with, on EC2 GPU instance:

00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma](rev 02)
00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 01)
00:02.0 VGA compatible controller: Cirrus Logic GD 5446
00:03.0 VGA compatible controller: NVIDIA Corporation GK104GL [GRID K520](rev a1)
00:1f.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device (rev 01)

[bittnt]

Nope, nothing to do with that message. This tells u that your GPU memory is
not big enough to run the network. Check the apply the pull request of
caffe which allows to save memory. BVLC/caffe#2016
On Wed, 17 Feb 2016 at 20:22, abdalafamsee notifications@github.com wrote:

@bittnt https://github.com/bittnt Great I think it worked however I
have a new error:
F0217 20:19:43.300274 31951 syncedmem.cpp:66] Check failed: error ==
cudaSuccess (2 vs. 0) out of memory.

Would this be associated with the warning:
Memory required for data: 1287634208
[libprotobuf WARNING google/protobuf/io/coded_stream.cc:505] Reading
dangerously large protocol message. If the message turns out to be larger
than 2147483647 bytes...... ?

—
Reply to this email directly or view it on GitHub
#25 (comment).

[dalafer]

Hi, I have pulled and compiled the caffe you suggested but i get:
'module' object has no attribute 'Segmenter'

I got https://github.com/BVLC/caffe, placed it inside crfasrnn, updated Makefile. then make and make pycaffe.

[bittnt]

@dalafer This is because the python wrapper are different in the customized caffe version in crf-rnn. You need to add those module such as Segmenter.py to the latest caffe. There are some changes recently in caffe python interface.

[dalafer]

Thank you I got it.
I used BVLC/caffe and added all the custom files from crfasrnn. I got it down to 9 seconds now.

[tharun2011]

These were my steps:
1)git clone https://github.com/mtourne/crfasrnn.git
2)Made necessary changes in Makefile.config
3)make
Running "make" command causing this error. What could be the issue? Suggestions needed.

./include/caffe/neuron_layers.hpp:505: error: ‘cudnnTensorDescriptor_t’ does not name a type
./include/caffe/neuron_layers.hpp:587: error: ‘cudnnTensorDescriptor_t’ does not name a type
./include/caffe/neuron_layers.hpp:588: error: ‘cudnnTensorDescriptor_t’ does not name a type
./include/caffe/neuron_layers.hpp:672: error: ‘cudnnTensorDescriptor_t’ does not name a type
./include/caffe/neuron_layers.hpp:673: error: ‘cudnnTensorDescriptor_t’ does not name a type
In file included from ./include/caffe/vision_layers.hpp:10,
                 from src/caffe/layers/prelu_layer.cpp:6:
./include/caffe/common_layers.hpp:598: error: ‘cudnnTensorDescriptor_t’ does not name a type
./include/caffe/common_layers.hpp:599: error: ‘cudnnTensorDescriptor_t’ does not name a type
In file included from src/caffe/layers/prelu_layer.cpp:6:
./include/caffe/vision_layers.hpp:337: error: ‘cudnnTensorDescriptor_t’ was not declared in this scope
./include/caffe/vision_layers.hpp:337: error: template argument 1 is invalid
./include/caffe/vision_layers.hpp:337: error: template argument 2 is invalid
./include/caffe/vision_layers.hpp:338: error: ‘cudnnTensorDescriptor_t’ does not name a type
./include/caffe/vision_layers.hpp:556: error: ‘cudnnTensorDescriptor_t’ does not name a type
make: *** [.build_release/src/caffe/layers/prelu_layer.o] Error 1

[5argon]

Um.. it means the program says has cudnnTensorDescriptor_t written as a type of some variable but could not find the definition. It would be that you are missing CUDNN library or installed but the program could not find it. (path problem, version problem, etc.)

(Anyway this is not related to the speed of G2 instances, why not make a new thread? I might as well close this since we found out that CPU -> GPU is the culprit)

[Eniac-Xie]

Hi, @dalafer
do you use the BVLC/caffe instead of caffe-crfrnn? I tried to clone BVLC/caffe and run crfrnn python demo, but it seems that the multi_stage_meanfield layer is not implemented by BVLC/caffe, did you add this layer by yourself? Thank you!

[5argon]

@Eniac-Xie did you read the author's paper yet? I think multi_stage_meanfield is the heart of this paper as this should be the RNN modeled after CRF that has been placed directly after FCN-8s. Since this is the new innovation by the paper, naturally it would not exist in the normal version of Caffe.

[Eniac-Xie]

thank you @5argon. I have read the paper. But I want to make sure that whether BVLC/caffe has merge the multi_stage_meanfield or anybody has release a new version caffe with multi_stage_meanfield.

[Hartlaw]

@bittnt @dalafer

Good afternoon,

Sorry to bother you but actually I did not get how you finally succeed to add Segmenter.py in your project.. Could you tell me how you proceed please ? I did not find such a file in the GitHub documents.

So far, I have installe caffe for windows (I am really new to programming and I have limited time so I cannot learn to use Ubuntu), installed all the dependencies, build caffe, pycaffe, and all the others but this file -Segmenter.py- is still missing.

Thank you in advance for your help !

[bittnt]

@Hartlaw https://github.com/bittnt/caffe/blob/aa533ad08288ba6a22198d49ce707eef0d347349/python/caffe/segmenter.py

[Hartlaw]

@bittnt Sorry guess I missed it, thank you very much !

[Hartlaw]

@bittnt Hello, sorry to bother you again.
I have added the segmenter.py file where it belongs but the error is still here, like if crfasrnn_demo.py could not see it.
Then I assumed that the file segmenter.py could have a problem on my computer so I tried to run it by using my command prompt language but when I did this, it was said that there is no attribute 'bool'.

Actually I have tried many things since yesterday but given that I am quite new to this, I am kind lost about what I should try..

Others things that can be leads:

• I run under Windows
  -I think that I have all the depedencies given I have followed the Caffe tutorial dedicated to Windows
• I did not do "make pycaffe" or else "make" action since I have understood that it is for Ubuntu
• I do not have any file called _caffe.so (only _caffe.cpp or _caffe.pyd)
• I did not make any change in Makefile given that each time I tried to modify it, I was not able to build my caffe.sln right after..
  -In crfasrnn_demo.py, I have put the following path : ..C\Users\user\Desktop\CRF_As_RNN\crfasrnn-master\caffe
• I have copied the folder: C:\Users\user\Desktop\CRF_As_RNN\crfasrnn-master\caffe\Build\x64\Release\pycaffe\caffe to C:\Miniconda2\Lib\site-packages

I think that I have said all that might be important to know;

Thank you in advance for your help !

[neobarney]

I'm able to complete the classification in about 4.5 s with a K80 or a P100.
Seems still pretty slow to me, does anybody got any update on this ?
thanks