Use of CRFRNN on different caffe build

[nk-dev0]

Thanks for this very useful paper. I'm looking to use another segmentation architecture (which requires a completely different build of caffe), but then append the crf-rnn to the last layer of the other network before the softmax loss. This will require combining both caffe versions, and since the other version has a lot of changes and custom layers from the main caffe branch, I was thinking it might be easier to merge multi_stage_meanfield.cpp (and whatever other files are needed) to their build.

I'm a bit unclear as to which files I need to copy over that you have changed in caffe to enable crf-rnn to work, or if that's even how to incorporate your caffe contributions, so any direction on this would be extremely helpful. Thanks!

[nk-dev0]

For anyone curious, I followed the steps outlined here: BVLC/caffe#684

I moved the relevant layers for meanfield iteration and multi_stage_meanfield_iteration (both the *.cpp and *.hpp files) to the other caffe build, and then updated the files as described in the post I linked above

[nathanin]

Hi nk,
I've been trying to move the layers over to another customized caffe version (here). As far as I can tell, the process is pretty smooth just like you described. In troubleshooting errors encountered in make runtest #125 I guess there's a problem with some cuda parts in modified_permutohedral.*pp.

Have you run into these problems yourself? Based purely on googling I guess the cuDNN version here is v3, whereas I have v5 installed. I'm not a cuda programmer, so don't know where to begin debugging.

[nk-dev0]

Wow, I actually used the exact same segnet caffe version that you linked as well. I believe I used this crfrnn repo for an updated cudnn implementation. However, with that repo, some of the file organization is different, so you may need to move some things around. For example, the header files are organized into groups such as vision_layers.hpp, instead of an individual file for each *.cpp file.

Unfortunately I didn't take notes on what exactly I did, but you could try pulling in just the modified_permutohedral.cu and other relevant *.cu files from the repo I linked and then go from there. I haven't gotten a chance to commit everything yet as this is fairly low on my to-do list, but I'll do it within the next few days/weeks when I get a chance.

In terms of actual performance, after training the segnet model on 3 classes, and then continuing training with the crf-rnn for around 15k iterations, I found a basically negligible improvement. For my task, mean IoU increased less than a percentage point, although I wasn't expecting much with my fairly noisy training data.

[nathanin]

Yes! I've been trying to consolidate, or find any differences at all, between the linked repo, and the one here. The latter version uses the unraveled hpp file for each cpp file structure so it was a bit more obvious which files were missing. So either A) I messed up the copy+pasting, B) it's more complicated than copy+paste the missing .cpp, .cu and .hpp's or C) the intended cuda version is older.

Also, thank you for letting me know the improvement is little. That helps curb how much effort to put in here. (At this point it's more about doing the merge successfully...) I was planning to try training the CRF layer at the same time as the initial seg-net. According to their paper doing it this way is superior to training one, then the other. Although maybe I misunderstood what you did. My data has large areas which should be all one class, separated by narrow but strongly segmented areas. That's why I think an approach like this will help in my dataset.