Python/net spec coordinate map and crop computation

[longjon]

This PR provides an updated version of #1975 (see also #1976; this is the new version described there). This is meant for use along with #3570 (new ND crop layer).

This version has several advantages over #1975, which make it a better candidate for merge.

1. Unlike Augment layers with their induced coordinate maps #1975, layers do not need a pointer back to their owning net, and are not responsible for inspecting each other. This preserves the clean separation between Net and Layer, whereas the strategy employed by Augment layers with their induced coordinate maps #1975 allowed for layers to interact in arbitrary ways that are difficult to reason about and make certain kinds of functionality difficult to implement. (This was the primary objection blocking the merge of Augment layers with their induced coordinate maps #1975.) The alternate strategy used here is to perform the net-level analysis at net definition time (through pycaffe's net spec), and bake the results into the parameters of the net, through a much more straightforward (e.g.) crop layer (now ND Crop layer #3570).
2. This alternate strategy makes the computed parameters user-visible and user-modifiable, which has value for debugging and understanding.
3. Cropping issues appear at specification time rather than run time.
4. Rather than adding C++ code scattered throughout layers, this patch adds a single Python file.
5. The C++ Layer interface remains unchanged; layers don't gain an extra method which may be obscure to some users.

In addition, the functionality of this code is significantly expanded from that of #1975. In particular,

1. The assumption that common ancestors can be reached by descending through first bottoms is removed. Any two spatially-mappable blobs with a common ancestor should work for cropping or otherwise computing coordinate maps.
2. Layers with multiple bottoms (e.g., Eltwise or Convolution with multiple inputs/outputs) should now be supported.
3. Rectangular filters should now be supported.
4. ND de/convolution should be supported.
5. Dilated de/convolution should be supported.

Like #1975, the coordinate maps computed with this code can be used for other purposes in addition to cropping.

The main disadvantage of this approach compared to #1975 is that the coordinate mapping formulae are kept apart from the definitions of their corresponding layers, which have to be manually kept in sync. Given that these formulae only come in a few basic forms, this seems like a reasonable approach for now.

This PR is not complete. The latest version of this code has just been written and has not been well-tested. Work still to be done:

• nudge this PR/ND Crop layer #3570 as needed for compatibility
• add a test for basic functionality
• add a test for rectangular filters
• add a test for ND convolution
• settle on names for functions (coord_map_from_to is more awkward than it needs to be)
• flesh out docstrings and comments (in particular, explain how to use coord_map_from_to)
• fix style issues
• consider whether any currently unsupported layers should be supported

[longjon]

By the way (this should become evident when tests are added): basic usage is like this:

from caffe.coord_map import crop
from caffe import layers as L

data = L.Data(...)
...
output = L.Convolution(...)
cropped_output = crop(output, data)

[seanbell]

If fn.type_name == 'Pooling', then this should check pooling_param I think, since this can get called in coord_map for pooling layers.

[shelhamer]

Not at work here, but a reminder to one day settle #1318.

[longjon]

Not quite; perhaps this warrants a comment. What's going on here is that the parameters are normally read from fn.params, which contains the conv params for a conv layer and the pooling params for a pooling layer. However, for layers which share the ConvolutionParameter message type (i.e., currently only deconv layer), we need to explicitly ask for convolution_param, since Convolution is not the name of the layer.

[SvenTwo]

Does this replace the "crop" layer types used in some fully convolutional networks? How do you perform training of fully convolutional networks if there is no crop layer to bring the per-pixel labels and the network output into the same coordinate space? Will you have to do the cropping manually as a pre-processing step on the training data? This looks rather inefficient to me.

[longjon]

@SvenTwo: #3570 becomes the new Crop layer; this code replaces #1975 in order to automatically determine crop parameters. There is no regression of functionality or speed compared to any previous branches; see details above.

[ahundt]

@longjon will #3570 and #3613 together provide the functionality in https://github.com/longjon/caffe/? I created longjon#11 to keep things going until the requisite parts are done. Also, is this close to merging? Things seem to have stalled in both for a month or so. Apologies if this is the wrong place to post this.

[shelhamer]

add ELU, Scale, Bias

[shelhamer]

@ahundt right, python coord map #3613/this and crop layer #3570 together deliver the same functionality and efficiency of longjon/caffe:future but with less intrusion into the framework and layers.

[ahundt]

@shelhamer thanks! Also are these two patches applied to master compatible with the exiting trained .caffemodel and .prototext of the version in https://github.com/longjon/caffe?

[ahundt]

The answer to my prior question is that no yes, they are not compatible.

From the model zoo:

• fcn-32s runs with expected output Update: it seems this just happens to work by chance, crop does nothing as mentioned in the next post.
• fcn-8s crashes Here are the relevant logs

printed output in python:

I0229 14:37:44.157938 2077495296 layer_factory.hpp:77] Creating layer fuse
I0229 14:37:44.157945 2077495296 net.cpp:91] Creating Layer fuse
I0229 14:37:44.157949 2077495296 net.cpp:425] fuse <- upscore2_upscore2_0_split_1
I0229 14:37:44.157953 2077495296 net.cpp:425] fuse <- score-pool4c
I0229 14:37:44.157958 2077495296 net.cpp:399] fuse -> score-fused
F0229 14:37:44.157968 2077495296 eltwise_layer.cpp:34] Check failed: bottom[i]->shape() == bottom[0]->shape() 
*** Check failure stack trace: ***

key lines of stack trace:

7   libglog.0.dylib                 0x000000011fa91015 google::LogMessageFatal::~LogMessageFatal() + 15
8   libglog.0.dylib                 0x000000011fa8e363 google::LogMessageFatal::~LogMessageFatal() + 9
9   libcaffe.so.1.0.0-rc3           0x000000011acf1f4a caffe::EltwiseLayer<float>::Reshape(std::__1::vector<caffe::Blob<float>*, std::__1::allocator<caffe::Blob<float>*> > const&, std::__1::vector<caffe::Blob<float>*, std::__1::allocator<caffe::Blob<float>*> > const&) + 202
10  libcaffe.so.1.0.0-rc3           0x000000011ad466b9 caffe::Net<float>::Init(caffe::NetParameter const&) + 3449
11  libcaffe.so.1.0.0-rc3           0x000000011ad47de6 caffe::Net<float>::Net(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, caffe::Phase, caffe::Net<float> const*) + 454

@shelhamer is this perhaps a bug in this pull request?

[longjon]

@ahundt, no (unless @shelhamer has made updates I'm not aware of), existing prototxts are not compatible with these two PRs. They will probably run with Crop layers doing nothing, so the FCN-8s dimension mismatch is exactly as expected.

[shelhamer]

@ahundt @longjon no, existing models are not compatible but we plan to bundle up nets in a standard format once master is sorted out. It's only an architectural/configuration difference however and the same weights will be fine.

[shelhamer]

With this PR @ 775a5c7 and #3570 I have been able to reproduce FCN experiments. a3359a4 adds tests so this can be merged once #3570 is in.

[ahundt]

@shelhamer Is it possible to post new or updated pre-trained fcn-xx models using this code in the model zoo?

[weiliu620]

Does pycaffe support 'Crop' layer? I run caffe.Net to load model

net = caffe.Net(model_file, model_weights, caffe.TEST)

and got error:

F0419 10:14:49.936854 10131 layer_factory.hpp:77] Check failed: registry.count(type) == 1 (0 vs. 1) Unknown layer type: Crop (known types: AbsVal, Accuracy, ArgMax, BNLL, Concat, ContrastiveLoss, Convolution, Data, Deconvolution, Dropout, DummyData, Eltwise, EuclideanLoss, Exp, Flatten, HDF5Data, HDF5Output, HingeLoss, Im2col, ImageData, InfogainLoss, InnerProduct, LRN, MVN, MemoryData, MultinomialLogisticLoss, Pooling, Power, ReLU, Sigmoid, SigmoidCrossEntropyLoss, Silence, Slice, Softmax, SoftmaxWithLoss, Split, TanH, Threshold, WindowData)

(I used latest caffe master branch)