Blobs are N-D arrays (for N not necessarily equals 4) #1970

jeffdonahue · 2015-02-25T23:01:05Z

Replaces #1486 -- this PR is to master instead of dev. This is rebased and is ready for review @longjon.

This PR gives Blobs a vector<int> shape_ of dimensions, rather than the old num, channels, height, width. The first commit is all the changes needed to get Caffe to compile and everything to run as before with the new vector of tensor dimensions. The remaining commits generalize some existing classes to use the new tensor dimensions (but they are not necessary to make it run, as it's still fine to just use all 4-D blobs with extra singleton dimensions where needed).

jeffdonahue · 2015-03-02T01:17:37Z

I had added a commit intended to add matcaffe support for N-D array blobs, but decided to defer to #1913 to add the support, as I don't have much experience with matcaffe and it might require some special cases (e.g., I remembered that MATLAB doesn't support arrays with <2 axes so would need to figure out what to do with those cases -- maybe the MATLAB default of adding extra axes with dimension 1 would work fine, but not sure). But if that commit might be helpful in the development of #1913, feel free to cherry-pick or just refer to it here (but note that I didn't test or even try to compile it...).

So for now, matcaffe will continue to work fine for blobs with <= 4 axes, but will die on attempts to access blobs with >4 dimensions (per their use of num/channels/height/width, which call LegacyShape).

Yangqing · 2015-03-02T06:20:39Z

src/caffe/proto/caffe.proto

@@ -2,13 +2,21 @@ syntax = "proto2";

 package caffe;

+// Specifies the shape (dimensions) of a Blob.
+message BlobShape {
+  repeated int64 dim = 1 [packed = true];


My 2 cents - uint32 should be enough? A 2G blob (8G in floats) looks big enough.

(But yeah, 640k used to be enough for everything.)

My feeling is that it's probably worth future-proofing given the relatively tiny amount of extra storage cost (unless you're really using blobs with INT_MAX axes, as are now allowed...). I could make it uint32 for now and then later change it to int64 if/when we really find practical uses for blobs that large, but that might cause backward compatibility problems with (de)serialization that I don't even want to think about... (Why not uint64? I think it's worth keeping -1 and other <0 values reserved for various purposes -- e.g. setting axis [default = -1] to signify "all axes", or that an axis must be manually specified and should fail otherwise, etc. And it does seem pretty unlikely that in the reasonably foreseeable future we'll need 2^64 byte blobs, so 2^63-1 bytes seems like an okay max...)

I'm open to discussion on any of this though.

Future-proof sounds better :) Thanks for checking with me!

But Jeff, there is no reason for any individual to have a > 2GB blob in their home. Haha no, I'm all for int64 dimensions.

longjon · 2015-03-02T23:02:34Z

Oops, just realized that I've been commenting on commits again. Agree with int64 blob dimensions. I'm ready to merge this today, pending comments made, the new type for Python Blob.reshape (which I'll do), and a nod from @shelhamer.

longjon · 2015-03-02T23:59:41Z

Added my Python update... this should be ready pending your approval of that, Travis, and @shelhamer.

jeffdonahue · 2015-03-03T00:05:05Z

Thanks Jon! Your updated Blob_Reshape looks good to me.

shelhamer · 2015-03-03T03:25:54Z

The axis field in the proto is inconsistently (u)int32 -- should it be int32 everywhere to allow positive and negative indexing in Slice, Concat, and InnerProduct dimension picking? Or it's fine to keep this the same for now as current behavior.

shelhamer · 2015-03-03T03:43:46Z

src/caffe/layers/inner_product_layer.cpp

-  K_ = bottom[0]->count() / bottom[0]->num();
+  const int dim = this->layer_param_.inner_product_param().axis();
+  // Dimensions starting from "axis" are "flattened" into a single
+  // length K_ vector. For example, if bottom[0]'s shape is (N, C, H, W),


For example, if bottom[0]'s shape is (N, C, H, W) and axis = 1

shelhamer · 2015-03-03T03:58:14Z

@jeffdonahue this is sweet. My only comments were minor, so do what you want with them and merge!

shelhamer · 2015-03-03T04:01:34Z

src/caffe/layers/softmax_loss_layer.cpp

@@ -35,6 +35,9 @@ void SoftmaxWithLossLayer<Dtype>::Reshape(
    const vector<Blob<Dtype>*>& bottom, const vector<Blob<Dtype>*>& top) {
  LossLayer<Dtype>::Reshape(bottom, top);
  softmax_layer_->Reshape(softmax_bottom_vec_, softmax_top_vec_);
+  softmax_axis_ = this->layer_param_.softmax_param().axis();
+  outer_num_ = bottom[0]->count(0, softmax_axis_);
+  inner_num_ = bottom[0]->count(softmax_axis_ + 1);


This could be a good time to add a check that the prediction and target dimensions (except channel) agree?

done in a717963 -- I was a little less strict about it, just checking that the number of labels matches the number of predictions, as I couldn't figure out a way to do it that's backward compatible and nice to use in the future. (For example, I didn't want to break the current behavior of (NxCxHxW) predictions with (Nx1xHxW) labels, but it seems silly to require the singleton axis in the labels, but requiring (NxHxW) labels -- no singleton axis -- breaks backwards compatibility.)

jeffdonahue · 2015-03-03T07:39:37Z

Hey Evan, thanks for the review! I made the additional commits above in response to your comments -- ~~currently marked with fixup! to be autosquashed later~~. I thought you might want to take a look since, re negative indexing, I applied it in SliceLayer, ConcatLayer, SoftmaxLayer, and InnerProductLayer, and ended up adding an additional Blob method CanonicalAxisIndex (as I started to realize I was writing the same checks several times). I added a single test that actually uses it to TestConcatLayer.

Edit: I went ahead and squashed the fixups, but I saved the state before the squash in another branch "tensor-blob-presquash" if you still want to look at only the additional changes I made after your review.

vector<int> shape_ instead of (num, channels, height, width).

…2063

Fixup AccuracyLayer like SoftmaxLossLayer in #1970

…tector.py

…#1694)

Update docs for ND blobs (#1970) and layer type is a string (#1694)

* master: (21 commits) Update docs for ND blobs (BVLC#1970) and layer type is a string (BVLC#1694) Add ReshapeParameter axis and num_axes to reshape only a particular span of the input shape basic tests (Forward, Gradient) for ReshapeLayer ReshapeLayer fixups for ND blobs Added a Reshape layer for copying-free modification of blob dimensions. Spatial Pyramid Pooling Layer remove bogus implementation of SigmoidCrossEntropyLossLayer::Forward_gpu remove superfluous empty destructors [pycaffe] use bp::object instead of PyObject* for self in Python layer python: PEP8; changed docstring documentation style to NumPyDoc style This imports the wrong io module in Python 3. check that count_ does not overflow in Blob::Reshape Modify for better readability regarding temporary bufffer for backward computation Fix redundancy of parameter backward computation Added support for original implementation, using (margin - d^2), through the legacy_version parameter. added epsilon to prevent possible division by zero in gradient calculation Fixed contrastive loss layer to be the same as proposed in Hadsell et al 2006 remove spurious net.hpp includes always call Layer::Reshape in Layer::Forward Increment iter_ before snapshotting, remove +1 logic -- fixes final snapshot being off by one ...

…#1694)

@dgmp88

fix BVLC#2041 reported by @dgmp88

…2063

…#1694)

shelhamer added JL ES ready for review labels Feb 25, 2015

jeffdonahue force-pushed the tensor-blob branch from d225201 to bc082bd Compare February 26, 2015 03:38

This was referenced Feb 28, 2015

Xavier filler and inner product parameters #1575

Closed

MSRA weight filler #1946

Closed

jeffdonahue force-pushed the tensor-blob branch 3 times, most recently from af72f0c to ff20e17 Compare March 2, 2015 00:58

Yangqing reviewed Mar 2, 2015
View reviewed changes

shelhamer mentioned this pull request Mar 2, 2015

Matcaffe2 #1913

Closed

jeffdonahue force-pushed the tensor-blob branch from ff20e17 to b8f3b3d Compare March 2, 2015 22:53

jeffdonahue force-pushed the tensor-blob branch 2 times, most recently from 221b2f0 to 7419b79 Compare March 2, 2015 23:36

shelhamer reviewed Mar 3, 2015
View reviewed changes

jeffdonahue force-pushed the tensor-blob branch from 025dd23 to 8b6e912 Compare March 3, 2015 07:34

jeffdonahue force-pushed the tensor-blob branch 3 times, most recently from 1ef0e32 to 8a83d07 Compare March 3, 2015 08:03

jeffdonahue added 3 commits March 3, 2015 15:55

Blobs are ND arrays (for N not necessarily equals 4).

1434e87

vector<int> shape_ instead of (num, channels, height, width).

Add BlobShape message; use for Net input shapes

5407f82

add offset, {data,diff}_at nd blob accessors

119a1c6

jeffdonahue added a commit to jeffdonahue/caffe that referenced this pull request Mar 9, 2015

Fixup AccuracyLayer like SoftmaxLossLayer in BVLC#1970 -- fixes BVLC#…

3385d1c

…2063

jeffdonahue mentioned this pull request Mar 9, 2015

FlattenLayer gets a FlattenParameter with an axis, end_axis #2082

Merged

jeffdonahue added a commit to jeffdonahue/caffe that referenced this pull request Mar 9, 2015

Fixup AccuracyLayer like SoftmaxLossLayer in BVLC#1970 -- fixes BVLC#…

6d5c8b2

…2063

jeffdonahue added a commit to jeffdonahue/caffe that referenced this pull request Mar 9, 2015

Fixup AccuracyLayer like SoftmaxLossLayer in BVLC#1970 -- fixes BVLC#…

7a40f74

…2063

jeffdonahue added a commit that referenced this pull request Mar 9, 2015

Merge pull request #2076 from jeffdonahue/accuracy-layer-fixes

77ab8f6

Fixup AccuracyLayer like SoftmaxLossLayer in #1970

jeffdonahue mentioned this pull request Mar 10, 2015

Very simple version of ReshapeLayer #2088

Closed

qinhongwei pushed a commit to qinhongwei/caffe that referenced this pull request Mar 12, 2015

[pycaffe] no need to squeeze output after BVLC#1970, add change to de…

844bdb6

…tector.py

seanbell mentioned this pull request Mar 16, 2015

Extract Features from fc7 #700

Closed

jeffdonahue added a commit to jeffdonahue/caffe that referenced this pull request Mar 26, 2015

Update docs for ND blobs (BVLC#1970) and layer type is a string (BVLC…

f10c43b

…#1694)

jeffdonahue added a commit to jeffdonahue/caffe that referenced this pull request Mar 26, 2015

Update docs for ND blobs (BVLC#1970) and layer type is a string (BVLC…

0afd1bf

…#1694)

jeffdonahue added a commit to jeffdonahue/caffe that referenced this pull request Mar 26, 2015

Update docs for ND blobs (BVLC#1970) and layer type is a string (BVLC…

a9d4adc

…#1694)

pclove1 mentioned this pull request Apr 7, 2015

fix #1362. prefetch HDF5DataLayer #2271

Closed

jeffdonahue added a commit to jeffdonahue/caffe that referenced this pull request May 15, 2015

Update docs for ND blobs (BVLC#1970) and layer type is a string (BVLC…

cf0e6b7

…#1694)

shelhamer added a commit that referenced this pull request May 15, 2015

Merge pull request #2201 from jeffdonahue/tutorial-fixes

af224c1

Update docs for ND blobs (#1970) and layer type is a string (#1694)

jeffdonahue mentioned this pull request May 17, 2015

int->size_t to support large datasets more than 2G instances #2473

Closed

futurely mentioned this pull request May 17, 2015

Add support for N-D arrays (for N greater than 4) arrayfire/arrayfire#677

Closed

This was referenced May 24, 2015

MatCaffe3 #2505

Merged

Matcaffe: output score value is not between [0 and 1] #2515

Closed

jeffdonahue mentioned this pull request Jun 2, 2015

ND convolution with im2col #2049

Merged

matthiasplappert pushed a commit to matthiasplappert/caffe that referenced this pull request Aug 10, 2015

Update docs for ND blobs (BVLC#1970) and layer type is a string (BVLC…

a7b2aab

…#1694)

cbfinn pushed a commit to cbfinn/caffe that referenced this pull request Aug 12, 2015

[pycaffe] no need to squeeze output after BVLC#1970

0bde346

fix BVLC#2041 reported by @dgmp88

cbfinn pushed a commit to cbfinn/caffe that referenced this pull request Aug 12, 2015

Fixup AccuracyLayer like SoftmaxLossLayer in BVLC#1970 -- fixes BVLC#…

d37d928

…2063

cbfinn pushed a commit to cbfinn/caffe that referenced this pull request Aug 12, 2015

Update docs for ND blobs (BVLC#1970) and layer type is a string (BVLC…

3bd95a4

…#1694)

lukeyeager mentioned this pull request Aug 20, 2015

Use input_shape instead of input_dim in examples #2950

Merged

jeffdonahue mentioned this pull request Aug 25, 2015

bugfix for ConcatLayer with propagate_down set #2972

Merged

seanbell mentioned this pull request Sep 16, 2015

Add argmax_param "axis" to maximise output along the specified axis #3069

Merged

wangyida pushed a commit to wangyida/caffe that referenced this pull request Sep 22, 2015

Update docs for ND blobs (BVLC#1970) and layer type is a string (BVLC…

233c981

…#1694)

cdoersch mentioned this pull request Oct 7, 2015

Batch Normalization Layer #3161

Closed

tiferet mentioned this pull request Jan 4, 2016

Training with HDF5: blob size exceeds INT_MAX #3510

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Blobs are N-D arrays (for N not necessarily equals 4) #1970

Blobs are N-D arrays (for N not necessarily equals 4) #1970

jeffdonahue commented Feb 25, 2015

jeffdonahue commented Mar 2, 2015

Yangqing Mar 2, 2015

jeffdonahue Mar 2, 2015

Yangqing Mar 2, 2015

shelhamer Mar 2, 2015

longjon commented Mar 2, 2015

longjon commented Mar 2, 2015

jeffdonahue commented Mar 3, 2015

shelhamer commented Mar 3, 2015

shelhamer Mar 3, 2015

shelhamer commented Mar 3, 2015

shelhamer Mar 3, 2015

jeffdonahue Mar 3, 2015

jeffdonahue commented Mar 3, 2015

Blobs are N-D arrays (for N not necessarily equals 4) #1970

Blobs are N-D arrays (for N not necessarily equals 4) #1970

Conversation

jeffdonahue commented Feb 25, 2015

jeffdonahue commented Mar 2, 2015

Yangqing Mar 2, 2015

Choose a reason for hiding this comment

jeffdonahue Mar 2, 2015

Choose a reason for hiding this comment

Yangqing Mar 2, 2015

Choose a reason for hiding this comment

shelhamer Mar 2, 2015

Choose a reason for hiding this comment

longjon commented Mar 2, 2015

longjon commented Mar 2, 2015

jeffdonahue commented Mar 3, 2015

shelhamer commented Mar 3, 2015

shelhamer Mar 3, 2015

Choose a reason for hiding this comment

shelhamer commented Mar 3, 2015

shelhamer Mar 3, 2015

Choose a reason for hiding this comment

jeffdonahue Mar 3, 2015

Choose a reason for hiding this comment

jeffdonahue commented Mar 3, 2015