Does CaffeOnSpark support multiple LMDB Files For Training/Testing

[stickFigure1235]

Does CaffeOnSpark support multiple LMDB files or is there an alternative?

An error occurs when I try to do something like the following:

layer {
name: "data"
type: "MemoryData"
top: "data"
top: "dummyLabel1"
include {
phase: TRAIN
}
source_class: "com.yahoo.ml.caffe.LMDB"
memory_data_param {
source: "file:train_data_lmdb/"
batch_size: 16
channels: 1
height: 1
width: 1072
share_in_parallel: true
}
}
layer {
name: "label"
type: "MemoryData"
top: "label"
top: "dummyLabel2"
include {
phase: TRAIN
}
source_class: "com.yahoo.ml.caffe.LMDB"
memory_data_param {
source: "file:train_label_lmdb/"
batch_size: 16
channels: 1
height: 1
width: 1072
share_in_parallel: true
}
}
layer {
name: "data"
type: "MemoryData"
top: "data"
top: "dummyLabel1"
include {
phase: TEST
}
source_class: "com.yahoo.ml.caffe.LMDB"
memory_data_param {
source: "file:test_data_lmdb/"
batch_size: 16
channels: 1
height: 1
width: 1072
share_in_parallel: true
}
}
layer {
name: "label"
type: "MemoryData"
top: "label"
top: "dummyLabel2"
include {
phase: TEST
}
source_class: "com.yahoo.ml.caffe.LMDB"
memory_data_param {
source: "file:test_label_lmdb/"
batch_size: 16
channels: 1
height: 1
width: 1072
share_in_parallel: true
}
}

layer{
name: "silence_layer"
type: "Silence"
bottom: "dummyLabel1"
bottom: "dummyLabel2"
}

layer{
name: "zero"
type: "Eltwise"
bottom: "data"
bottom: "label"
top: "zero"
eltwise_param {
operation: SUM
coeff: 1
coeff: -1
}
}

I get the following error:

F0613 18:06:55.973876 14601 memory_data_layer.cpp:112] Check failed: data_ MemoryDataLayer needs to be initalized by calling Reset

Thanks.

[junshi15]

No multiple sources at this moment. This is a feature we plan to support in the future.

[stickFigure1235]

Is HDF5 supported in CaffeOnSpark?

[junshi15]

No, our focus is distributed file formats, such as Spark DataFrame, Hadoop SequenceFile, etc. In the future, better support of DataFrame will be our main development effort. Those single-node file formats, such as LMDB, HDF5 are not quite suitable in a distributed environment. I have seen people used a single LMDB file of size 1TB for training! That's very much the opposite of distributed storage.