Training on Multiple GPU

[vahidk]

Can we use multiple gpus for training with caffe? What's the best strategy to do this? I couldn't find any documentation about this.

[shelhamer]

Caffe doesn't (yet) support this out-of-the-box, though see #194 for how CUDA 6 can distribute the BLAS computation across GPUs.

A DIY version of distributed solving could run training minibatches across GPUs, aggregate the diffs, then update the params of the replicated models (or just average the weights every now and again). How to effectively do this is still a research (+ engineering) question. None of the core Caffe devs are working on this at the moment, but that doesn't mean you shouldn't if you're interested.

See #65 for others who are interested in this line of work.

[vahidk]

Thanks for the information Evan.