Multi-device support

[neubig]

Currently, cnn supports running computation on either CPU or GPU, based on the setting of the -DHAVE_CUDA flag. It would be better if a single program could utilize both CPU and one or more GPUs within the same program. The framework for this is mostly there, minus a few remaining issues:

• There are a number of places, mainly in the parameters and tensor operations that are still hard-coded based on the setting of HAVE_CUDA. These all need to be changed to be dynamic based on the device setting for the tensor.
• Currently parameters are all declared on the default device. There needs to be an option to let parameters live on other devices.
• We need to create a node that moves memory between devices.
• Test code for multi-device support needs to be created.

[iamyoungjo]

@neubig Hello. Is this enhancement still on-going? If not, I would like to jump on this as I have some time & purpose to work on this.

[xunzhang]

Hi, I have an open pull request #382 but I am currently working on other features so you can go ahead. If I have time, we can work on it together. @neubig what about merging the slight change in that pull request first? Thanks.

[neubig]

Done. Also, I think this would be a good place to start for multi-device support: #443

[xunzhang]

Thanks a lot, I will work on it after finishing dynet-84 and will keep following on this as well.

[neubig]

I think this is basically done, although it might have some rough edges. Thanks @xunzhang !

[xunzhang]

Add reference for the related pull request #704 .