Feature request: CUDA support

[njzjz]

Comment:

Support CUDA.

mpich README contains documentation:

GPU support
***********

GPU support is automatically enabled if CUDA, ZE, or HIP runtime is
detected during configure. To specify where your GPU runtime is
installed, use:

  --with-cuda=<path>  or --with-ze=<path> or --with-hip=<path>

If the lib/ and include/ are not in the same path, both can be specified
separately, for example:

  --with-cuda-include= and --with-cuda-lib=

In addition, GPU support can be explicitly disabled by using:

  --without-cuda  or  --without-ze or --without-hip

If desirable, GPU support can be disabled during runtime by setting
environment variable MPIR_CVAR_ENABLE_GPU=0. This may help avoid the GPU
initialization and detection overhead for non-GPU applications.

[jakirkham]

Likely we can take the same approach here as was taken with OpenMPI

[minrk]

I don't think that mpich loads optional components as DSOs like openmpi, so I think we'd at least need a cuda build and a non-cuda build to avoid a dependency on libcudart even when unused, since it will still need to resolve the symbols, even if they aren't called, but we can see.

I had hoped that one mpi implementation with cuda support would be enough for folks, since it is such a pain to maintain and nearly impossible to test.