non-contiguous data-handling strategy

[cwpearson]

Removes KokkosComm_pack_traits.hpp and replaces it with a more comprehensive non-contiguous data-handling interface.

Warning

under construction

• irecv
• Kokkos::deep_copy
• MPI_Datatypes

Outline

A KokkosComm::NonContigXXX{SendRecv, Reduce, Alltoall} is responsible for converting View arguments to buffer/count/datatype tuples for the underlying MPI calls.

• ...SendRecv hands send and recv-like calls.
• ...Reduce handles reduce-like calls.
• ...Alltoall handles all-to-all like calls.
• I'm imagining other interfaces as needed

I found it very challenging to collapse these down to a single interface because the underlying MPI operations have a variety of different parameters with different meanings; however, in the final evaluation some refactoring to reduce the size of the interface may be possible.

On the plus side, if a vendor library only implements a subset of our communication interface, we would only need to implement a subset of the non-contiguous handing interface as well.

The basic outline of every MPI implementation now looks like this (using recv as an example)

void recv(const ExecSpace &space, RecvView &rv, int src, int tag, MPI_Comm comm) {
  CtxBufCount ctx = NC::pre_recv(space, rv);
  space.fence();  // space may be allocating our view
  for (const CtxBufCount::MpiArgs &args : ctx.mpi_args) {
    MPI_Recv(args.buf, args.count, args.datatype, src, tag, comm, MPI_STATUS_IGNORE);
  }
  NC::post_recv(space, rv, ctx);
}

Line (2) looks at the incoming view and does something specific to the Kokkos::deep_copy non-contiguous approach to decide what intermediate data to allocate, how to convert that into MPI arguments, etc.
The result of that is stashed in a CtxBufCount object, so named because it is used for any MPI calls that take a buffer and a count.
Line (3): fence, because space may be allocating our incoming data
Line (4-6): In this case, after packing, a single MPI_Recv call is make, but in general, that is not a requirement of the approach. So we consume all generated arguments and make the appropriate calls.
Line (7): maybe some work needs to be done on the receive side as well. No fence here, any required operations are inserted into space.

NC in this example is a struct that implements this pre_recv and post_recv interface. Other structs with the same interface can be used to implement different non-contiguous data handling strategies.