Synchronization on Partitioned Communication for Accelerator Optimization (Pbuf_Prepare et al)

[regrant]

Problem

This ticket introduces MPI_Pbuf_prepare, a call for guaranteeing that remote buffers are available before calling MPI_Pready for partitioned communication. This is important for optimizations to the MPI_Pready call that can be implemented on accelerators like GPUs/FPGAs.

Proposal

Introduce MPI_Pbuf_prepare and MPI_Pbuf_prepareall that provide remote buffer readiness guarantees from MPI. This enables GPU/Accelerator side MPI implementation of MPI_Pready with a single code path that is ideal for those architectures. MPI_Psync allows the MPI library to utilize accelerator triggered communications that are set up on the host CPU efficiently for kernel triggered communication. By avoiding buffer management and branching code paths, MPI_Pready and MPI_Parrived can be implemented using fast instructions on data flow centric architectures.

The proposed operation flow is as follows:

Changes to the Text

This ticket adds two calls to the partitioned communication chapter, MPI_Pbuf_prepare and MPI_Pbuf_prepareall

Impact on Implementations

Implementations will have to add support for these calls. This involves implementing a CTS-RTS type handshake for synchronization.

Impact on Users

Users will have a new mechanism to help writing code on accelerators for MPI_Pready that provides consistently optimized performance.

References

Pull request Synchronization on Partitioned Communication for Accelerator Optimization

Semantics table pull request

Please see only changes for this ticket to avoid pending partitioned communication merges.

[raffenet]

Isn't this a requirement for any RDMA-based implementation?

[regrant]

@raffenet most RDMA implementations would arrange this in the setup/start yes, but it's not guaranteed to happen by the semantics, you could call all of the pready calls on the initiator before the remote buffer was available at all and finish the buffer exchange and data payload in the wait completion. This guarantees that an accelerator like a GPU can be confident that triggerable operations on a NIC could be directly triggered and succeed from inside a kernel without knowledge of the state of the remote buffers in MPI (you know the state is guaranteed to be a certain one).

[raffenet]

What I was asking is doesn't this benefit non-GPU use-cases? I don't see why this is specific to accelerators.

[regrant]

Yes, it's also an optimization for non-GPUs, the host CPU could benefit from this synchronization in some cases as well, but it does have the negative impact of reducing some opportunities for earlybird communication. So on the CPU side it's less clear if this is always better as opposed to it's use with a GPU where it is in practice going to be always better due to the slow execution of the non-synchronized code path.

[raffenet]

@regrant makes sense. Thanks for the explanation.

[raffenet]

Yes, it's also an optimization for non-GPUs, the host CPU could benefit from this synchronization in some cases as well, but it does have the negative impact of reducing some opportunities for earlybird communication.

Actually, I'm a still curious about this statement. The finepoints paper mentions using RDMA, but makes no mention of earlybird communication or any additional synchronization to ensure the target buffer is available. How did it work? Is the source code available?

[raffenet]

What is the recv-side process responsibility with regard to MPI_Psync? Does it have to make the same call? The proposed text seems to be send-side focused.

[regrant]

Sorry, the early work on finepoints used the term earlybird communication, here's an illustration of it. Bascially you want to move data as it becomes available, this is possible with RDMA if you know your buffer is setup. If you don't or you want to aggregate partitions you can easily hold back sending the ready portions of the local buffer and send them when the remote buffer is available or you have the parts you want aggregated. The remote buffer being setup is probably something you want in practice (out prototype implementations guaranteed it, but that's not required).