MPIR_gpu_req is a union type for either a MPL_gpu_request or a
MPIR_Typerep_req, thus it is not just for gpu. Potentially this type can
be extended to include other internal async task handles. Thus we rename
it to MPIR_async_req.

We also establish the convention of naming the variable async_req.

Add an inline wrapper for testing MPIR_async_req.

Modify the order of header inclusion due to the dependency on
typerep_pre.h.

Refactor the async copy in receive events using MPIR_async facilities.

Refactor the async copy before sending a chunk.

Both gpu_send_task_queue and gpu_recv_task_queue have been ported to
async things.

Pipeline send allocates chunk buffers then spawns async copy. The
allocation may run out of genq buffers, thus it is disigned as async
tasks.

The send copy are triggered upon completion of buffer alloc, thus it is
renamed into spawn_send_copy and turned into internal static function.

This removes MPIDI_OFI_global.gpu_send_queue.

Pipeline recv allocates chunk buffers and then post fi_trecv. The
allocation may run out of genq buffers and we also control the number of
outstanding recvs, thus it is designed as async tasks.

The async recv copy are triggered in recv event when data arrives.

This removes MPIDI_OFI_global.gpu_recv_queue.

All ofi-layer progress routines for gpu pipelining are now removed.

Consolidate the gpu pipeline code.

MPIDI_OFI_gpu_pipeline_request is now an internal struct in
ofi_gpu_pipeline.c, rename to struct chunk_req.

MPIDI_OFI_gpu_pipeline_recv_copy is now an internal function, rename to
start_recv_copy.

Move all gpu pipeline specific code into ofi_gpu_pipeline.c.

Make a new function MPIDI_OFI_gpu_pipeline_recv that fills rreq with
persistent pipeline_info data. Rename the original
MPIDI_OFI_gpu_pipeline_recv into static function start_recv_chunk.

Make the code cleaner to separate the pipeline_info type into a union of
send and recv.

Don't mix the usage of cc_ptr, use separate and explicit counters to
track the progress and completion of chunks.

Follow a similar approach as nonblocking collectives, internal pipeline
chunks use separate tag space (MPIDI_OFI_GPU_PIPELINE_SEND) and
incrementing tags to avoid mismatch with regular messages.

Separate the recv tasks between the initial header and chunks since the
paths clearly separates them.

Use a single async item for all chunk recvs rather than unnecessarily
enqueuing individual chunks since we can track the chunks in the state.

It is needed to compile under noinline configuration.

Move these utility functions to ofi_impl.h since they are simple and
non-specific. It also simplifies figuring out which file to include
especially for .c files.

Separate non-intercepting paths with distinct union members to make the
code more explicit.

The paths of pipeline code is not intercepting with other paths, thus
pipeline_info can be part of the same union.

Rather than sharing the pack_buffer field in pack_send, which obfuscates
the code, use a separate union member for the huge send path.

It is part of the recv path, thus need be in the recv struct.

The recv paths may switch paths depending on the actual protocols
used by sender, thus some of the fields from different paths need live
in the same struct and use the NULL sentinel to tell whether certain
path is in effect. Currently this include remote_info for huge_send
protocol, and pack_buffer for pack_recv. It's possible to have
huge_recv+pack_recv.

The am emulated inject path does not intercept with any native paths,
thus it should be part of the big union.

The util.iov field is used by fi_trecvmsg with FI_CLAIM flag or the huge
recv path for threshold checking and fi_read.

The fast path of MPIDI_OFI_EVENT_SEND and MPIDI_OFI_EVENT_RECV has
already been handled in MPIDI_OFI_dispatch_optimized. Thus it is not
critical to mark likely/unlikely branches in
MPIDI_OFI_dispatch_function. Just use a big switch for simplicity.

There is no way to recover a pipelined message in the native ofi path if
the receiver didn't expect it. We could always save the necessary
information in all recv paths, but I am not sure we are willing to
accept the overhead, which harms the small message latencies.

Assert failure for now.

The proper pipeline implementation need to happen as active messages or
even at the MPIR-layer.

Remove a redundant assignment of *request.