New API for memory management via VFDs

[derobins]

From PR #1500 - I'm moving this here for discussion so we can close the PR (@derobins)

With the introduction of VFD plugins, it is now possible to have virtual
file drivers that specialize on data transfer between storage and
hardware accelerators. The GDS (GPUDirect Storage) VFD is one such
example that enables I/O between NVMe storage and GPU memory.

GDS allocates device memory using vendor-specific APIs that are not meant
to be exposed elsewhere in the HDF5 core library. Yet, other components
such as I/O filters would benefit from having access to the memory
allocator used by the VFD, as that could save data copies across device
and host memory, for instance.

This commit introduces two high-level memory management interfaces that
enable memory allocation/deallocation through the VFD driver (falling
back to the existing H5MM_malloc/xfree functions when the VFD does not
provide a custom implementation). The functions are implemented on top
of the H5FD ctl interface and are described in details below.

H5FDalloc_mem(): allocate memory via the file driver. If the driver does
not support the H5FD_CTL__MEM_ALLOC opcode then the operation falls back
to allocation via system memory (i.e., H5MM_malloc()). The input
arguments to that CTL operation are given by the H5FD_ctl_alloc_args_t
structure:

typedef struct H5FD_ctl_alloc_args_t {
    hsize_t       size;      /**< Memory allocation size */
    hsize_t       alignment; /**< Desired memory alignment */
    unsigned long flags;     /**< Request flags (H5FD_ALLOC*) */
    const void   *args;      /**< Optional arguments wanted by the VFD
                                  given the request flags */
} H5FD_ctl_alloc_args_t;

The flags and args members have been introduced to support fancy
memory allocators such as cudaMallocAsync() (which takes a stream ID
as argument), and Xilinx' platform extension to OpenCL which enables
clCreateBuffer() to take a target memory bank ID argument.

H5FDfree_mem(): release memory previously allocated via H5FDalloc_mem().
If the driver does not support the H5FD_CTL__MEM_FREE opcode, memory is
freed via H5MM_xfree(). Input arguments are given by the
H5FD_ctl_free_args_t structure:

typedef struct H5FD_ctl_free_args_t {
    void         *buf;   /**< Pointer to memory space to be freed */
    unsigned long flags; /**< Request flags (H5FD_MEM*) */
    const void   *args;  /**< Optional arguments wanted by the VFD
                              given the request flags */
} H5FD_ctl_free_args_t;

The same observations regarding flags and args apply here.

An application can query the supported flags by calling H5FDctl with
the new H5FD_CTL__MEM_QUERY_SUPPORTED_FLAGS opcode:

unsigned long supported_flags = 0;
unsigned long *flag_ptr = &supported_flags;
opcode = H5FD_CTL__MEM_QUERY_SUPPORTED_FLAGS;
H5FDctl(file_drv_ptr, opcode, 0, NULL, (void **) &flag_ptr);

This commit introduces a single H5FD_MEM_ASYNC flag. Other flags may be
added in the future depending on use-cases and demand.

Signed-off-by: Lucas C. Villa Real lucasvr@br.ibm.com

[qkoziol]

How would the library use these calls? i.e. what algorithm inside the library code would want to allocate a buffer on an accelerator device (like a GPU)?